A data preprocessing method based on multi-source information fusion

Abstract

The present application relates to the technical field of data processing, and specifically provides a data preprocessing method based on multi-source information fusion, which comprises the following steps: obtaining basic information of multi-source information data; traversing the data uploading operation content chapter by chapter to generate an information point architecture tree of a corresponding information database; loading a preconfigured information database, and establishing a model based on pre-trained modification backup; according to the selection result of the information point architecture tree, identifying keyword data of the point selection content, and pushing resource data filtered according to the keyword data into the modification backup for import; through the backup establishment model, the present application can backtrack the user's misoperation in the operation process, such as data misdeletion and mismodification, so as to avoid the phenomenon that data cannot be recovered due to misoperation, and can preprocess multi-source information, facilitating data retrieval and query.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data processing, and particularly relates to a data preprocessing method based on multi-source information fusion. BACKGROUND

[0002] With the rise of science and technology, intelligence and digitization, more and more behaviors of users can be collected and digitally recorded, for example, click behavior, purchase behavior, social behavior, entertainment behavior and the like. With more and more recorded behaviors, the amount of data is also increasing. Therefore, when a communication system transmits various information data of users, the load is too large, thereby affecting the working efficiency of the communication system and a response platform.

[0003] Chinese patent CN111291027A discloses a data preprocessing method, which comprises the following steps: collecting historical information, and arranging the historical information to form complete historical data; the complete historical data has several dimensions, and the dimensions are data features; resampling the complete historical data or selecting data features to obtain a training set. However, when the method processes data, it cannot backtrack the misoperation of a data user in an operation process, such as data deletion and modification errors, thereby increasing the difficulty of later tracing. Therefore, the application provides a data preprocessing method based on multi-source information fusion. SUMMARY

[0004] The application aims to provide a data preprocessing method based on multi-source information fusion, so as to solve the problem that the current method cannot backtrack the misoperation of a data user in an operation process when processing data.

[0005] To achieve the above-mentioned purpose, the application provides the following technical scheme.

[0006] The application provides a data preprocessing method based on multi-source information fusion, which comprises the following steps:

[0007] Obtaining basic information of multi-source information data, wherein the multi-source information basic information comprises data uploading operation corresponding to to-be-imported modification backup and multi-information database;

[0008] Traversing the data uploading operation content chapter by chapter to generate an information point architecture tree of the corresponding information database;

[0009] Loading a preconfigured information database, establishing a model based on pre-trained modification backup, and screening a resource sub-data set of the corresponding information point architecture tree;

[0010] With login information as input, input modification backup establishment model, according to information point architecture tree selection result, identify point selection content keyword data, and push resource data filtered according to the keyword data to import into the modification backup.

[0011] Further, the information point architecture tree of the corresponding information database comprises:

[0012] Obtain the establishment architecture tree message creation command by operating the message collection device:

[0013] Extract the architecture tree message creation command, judge the creation type of the creation architecture tree, and the creation architecture tree type includes single source data, multi-source data, multimedia data, geographic position and positioning data;

[0014] Input the type of creating the architecture tree, and input the type of creating the architecture tree into the architecture tree creation database;

[0015] Obtain the standard information point architecture tree based on the input creation architecture tree type.

[0016] Further, the training method of the pre-trained modification backup establishment model comprises:

[0017] Obtain key data sample data;

[0018] With key data sample data as input, execute the backup establishment model to be trained, extract the key data sample data corresponding to the question bank resource sample feature through the backup establishment model, and obtain the backup establishment resource of the key data sample data;

[0019] Based on the backup establishment resource, judge the correlation degree of the modification backup establishment model, if the correlation degree is greater than a preset correlation threshold, adjust the parameters of the modification backup establishment model, and continue training until the correlation degree is less than the preset correlation threshold.

[0020] Further, it also includes modification backup import, and the modification backup import method comprises:

[0021] Obtain the access request of the user end, and the access request comprises user identification and access password;

[0022] Verify the user identification and access password, and log in the personal data preprocessing account;

[0023] According to the loaded historical modification backup, modify or newly create the modification backup to be imported.

[0024] Further, the modification backup import method further comprises establishing a communication connection, and the establishing a communication connection comprises:

[0025] Based on the cloud platform protocol interface specification, execute the short message message execution command protocol code conversion.

[0026] Further, the cloud platform protocol interface specification is based on 3GPP, RFC, RCC, GSMA IR.92, IETF, OMA international technical standards.

[0027] Further, the screening of the resource sub-data set corresponding to the information point architecture tree specifically includes:

[0028] Judging sensitive information;

[0029] Integrating sensitive information in the short message message creation command to form a sensitive information database;

[0030] Iterating through the resource sub-data set and the sensitive information database one by one to filter out interference data.

[0031] Further, the judging of sensitive information types includes text recognition audit, picture recognition audit, video recognition audit, voice recognition audit, and link recognition audit.

[0032] Further, the sensitive information database establishment method specifically includes:

[0033] Receiving multi-source information standard sensitive data corresponding to multiple scenes, wherein the multi-source information standard sensitive data at least includes standard sensitive data corresponding to a scene and scene parameter data corresponding to the scene;

[0034] According to the multi-source information standard sensitive data corresponding to multiple scenes and the corresponding scene parameter data, the multi-source information standard sensitive data and the corresponding scene parameter data are verified;

[0035] Judging whether the multi-source information standard sensitive data corresponding to multiple scenes and the corresponding scene parameter data are associated with the judging sensitive information rule, if associated, determining that the multi-source information standard sensitive data corresponding to multiple scenes and the corresponding scene parameter data are sensitive data, if not, discarding the multi-source information standard sensitive data corresponding to multiple scenes and the corresponding scene parameter data, forming a sensitive information database, and setting a judgment threshold based on the multi-source information standard sensitive data corresponding to multiple scenes and the corresponding scene parameter data.

[0036] Further, the iteration through the resource sub-data set and the sensitive information database one by one to filter out interference data specifically includes:

[0037] Obtaining the resource sub-data set, determining the priority of the data in the resource sub-data set, and arranging the priority;

[0038] Taking the resource sub-data set determined by the priority as input, extracting the sensitive information database, iterating through the sensitive information database, judging the association degree of the sensitive information database and the data in the resource sub-data set, if the association degree is less than the set threshold, determining as sensitive information, if greater than the threshold, filtering out the data in the resource sub-data set.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] The embodiment of the present application can trace back the user's misoperation in the operation process, such as data deletion and modification, to avoid the phenomenon that data cannot be recovered due to misoperation, and can preprocess multi-source information to facilitate data retrieval and query. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The implementation flowchart of the data preprocessing method based on multi-source information fusion provided by the embodiment of the present application is shown.

[0042] Figure 2 The implementation flowchart of the information point architecture tree of the corresponding information database provided by the embodiment of the present application is shown.

[0043] Figure 3 The implementation flowchart of the training method of the pre-trained modification backup establishment model provided by the embodiment of the present application is shown.

[0044] Figure 4 The implementation flowchart of the modification backup import method provided by the embodiment of the present application is shown.

[0045] Figure 5 The implementation flowchart of the resource sub-data set of the corresponding information point architecture tree provided by the embodiment of the present application is shown.

[0046] Figure 6 The implementation flowchart of the sensitive information database establishment provided by the embodiment of the present application is shown.

[0047] Figure 7 The implementation flowchart of the resource sub-data set and the sensitive information database provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] The prior method cannot trace back the user's misoperation in the operation process, such as data deletion and modification, which increases the difficulty of later tracing, therefore, we propose a data preprocessing method based on multi-source information fusion,

[0050] The embodiment of the present application obtains basic information of multi-source information data, the multi-source information basic information includes data upload operation corresponding to to-be-imported modification backup and multi-information database, then traverses the data upload operation content chapter by chapter, generates information point architecture tree of the corresponding information database, loads a preconfigured information database, establishes a model based on pre-training modification backup, screens resource sub-dataset of the corresponding information point architecture tree, finally inputs the login information as input, inputs the modification backup establishment model, identifies keyword data of the point selection content according to the information point architecture tree selection result, and pushes the resource data screened according to the keyword data to the import modification backup. The embodiment of the present application can backtrack the user's misoperation in the operation process, such as data misdeletion and mismodification, thereby avoiding the phenomenon that data cannot be recovered due to misoperation, and can preprocess multi-source information, facilitating data retrieval and query.

[0051] As shown in Figure 1 The multi-source information fusion-oriented data preprocessing method provided by the embodiment of the present application comprises the following steps:

[0052] Step S1, obtaining basic information of multi-source information data, the multi-source information basic information including data upload operation corresponding to to-be-imported modification backup and multi-information database.

[0053] In the embodiment, the basic information of multi-source information data includes data upload operation corresponding to to-be-imported modification backup and multi-information database, wherein the information database contains original input record, modification record backup and storage record of multi-source information.

[0054] Step S2, traversing the data upload operation content chapter by chapter, generating information point architecture tree of the corresponding information database.

[0055] Step S3, loading a preconfigured information database, establishing a model based on pre-training modification backup, and screening resource sub-dataset of the corresponding information point architecture tree.

[0056] Step S4, inputting the login information as input, inputting the modification backup establishment model, identifying keyword data of the point selection content according to the information point architecture tree selection result, and pushing the resource data screened according to the keyword data to the import modification backup.

[0057] As shown in Figure 2 The specific steps of generating the information point architecture tree of the corresponding information database comprise the following steps:

[0058] The information point architecture tree for generating the corresponding information database includes:

[0059] Step S201: Obtain the architecture tree creation command by operating the message acquisition device.

[0060] For example, the message collection device can be a card reader or a mobile terminal, wherein the mobile terminal is a smartphone, home computer, industrial computer or service terminal, which can collect the original records of user input and back up the records for input errors. For example, if a user accidentally deletes or touches a device at a bank self-service terminal, resulting in damage to personal data, the user can perform a backup operation without needing to retrieve the data from the backend system, thus reducing the workload of the backend system and simplifying the process.

[0061] For example, users can collect users' input records through a message collection device for bookkeeping personnel. When erroneous input occurs, the system can directly perform traceability work, thereby reducing the workload of financial personnel.

[0062] Step S202: Extract the architecture tree message creation command and determine the creation type of the architecture tree. The creation type of the architecture tree includes single-source data, multi-source data, multimedia data, geographic location and positioning data.

[0063] For example, single-source data represents data from an individual user, while multi-source data represents data from multiple users or a group of users. Multimedia data can be data input by the user that contains audio, images, and video, while geolocation and positioning data can be location data sent on behalf of the user.

[0064] Step S203: Input the type of architecture tree to be created. Input the architecture tree type to create the database.

[0065] Step S204: Obtain a standard information point architecture tree based on the input to create an architecture tree type.

[0066] In this embodiment, the method for creating a standard information point architecture tree based on input includes:

[0067] The single multi-source data in the creation architecture tree type is parsed to obtain the data parameters corresponding to each creation architecture tree type; the data parameters that do not conform to the creation architecture tree type are discarded to obtain at least one filtered preprocessed data; and then the at least one filtered preprocessed data is sorted.

[0068] like Figure 3 The diagram illustrates the specific steps of a training method for a pre-trained modified backup model, provided in an embodiment of the present invention. The training method specifically includes:

[0069] Step S301, obtain key data sample data;

[0070] Step S302, input the key data sample data, execute the backup establishment model to be trained, extract the backup establishment resource of the key data sample data through the backup establishment model corresponding to the question bank resource sample feature of the key data sample data;

[0071] Step S303, based on the backup establishment resource, judge the correlation degree of the modified backup establishment model, if the correlation degree is greater than a preset correlation threshold, adjust the parameters of the modified backup establishment model, and continue training until the condition is met.

[0072] The backup establishment model training model is continuously trained, so that the training result output is more accurate. The message training model is a neural network model, that is, a neural network model in machine learning. Neural networks (Neural Networks, NN) are a complex neural network system formed by a large number of simple processing units (called neurons) widely connected with each other, which reflects many basic characteristics of brain function, and is a highly complex nonlinear dynamic learning system. Simply speaking, it is a mathematical model. In the embodiment of the application, the backup establishment model is continuously trained until the condition is met.

[0073] As shown in the figure, the specific steps of the modification backup import method provided by an embodiment of the application also include a modification backup import, and the modification backup import method specifically includes: Figure 4

[0074] Step S501, obtain an access request of a user end, the access request including a user identifier and an access password;

[0075] Step S502, verify the user identifier and the access password, and log in a personal data preprocessing account;

[0076] Step S503, modify or newly create a modification backup to be imported according to a loaded historical modification backup.

[0077] In the embodiment, the modification backup import method further includes establishing a communication connection, and the establishing of the communication connection includes:

[0078] The short message message execution command protocol is transcoded based on a cloud platform protocol interface specification, and the cloud platform protocol interface specification is based on 3GPP, RFC, RCC, GSMA IR.92, IETF, and OMA international technical standards.

[0079] As shown in the figure, the specific steps of the modification backup import method provided by an embodiment of the application also include a modification backup import, and the modification backup import method specifically includes: Figure 5 ​The diagram illustrates the specific steps for filtering resource subsets of a corresponding information point architecture tree according to an embodiment of the present invention. The specific method for filtering resource subsets of a corresponding information point architecture tree includes:

[0080] Step 401, determine sensitive information;

[0081] Step 402: Integrate sensitive information from the SMS message creation command to form a sensitive information database;

[0082] Step 403: Go through the resource subset and sensitive information database one by one to filter out interfering data.

[0083] In this embodiment, the determination of sensitive information types includes text recognition review, image recognition review, video recognition review, voice recognition review, and link recognition review.

[0084] like Figure 6 The diagram illustrates the specific steps of a sensitive information database establishment method provided in an embodiment of the present invention. The specific method for filtering resource subsets of the corresponding information point architecture tree includes:

[0085] Step 4021: Receive multi-source information standard sensitive data corresponding to multiple scenarios, wherein the multi-source information standard sensitive data includes at least one standard sensitive data corresponding to a scenario and scenario parameter data corresponding to the scenario;

[0086] Step 4022: Verify the multi-source information standard sensitive data and the corresponding scene parameter data based on the multi-scenario corresponding multi-source information standard sensitive data and the corresponding scene parameter data;

[0087] Step 4023: Determine whether the multi-source information standard sensitive data and the corresponding scene parameter data corresponding to multiple scenarios are related to the rules for judging sensitive information. If they are related, determine that the multi-source information standard sensitive data and the corresponding scene parameter data corresponding to multiple scenarios are sensitive data. If not, discard the multi-source information standard sensitive data and the corresponding scene parameter data corresponding to multiple scenarios to form a sensitive information database, and set a judgment threshold based on the multi-source information standard sensitive data and the corresponding scene parameter data corresponding to multiple scenarios.

[0088] like Figure 7 The illustration shows the specific steps for sequentially traversing a resource subset and a sensitive information database to filter out interfering data, as provided in an embodiment of the present invention. The specific method for sequentially traversing a resource subset and a sensitive information database to filter out interfering data includes:

[0089] Step 4031: Obtain the resource subset, determine the priority of the data in the resource subset, and sort the priorities;

[0090] Step 4032, taking the resource sub-dataset determined by priority as input, extracting the sensitive information database, traversing the sensitive information database, judging the association degree of the sensitive information database and the data in the resource sub-dataset, if the association degree is less than the set threshold, determining the sensitive information, if greater than the threshold, screening out the data in the resource sub-dataset.

[0091] The embodiment of the application can trace back the misoperation of the user in the operation process, such as data misdeletion and mismodification, thereby avoiding the phenomenon that data cannot be recovered due to misoperation, and can preprocess multi-source information, thereby facilitating data retrieval and query.

[0092] The embodiment of the application further provides a readable storage medium, the readable storage medium stores a computer program, and the computer program is executed by a processor to enable the processor to execute steps of the data preprocessing method based on multi-source information fusion.

[0093] For example, the computer program can be divided into one or more modules, the one or more modules are stored in the memory and executed by the processor to complete the application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the terminal device. For example, the above computer program can be divided into the units or modules for data preprocessing based on multi-source information fusion provided by the above various system embodiments.

[0094] Those skilled in the art can understand that the description of the terminal device is only an example and does not constitute a limitation on the terminal device, and can include more or less components than the above description, or combine certain components or different components, for example, can include input and output devices, network access devices, buses, etc.

[0095] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The above processor is the control center of the terminal device, and is connected with various parts of the terminal device through various interfaces and lines.

[0096] The memory can be configured to store computer programs and / or modules, and the processor can realize the various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, where the program storage area can store an operating system, at least one application program required by a function (such as an information collection template display function, a product information publishing function, etc.), and the like; and the data storage area can store data created according to the use of the berth state display system (such as product information collection templates corresponding to different product categories, product information to be published by different product providers, etc.). In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0097] If the modules / units integrated in the terminal device are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the modules / units in the above-mentioned embodiment systems can also be instructed by a computer program to related hardware to complete, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the functions of the above-mentioned various system embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0098] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

[0099] The above merely provides the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-source information fusion oriented data preprocessing method, characterized in that, The multi-source information fusion-oriented data preprocessing method comprises the following steps: Obtain multi-source information data basic information, which includes data upload operations corresponding to the to-be-imported modified backup and multi-information databases; Traverse the data upload operation content chapter by chapter to generate an information point architecture tree of the corresponding information database; Load a preconfigured information database, establish a model based on the modified backup, and filter a resource sub-data set of the corresponding information point architecture tree; Input the login information as input, input the modified backup establishment model, identify the keyword data of the selected content according to the information point architecture tree selection result, and push the resource data filtered according to the keyword data to the modified backup for import; The generation of the information point architecture tree of the corresponding information database comprises the following steps: Obtain the establishment architecture tree message creation command through the operation message acquisition device: Extract the architecture tree message creation command, judge the creation type of the created architecture tree, and the creation type of the created architecture tree includes single-source data, multi-source data, multimedia data, geographic position and positioning data; Input the type of the created architecture tree, and input the type of the created architecture tree into the architecture tree creation database; Obtain the standard information point architecture tree based on the input type of the created architecture tree; The training method of the modified backup establishment model comprises the following steps: Obtain key data sample data; Input the key data sample data, execute the to-be-trained modified backup establishment model, extract the key data sample data corresponding to the question bank resource sample feature through the modified backup establishment model, obtain the backup establishment resource of the key data sample data, judge the correlation degree of the modified backup establishment model based on the backup establishment resource, and if the correlation degree is greater than a preset correlation threshold, adjust the parameters of the modified backup establishment model and continue training until the condition is met. The modified backup import method comprises the following steps:

2. The data preprocessing method based on multi-source information fusion according to claim 1, characterized in that, Obtain an access request of a user terminal, which includes a user identifier and an access password; Verify the user identifier and the access password, and log in to a personal data preprocessing account; According to the loaded historical modified backup, modify or newly create the to-be-imported modified backup. The modified backup import method further comprises establishing a communication connection, which comprises the following steps:

3. The data preprocessing method based on multi-source information fusion according to claim 2, characterized in that, Protocol transcoding of short message message execution commands based on cloud platform protocol interface specifications. The cloud platform protocol interface specification is based on 3GPP, RFC, RCC, GSMA IR.92, IETF, OMA international technical standards.

4. The data preprocessing method based on multi-source information fusion according to claim 3, characterized in that, The filtering of the resource sub-data set of the corresponding information point architecture tree specifically comprises the following steps:

5. The data preprocessing method based on multi-source information fusion according to claim 2, characterized in that, Determine the sensitive information; Integrate the sensitive information in the short message message creation command to form a sensitive information database; Traverse the resource sub-data set and the sensitive information database one by one to filter out interference data. The judgment of the sensitive information type includes text recognition audit, picture recognition audit, video recognition audit, voice recognition audit and link recognition audit.

6. The data preprocessing method based on multi-source information fusion according to claim 5, characterized in that, The sensitive information database establishment method specifically comprises the following steps:

7. The data preprocessing method based on multi-source information fusion according to claim 5, characterized in that, Receive multi-source information standard sensitive data corresponding to multiple scenes, wherein the multi-source information standard sensitive data at least includes standard sensitive data corresponding to a scene and scene parameter data corresponding to the scene; ​ According to the multi-scene corresponding multi-source information standard sensitive data and the corresponding scene parameter data, the multi-source information standard sensitive data and the corresponding scene parameter data are checked; Determine whether the multi-scene corresponding multi-source information standard sensitive data and the corresponding scene parameter data are associated with the sensitive information rule, if associated, determine that the multi-scene corresponding multi-source information standard sensitive data and the corresponding scene parameter data are sensitive data, if not, discard the multi-scene corresponding multi-source information standard sensitive data and the corresponding scene parameter data, form a sensitive information database, and set a judgment threshold based on the multi-scene corresponding multi-source information standard sensitive data and the corresponding scene parameter data.

8. The data preprocessing method based on multi-source information fusion according to claim 6, characterized in that, Traverse the resource sub-data set and the sensitive information database one by one, and exclude interference data, which specifically includes: Obtain the resource sub-data set, determine the priority of the data in the resource sub-data set, and arrange the priority; With the resource sub-data set determined by the priority as the input, extract the sensitive information database, traverse the sensitive information database, judge the association degree of the sensitive information database and the data in the resource sub-data set, if the association degree is less than the set threshold, it is determined as sensitive information, if it is greater than the threshold, the data in the resource sub-data set is excluded.

Images