Data processing method based on multi-dimensional data

Abstract

The invention relates to the technical field of data analysis, in particular to a data processing method based on multi-dimensional data, which comprises the following steps: acquiring historical identification data of a target database, performing data cleaning on the acquired historical identification data, and preprocessing and screening the cleaned data to obtain abnormal identification data; grouping and marking the abnormal identification data, and performing validity calculation and reliability calculation on each abnormal identification data to obtain corresponding validity and reliability; comparing and analyzing the reliability of the abnormal identification data with a preset reliability critical threshold value to obtain deviation data; the deviation occurrence rate of the deviation data is calculated and analyzed, and a deviation data management evaluation criterion is dynamically formulated according to an analysis result; the method and the device are used for solving the technical problem that abnormal identification data deviation cannot be analyzed and intelligently managed in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a data processing method based on multidimensional data. Background Art

[0002] Artificial intelligence refers to the process of using artificial intelligence technology and algorithms to analyze, judge and classify input data. It can recognize and understand various forms of data such as images, voice, text, video, etc., and extract useful information and patterns from them.

[0003] However, current technologies cannot actively and efficiently monitor and analyze the deviation and anomaly of recognition data, and it is difficult to conduct targeted management to improve the monitoring effect of data recognition. To this end, the present invention proposes a data processing method based on multidimensional data, which processes and screens the acquired historical recognition data to obtain abnormal recognition data, calculates and analyzes the validity and reliability of the abnormal recognition data to obtain deviation data, and evaluates and manages the deviation data, so as to timely and efficiently improve the supervision and processing effect of abnormal data. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the background technology and to propose a data processing method based on multidimensional data.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A data processing method based on multidimensional data, comprising:

[0007] Acquire historical identification data of the target database, perform data cleaning on the acquired historical identification data, and pre-process and screen the cleaned data to obtain abnormal identification data;

[0008] The abnormal identification data are grouped and marked, and the validity and reliability calculations are performed on each abnormal identification data to obtain the corresponding validity and reliability;

[0009] Compare and analyze the reliability of abnormal identification data with the preset critical reliability threshold to obtain deviation data;

[0010] Calculate and analyze the deviation incidence rate of deviation data, and dynamically formulate deviation data management evaluation criteria based on the analysis results.

[0011] Preferably, the steps of cleaning the acquired historical identification data and preprocessing the cleaned data include:

[0012] Automatically identify and correct data quality issues and standardize data formats through data quality tools;

[0013] Acquire historical recognition data based on the historical recognition data collection module, and group the historical recognition data according to different types, including a natural language data group, a voice data group, and an image data group;

[0014] Each type of data set is labeled Hj, j=1, 2, 3.

[0015] Preferably, the recognition rate of the data set Hj is counted; the recognition rate of the data set is marked as SLj;

[0016] Set the data group recognition rate standard value Bj; and calculate the recognition index ZS corresponding to the data group set through the formula ZS = (SLj / Bj) × 100%;

[0017] When drawing a recognition index curve based on the calculated recognition index value and analyzing the recognition index change trend of historical recognition data;

[0018] Obtain an identification index change curve corresponding to any set of identification data; determine an abnormal critical value through the identification index change curve;

[0019] A number of calculation cycles are determined according to the abnormal critical value, and the abnormal critical values ​​of all data groups are averaged based on the calculation cycle; data below the mean is marked as abnormal identification data.

[0020] Preferably, based on a number of calculation cycles, the abnormal identification data are grouped according to corresponding data types and marked Ai, i=1, 2, 3, and the recognition rates Si corresponding to the abnormal identification data are recorded by group;

[0021] Calculate the validity of each abnormal identification data to obtain the validity D of each corresponding abnormal identification data; continue to calculate the reliability of each abnormal identification data according to its validity to obtain the reliability XD of the abnormal identification data;

[0022] Preferably, when calculating the validity of each abnormal identification data, the normal response time Ti0 of each identification data group is determined; the response time Ti and the recognition rate Si of the identification process of each abnormal identification data are counted; and the validity D of each abnormal identification data is calculated by the formula D = (Ti-Ti0) × Si.

[0023] Preferably, the reliability of each abnormal identification data is further calculated according to its validity, and the identification complexity Fi of the identification process of each abnormal identification data is counted; by formula The reliability XD of the abnormal identification data is calculated; wherein α is a preset proportional coefficient of the reliability of the abnormal identification data, and 0<α<1.

[0024] Preferably, the reliability XD of the abnormal identification data is compared and analyzed with the preset reliability critical threshold XD0; if XD is not greater than XD0, the reliability of the abnormal identification data is determined to be invalid and an invalid label is generated, and the corresponding abnormal identification data is marked as deviation data according to the invalid label; if XD is greater than XD0, the reliability of the abnormal identification data is determined to be valid and a valid label is generated.

[0025] Preferably, obtain the reliability corresponding to the deviation data and use the formula The deviation occurrence rate PF is calculated, where β is a variable constant parameter;

[0026] An assessment is conducted based on the numerical value of the deviation occurrence rate, management guidelines are formulated, the deviation occurrence rate is traversed and matched with all preset deviation sending ranges to obtain the corresponding deviation sending ranges and generate corresponding management instructions, and the obtained management instructions are sent to technicians of different levels for processing and analysis.

[0027] Compared with the prior art, the advantages of the data processing method based on multidimensional data provided by the present invention are:

[0028] The present invention collects historical recognition data of an artificial intelligence database, groups and marks them, and screens and cleans the data to avoid duplicate, incomplete and inaccurate data; receives preprocessed data through an abnormal recognition processing and analysis module, and obtains abnormal recognition data, which is conducive to better management and use of data and efficient analysis of data; obtains corresponding validity and reliability by performing data calculation on all abnormal recognition data, compares and analyzes the reliability of the abnormal recognition data with a preset reliability critical threshold to obtain deviation data, conducts comprehensive evaluation management based on the deviation data, formulates deviation data management evaluation criteria, and generates management instructions of different levels.

[0029] To sum up, the present invention can calculate the deviation incidence rate according to actual conditions and in combination with the deviation data, analyze and process the calculation results, and determine whether the numerical value of the deviation incidence rate exceeds a preset range by evaluating the system. The system generates management instructions of different levels and sends the information to technicians of different levels for processing operations, thereby ensuring the normal operation of a subsequent data processing method based on multidimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flowchart of a data processing method based on multidimensional data. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Reference Figure 1 , a data processing method based on multidimensional data, comprising:

[0033] The historical recognition data of the target database is obtained through the historical recognition data collection module, the acquired historical recognition data is cleaned, and the cleaned data is preprocessed and screened to obtain abnormal recognition data;

[0034] Among them, the historical identification data acquisition module includes an instruction receiving unit, a data acquisition unit, a data output unit and a data preprocessing unit;

[0035] The historical identification data acquisition module is connected to the PC end, and the instruction receiving unit is used to receive the query instruction for the historical identification data issued by the system;

[0036] The data acquisition unit is used to obtain the row number and table of the database where the historical identification data is located according to the query instruction, obtain the historical identification data according to the row number and table, and input the obtained historical identification data into the target table;

[0037] The data output unit is used to extract information from the target data of the target table and store the acquired target data;

[0038] The data preprocessing unit is used to clean the stored target data, automatically identify and correct data quality problems through data quality tools, standardize the data format, and preprocess the cleaned data;

[0039] The abnormal identification data are grouped and marked, and the validity and reliability calculations are performed on each abnormal identification data to obtain the corresponding validity and reliability;

[0040] Compare and analyze the reliability of abnormal identification data with the preset critical reliability threshold to obtain deviation data;

[0041] Calculate and analyze the deviation incidence rate of deviation data, and dynamically formulate deviation data management evaluation criteria based on the analysis results.

[0042] It should be noted that the application object in the embodiment of the present invention can be a data deviation monitoring database of an artificial intelligence recognition platform developed by a certain technology company, which is used for statistical analysis and monitoring of whether the recognition data is abnormal, and for managing and storing the deviation and abnormal data, wherein the indicators for monitoring the recognition data include other indicators such as image data, voice data, and natural language data of the artificial intelligence recognition platform.

[0043] The steps of cleaning the acquired historical identification data and preprocessing and screening the cleaned data to obtain abnormal identification data include:

[0044] Acquire historical recognition data based on the historical recognition data collection module, and group the historical recognition data according to different types, including a natural language data group, a voice data group, and an image data group;

[0045] Label each type of data set Hj, j = 1, 2, 3;

[0046] Among them, artificial intelligence recognition has a wide range of fields. In natural language processing tasks, such as text classification and sentiment analysis, artificial intelligence systems with high recognition rates can improve the accuracy of text understanding and semantic analysis; in the field of speech and image recognition, the improvement of artificial intelligence recognition rates can help the system accurately recognize speech and classify objects, scenes, and people in images, which is helpful for applications such as smart homes, image search, autonomous driving, and security monitoring.

[0047] Calculate the recognition rate of the data set Hj; mark the data set recognition rate as SLj;

[0048] It should be noted that the recognition rate of artificial intelligence refers to the accuracy of correct recognition or classification of a given task or problem in an artificial intelligence system, usually expressed as a percentage, indicating the proportion of correctly recognized samples to the total number of samples;

[0049] Set the data group recognition rate standard value Bj; and calculate the recognition index ZS corresponding to the data group set through the formula ZS = (SLj / Bj) × 100%;

[0050] When drawing a recognition index curve based on the calculated recognition index value and analyzing the recognition index change trend of historical recognition data;

[0051] Obtain an identification index change curve corresponding to any set of identification data; determine an abnormal critical value through the identification index change curve;

[0052] Determine a number of calculation cycles according to the abnormal critical value, take the average of the abnormal critical values ​​of all data groups based on the calculation cycle; mark the data below the average as abnormal identification data;

[0053] For example, taking a certain speech data group as an example, the recognition rate SL of the speech data group is obtained as {80%; 75%; 55%; 60%; 45%; 95%}; the speech data group recognition rate standard value B is set as 90%; then the data group recognition index ZS is {89%; 83%; 61%; 67%; 50%; 101%}; a recognition index curve is drawn with the horizontal axis as the speech data group number and the vertical axis as the recognition index of the speech data group; the normal value of the recognition index is set to 80%; the abnormal critical value is determined to be {61%; 67%; 50%} through the recognition index change curve; the abnormal critical values ​​of all speech data groups are averaged; and the data below the average is marked as abnormal recognition data.

[0054] Based on several calculation cycles, the abnormal identification data are grouped according to the corresponding data types and marked Ai, i = 1, 2, 3, and the recognition rate Si corresponding to the abnormal identification data is recorded by group;

[0055] Calculate the validity of each abnormal identification data to obtain the validity D of each corresponding abnormal identification data; continue to calculate the reliability of each abnormal identification data according to its validity to obtain the reliability XD of the abnormal identification data;

[0056] Specifically, when calculating the validity of each abnormal identification data, determine the normal response time Ti0 of each identification data group; count the response time Ti and the recognition rate Si of the identification process of each abnormal identification data; calculate the validity D of each abnormal identification data by the formula D=(Ti-Ti0)×Si;

[0057] In addition, the reliability of each abnormal recognition data is further calculated according to its validity, and the recognition complexity Fi of the recognition process of each abnormal recognition data is counted; through the formula The reliability XD of the abnormal identification data is calculated; wherein α is a preset proportional coefficient of the reliability of the abnormal identification data, and 0<α<1.

[0058] Among them, recognition complexity refers to the complexity of the AI ​​recognition object, which can be obtained according to the established level standards. The first level of complexity of AI recognition is to collect data through sensing the environment for preliminary understanding and analysis; the second level of complexity is to make inferences and decisions based on existing knowledge and predict future situations; the third level of complexity is to conduct autonomous learning and innovation and apply it to the actual environment; in addition, the higher the complexity, the longer the corresponding response time;

[0059] Compare and analyze the reliability XD of the abnormal identification data with the preset critical reliability threshold XD0;

[0060] If XD is not greater than XD0, the reliability of the abnormal identification data is determined to be invalid and an invalid label is generated. The corresponding abnormal identification data is marked as deviation data according to the invalid label;

[0061] If XD is greater than XD0, it is determined that the reliability of the anomaly recognition data is valid and a valid label is generated;

[0062] Obtain the reliability corresponding to the deviation data, and calculate the deviation incidence PF through the formula where β is a variable constant parameter;

[0063] Evaluate according to the value of the deviation incidence, formulate management guidelines, traverse and match the deviation incidence with all preset deviation sending ranges to obtain the corresponding deviation sending range and generate the corresponding management instructions, and send the obtained management instructions to technicians at different levels for processing and analysis;

[0064] For example, if PF belongs to (Y3, Y4), the system generates a first-level management instruction;

[0065] If PF belongs to (Y2, Y3), the system generates a second-level management instruction;

[0066] If PF belongs to (Y1, Y2), the system generates a third-level management instruction; where Y1 < Y2 < Y3 < Y4, and Y1, Y2, Y3, Y4 all belong to (0, 1);

[0067] The management levels corresponding to the first-level management instruction, the second-level management instruction, and the third-level management instruction decrease in turn.

[0068] In the embodiments of the present invention, by obtaining the historical recognition data of the target database and performing grouping and marking, the preprocessing data is received by the anomaly recognition processing and analysis module, and the anomaly recognition data is obtained. The anomaly recognition data calculation module performs data calculation on all anomaly recognition data to obtain the corresponding reliability and validity, compares and analyzes the reliability of the anomaly recognition data with the preset reliability critical threshold to obtain the deviation data, and performs comprehensive evaluation and management on the deviation data through the comprehensive evaluation and management module, formulates the deviation data management evaluation criteria, and generates management instructions at different levels. In summary, the embodiments of the present invention involve data collection and analysis, result generation, and decision-making on optimization measures, and solve the problem of intelligent management of the deviation of recognition data in a data processing method based on multi-dimensional data. In actual situations, more data and context information may be required to make specific decisions and optimization plans.

[0069] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device embodiments, since they are basically based on the method embodiments, they are described relatively simply, and the relevant parts can be referred to the partial description of the method embodiments.

[0070] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0071] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0075] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;

[0076] Finally: The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. Obtain historical identification data from the target database, perform data cleaning on the acquired historical identification data, and pre-process and screen the cleaned data to obtain abnormal identification data; in, Automatically identify and correct data quality issues and standardize data formats through data quality tools; Acquire historical recognition data based on the historical recognition data collection module, and group the historical recognition data according to different types, including a natural language data group, a voice data group, and an image data group; Label each type of data set Hj, j = 1, 2, 3; Calculate the recognition rate of the data set Hj; mark the data set recognition rate as SLj; Set the data group recognition rate standard value Bj; and calculate the recognition index ZS corresponding to the data group set through the formula ZS = (SLj / Bj) × 100%; When drawing a recognition index curve based on the calculated recognition index value and analyzing the recognition index change trend of historical recognition data; Obtain an identification index change curve corresponding to any set of identification data; determine an abnormal critical value through the identification index change curve; Determine a number of calculation cycles according to the abnormal critical value, and average the abnormal critical values ​​of all data groups based on the calculation cycle; Mark the data below the mean as outlier identification data; The abnormal identification data are grouped and marked, and the validity and reliability calculations are performed on each abnormal identification data to obtain the corresponding validity and reliability; Among them, based on several calculation cycles, the abnormal identification data are grouped according to the corresponding data type and marked Ai, i = 1, 2, 3, and the recognition rate Si corresponding to the abnormal identification data is recorded by group; Calculate the validity of each abnormal identification data to obtain the validity D of each corresponding abnormal identification data; continue to calculate the reliability of each abnormal identification data according to its validity to obtain the reliability XD of the abnormal identification data; When calculating the validity of each abnormal identification data, determine the normal response time Ti0 of each identification data group; count the response time Ti and the recognition rate Si of the identification process of each abnormal identification data; calculate the validity D of each abnormal identification data by the formula D = (Ti-Ti0) × Si; According to the validity of each abnormal recognition data, the reliability is further calculated, and the recognition complexity Fi of the recognition process of each abnormal recognition data is counted; through the formula Calculate and obtain the reliability XD of the abnormal identification data; where α is a preset proportional coefficient of the reliability of the abnormal identification data, and 0<α<1; Compare and analyze the reliability of abnormal identification data with the preset critical reliability threshold to obtain deviation data; Calculate and analyze the deviation rate of deviation data, and dynamically formulate deviation data management evaluation criteria based on the analysis results; obtain the reliability of the deviation data, and use the formula The deviation occurrence rate PF is calculated, where β is a variable constant parameter; XD0 is the preset critical threshold of reliability; An assessment is conducted based on the numerical value of the deviation occurrence rate, management guidelines are formulated, the deviation occurrence rate is traversed and matched with all preset deviation sending ranges to obtain the corresponding deviation sending ranges and generate corresponding management instructions, and the obtained management instructions are sent to technicians of different levels for processing and analysis.

2. A data processing method based on multidimensional data according to claim 1, characterized in that: Compare and analyze the reliability XD of the abnormal identification data with the preset critical reliability threshold XD0; If XD is greater than XD0, the reliability of the abnormal identification data is determined to be invalid and an invalid label is generated. The corresponding abnormal identification data is marked as deviation data according to the invalid label; if XD is not greater than XD0, the reliability of the abnormal identification data is determined to be valid and a valid label is generated.

Images