Data element extraction and analysis system based on artificial intelligence

Abstract

The invention relates to data processing, in particular to a data element extraction and analysis system based on artificial intelligence, and the system comprises an application scene management unit which carries out the classification management of data application scenes belonging to each specific application field, extracts data elements which are depended in each type of data application scenes, and carries out the analysis of the data elements. Establishing an association relationship between the data element and a data application scene so as to provide a training sample for a data element feature training unit; the data element feature training unit is used for screening data element samples used for model training from the training samples stored in the application scene management unit, continuously improving the accuracy of data element recognition by the model through a sample training self-learning process, and optimizing the model according to a model comprehensive efficiency value provided by the quality evaluation unit; according to the technical scheme provided by the invention, the defect that data elements are difficult to accurately and efficiently extract and analyze according to a data application scene in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to data processing, and in particular to a data element extraction and analysis system based on artificial intelligence. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, and data has become an important asset for enterprises and society. However, how to extract valuable digital information, commonly known as data elements, from massive, complex, multi-source heterogeneous data and conduct in-depth analysis and utilization has become a problem that needs to be solved urgently. As the overall development trend and requirements from data elementization to data resourceization and then to data assetization become increasingly obvious, traditional big data analysis methods face new challenges both at the data extraction level and at the application model level, and it is difficult to meet the overall needs of current data assetization.

[0003] With the rapid development of the new generation of artificial intelligence technology, especially the increasing maturity of large-scale model analysis technology, the combination of cutting-edge artificial intelligence technology and data element extraction and analysis, and the efficient identification and extraction of data elements through various training and prediction technologies including large models, and the provision of efficient data element library resources for data elements to form data resources, will inevitably provide a reliable and efficient technical foundation for the application of various types of data assets in the table, and ultimately accelerate the process of entering various types of data assets into the table.

[0004] Traditional big data mining mainly involves cleaning massive data and annotating it for subsequent information mining by various big data analysis systems. This processing method can perform relatively efficient data processing for data application scenarios in the emerging stage of big data technology. However, with the massive growth of data, especially the diversified development trend of data types and structures, traditional big data mining technology has been unable to meet the needs of today's data elements as the fifth production factor in the future. The main problem is that there are no fixed rules for data elements. For different data sources, data application scenarios and application purposes, the data elements that need to be extracted from the same data may be completely different; at the same time, for data of different structures, their own data characteristics are quite different, and it is impossible to extract various data elements that meet the specified requirements by formulating some simple pre-made rules. Before cleaning the data, it is necessary to use more intelligent methods for efficient extraction and analysis to avoid losing important data elements after data cleaning under simple and unified pre-made rules. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a data element extraction and analysis system based on artificial intelligence, which can effectively overcome the defects of the prior art that it is difficult to accurately and efficiently extract and analyze data elements according to data application scenarios.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] An artificial intelligence-based data element extraction and analysis system, comprising an application scenario management unit, a data element feature training unit, a data element identification unit, a data element management unit, and a quality assessment unit;

[0010] The application scenario management unit classifies and manages the data application scenarios belonging to each specific application field, extracts the data elements that each data application scenario relies on, establishes the association between the data elements and the data application scenarios, and provides training samples to the data element feature training unit;

[0011] The data element feature training unit selects data element samples for model training from the training samples stored in the application scenario management unit, continuously improves the accuracy of the model in identifying data elements through the sample training self-learning process, and optimizes the model according to the comprehensive efficiency value of the model provided by the quality assessment unit;

[0012] The data element recognition unit deploys the model trained and optimized by the data element feature training unit to extract data elements from the input data set, complete the automatic recognition and sorting of multi-source heterogeneous data, and send the extracted data elements to the data element management unit;

[0013] The data element management unit stores and manages the data elements sent by the data element identification unit according to preset rules, and outputs them in a targeted manner according to requirements;

[0014] The quality assessment unit samples the data elements stored in the data element management unit and sends them to the application scenario management unit to verify the validity of the data application scenario corresponding to the data element samples to be tested, obtain the comprehensive efficiency value of the model corresponding to the specific data application scenario, and update it to the corresponding model.

[0015] Preferably, the application scenario management unit includes an application scenario registration module, a data element feature extraction module and a data element verification module;

[0016] The application scenario registration module classifies and registers typical data application scenarios in specific application fields and generates corresponding scenario identifiers;

[0017] The data element feature extraction module extracts the data elements relied on in various data application scenarios, organizes them into data element sets corresponding to various data application scenarios, and associates the scenario identifier with the data element set of the corresponding category data application scenario;

[0018] The data element verification module places the data element samples to be detected sent by the quality assessment unit into the associated data application scenario for in-scenario validity verification, and feeds back to the quality assessment unit the degree of fit between the data element samples to be detected and the associated data application scenario.

[0019] Preferably, the data element feature training unit includes a data element sample extraction module, a sample training module and a training model generation module;

[0020] A data element sample extraction module, which selects data element samples for model training from the training samples stored in the application scenario management unit;

[0021] The sample training module uses the data element samples to extract the overall characteristics of the data element samples through model training;

[0022] The training model generation module continuously optimizes the model parameters according to the model training results of the sample training module, and finally forms a model for data element extraction. Multiple models are trained for the same data application scenario. At the same time, the model comprehensive efficiency value provided by the quality assessment unit is compared with the preset threshold. When the model comprehensive efficiency value is less than the preset threshold, the model is trained for the second time, and the tracking detection is carried out until the model comprehensive efficiency value is not less than the preset threshold.

[0023] Preferably, the data element identification unit includes a data pre-classification module, a data element extraction module and a data element marking module;

[0024] The data pre-classification module divides the corresponding classification identification number according to the characteristic information of the input data set, including the source, storage format and overall size;

[0025] A data element extraction module determines the optimal model according to the classification identification number and the comprehensive efficiency value of the model corresponding to the specific data application scenario, and uses the optimal model to extract data elements from the data in the input data set of the corresponding classification identification number;

[0026] The data element marking module performs auxiliary attribute marking on the data elements extracted by the data element extraction module, provides auxiliary information for the subsequent processing module, and sends the marked data elements to the data element management unit.

[0027] Preferably, the data element management unit includes a data element classification module, a data element storage module and a data element service module;

[0028] The data element classification module calculates the classification identification value corresponding to the data element sent by the data element identification unit according to the corresponding attribute information;

[0029] The data element storage module optimizes the storage and management of data elements according to their classification identification values ​​and comprehensive characteristics, so as to facilitate subsequent retrieval and update;

[0030] The data element service module matches the target data elements in the data element storage area according to the data element feature description information contained in the external request, and packages and outputs it to the external requester.

[0031] Preferably, the data element classification module calculates the classification identification value corresponding to the data element sent by the data element identification unit according to the corresponding attribute information, including:

[0032] The classification identification value corresponding to the data element sent by the data element identification unit is calculated using the following formula:

[0033] CID i =SNID i *(α*RI i +β*SRI i +γ*WI i +δ*TVI i );

[0034] Among them, CID i is the classification identification value of the i-th data element, SNID i is the scene identifier of the i-th data element, RI i is the identification reliability reference value of the i-th data element, SRI i is the scene relevance value of the i-th data element, WI i is the weight value of the i-th data element, TVI i is the timeliness value of the ith data element, α, β, γ, and δ are the weight coefficients of the data element's recognition reliability reference value, scene relevance value, weight value, and timeliness value, respectively. The four weight coefficients α, β, γ, and δ are adjusted as needed to optimize the rationality of the calculation of the data element's classification identification value.

[0035] Preferably, the quality assessment unit includes a data element sampling detection module, a data element quality scoring module and a model efficiency assessment module;

[0036] The data element sampling detection module samples the data elements stored in the data element storage area in the data element management unit to form a sample of the data elements to be detected, and sends the sample of the data elements to be detected to the application scenario management unit to verify the validity of the data application scenario corresponding to the sample of the data elements to be detected, and calculates the validity value of the data element according to the degree of fit fed back by the application scenario management unit;

[0037] The data element quality scoring module calculates the quality score corresponding to the data element based on the validity value of the data element and the corresponding attribute information;

[0038] The model efficiency evaluation module extracts the comprehensive efficiency value of the data element model based on the comprehensive calculation of the quality scores corresponding to the data elements, and updates it to the corresponding model.

[0039] Preferably, the data element quality scoring module calculates the quality score corresponding to the data element according to the validity value of the data element and the corresponding attribute information, including:

[0040] The quality score corresponding to the data element is calculated using the following formula:

[0041] QS i =DSVI i *(α*RI i +β*SRI i +γ*WI i +δ*TVI i );

[0042] Among them, QS i is the quality score of the i-th data element, DSVI i is the validity value of the ith data element, RI i is the identification reliability reference value of the i-th data element, SRI i is the scene relevance value of the i-th data element, WI i is the weight value of the i-th data element, TVI i is the timeliness value of the ith data element, α, β, γ, and δ are the weight coefficients of the data element's recognition reliability reference value, scene relevance value, weight value, and timeliness value, respectively. The four weight coefficients α, β, γ, and δ are adjusted as needed to optimize the rationality of the quality score calculation of the data element.

[0043] (III) Beneficial effects

[0044] Compared with the prior art, the artificial intelligence-based data element extraction and analysis system provided by the present invention adopts reverse thinking and considers from the data application scenario end to extract the data element sets that various data application scenarios depend on to provide training samples for the big model. It uses the sample training self-learning technology of the big model to accurately and efficiently extract the data elements that best match the data in the input data set corresponding to the specific data application scenario, and reversely optimizes the model through dynamic quality assessment to improve model performance, ultimately achieving the purpose of accurate and efficient data element extraction and analysis based on data application scenarios, helping data application parties to more economically and efficiently convert multi-source heterogeneous data into the most matching data elements, and then form data resources, and ultimately form data assets, helping enterprises to add more new development momentum based on data elements, a new type of production factor. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a system schematic diagram of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] A data element extraction and analysis system based on artificial intelligence, such as Figure 1 As shown, it includes an application scenario management unit, a data element feature training unit, a data element identification unit, a data element management unit and a quality assessment unit;

[0049] The application scenario management unit classifies and manages the data application scenarios belonging to each specific application field, extracts the data elements that each data application scenario relies on, establishes the association between the data elements and the data application scenarios, and provides training samples to the data element feature training unit;

[0050] The data element feature training unit selects data element samples for model training from the training samples stored in the application scenario management unit, continuously improves the accuracy of the model in identifying data elements through the sample training self-learning process, and optimizes the model according to the comprehensive efficiency value of the model provided by the quality assessment unit;

[0051] The data element recognition unit deploys the model trained and optimized by the data element feature training unit to extract data elements from the input data set, complete the automatic recognition and sorting of multi-source heterogeneous data, and send the extracted data elements to the data element management unit;

[0052] The data element management unit stores and manages the data elements sent by the data element identification unit according to preset rules, and outputs them in a targeted manner according to requirements;

[0053] The quality assessment unit samples the data elements stored in the data element management unit and sends them to the application scenario management unit to verify the validity of the data application scenario corresponding to the data element samples to be tested, obtain the comprehensive efficiency value of the model corresponding to the specific data application scenario, and update it to the corresponding model.

[0054] ① The application scenario management unit includes an application scenario registration module, a data element feature extraction module and a data element verification module;

[0055] The application scenario registration module classifies and registers typical data application scenarios in specific application fields and generates corresponding scenario identifiers;

[0056] The data element feature extraction module extracts the data elements relied on in various data application scenarios, organizes them into data element sets corresponding to various data application scenarios, and associates the scenario identifier with the data element set of the corresponding category data application scenario;

[0057] The data element verification module places the data element samples to be detected sent by the quality assessment unit into the associated data application scenario for in-scenario validity verification, and feeds back to the quality assessment unit the degree of fit between the data element samples to be detected and the associated data application scenario.

[0058] ②The data element feature training unit includes a data element sample extraction module, a sample training module and a training model generation module;

[0059] A data element sample extraction module, which selects data element samples for model training from the training samples stored in the application scenario management unit;

[0060] The sample training module uses the data element samples to extract the overall characteristics of the data element samples through model training;

[0061] The training model generation module continuously optimizes the model parameters according to the model training results of the sample training module, and finally forms a model for data element extraction. Multiple models are trained for the same data application scenario. At the same time, the model comprehensive efficiency value provided by the quality assessment unit is compared with the preset threshold. When the model comprehensive efficiency value is less than the preset threshold, the model is trained for the second time, and the tracking detection is carried out until the model comprehensive efficiency value is not less than the preset threshold.

[0062] ③The data element identification unit includes a data pre-classification module, a data element extraction module and a data element marking module;

[0063] The data pre-classification module divides the corresponding classification identification number according to the characteristic information of the input data set, including the source, storage format and overall size;

[0064] A data element extraction module determines the optimal model according to the classification identification number and the comprehensive efficiency value of the model corresponding to the specific data application scenario, and uses the optimal model to extract data elements from the data in the input data set of the corresponding classification identification number;

[0065] The data element marking module performs auxiliary attribute marking on the data elements extracted by the data element extraction module, provides auxiliary information for the subsequent processing module, and sends the marked data elements to the data element management unit.

[0066] ④The data element management unit includes a data element classification module, a data element storage module and a data element service module;

[0067] The data element classification module calculates the classification identification value corresponding to the data element sent by the data element identification unit according to the corresponding attribute information;

[0068] The data element storage module optimizes the storage and management of data elements according to their classification identification values ​​and comprehensive characteristics, so as to facilitate subsequent retrieval and update;

[0069] The data element service module matches the target data elements in the data element storage area according to the data element feature description information contained in the external request, and packages and outputs it to the external requester.

[0070] Specifically, the data element classification module calculates the classification identification value corresponding to the data element sent by the data element identification unit according to the corresponding attribute information, including:

[0071] The classification identification value corresponding to the data element sent by the data element identification unit is calculated using the following formula:

[0072] CID i =SNID i *(α*RI i+β*SRI i +γ*WI i +δ*TVI i );

[0073] Among them, CID i is the classification identification value of the i-th data element, SNID i is the scene identifier of the i-th data element, RI i is the identification reliability reference value of the i-th data element, SRI i is the scene relevance value of the i-th data element, WI i is the weight value of the i-th data element, TVI i is the timeliness value of the ith data element, α, β, γ, and δ are the weight coefficients of the data element's recognition reliability reference value, scene relevance value, weight value, and timeliness value, respectively. The four weight coefficients α, β, γ, and δ are adjusted as needed to optimize the rationality of the calculation of the data element's classification identification value.

[0074] ⑤The quality assessment unit includes data element sampling detection module, data element quality scoring module and model efficiency assessment module;

[0075] The data element sampling detection module samples the data elements stored in the data element storage area in the data element management unit to form a sample of the data elements to be detected, and sends the sample of the data elements to be detected to the application scenario management unit to verify the validity of the data application scenario corresponding to the sample of the data elements to be detected, and calculates the validity value of the data element according to the degree of fit fed back by the application scenario management unit;

[0076] The data element quality scoring module calculates the quality score corresponding to the data element based on the validity value of the data element and the corresponding attribute information;

[0077] The model efficiency evaluation module extracts the comprehensive efficiency value of the data element model based on the comprehensive calculation of the quality scores corresponding to the data elements, and updates it to the corresponding model.

[0078] Specifically, the data element quality scoring module calculates the quality score corresponding to the data element according to the validity value of the data element and the corresponding attribute information, including:

[0079] The quality score corresponding to the data element is calculated using the following formula:

[0080] QS i =DSVI i *(α*RI i +β*SRI i +γ*WI i +δ*TVI i );

[0081] Among them, QS i is the quality score of the i-th data element, DSVI i is the validity value of the ith data element, RI i is the identification reliability reference value of the i-th data element, SRI i is the scene relevance value of the i-th data element, WI i is the weight value of the i-th data element, TVI i is the timeliness value of the ith data element, α, β, γ, and δ are the weight coefficients of the data element's recognition reliability reference value, scene relevance value, weight value, and timeliness value, respectively. The four weight coefficients α, β, γ, and δ are adjusted as needed to optimize the rationality of the quality score calculation of the data element.

[0082] The technical solution of this application adopts reverse thinking and considers the data application scenario side. It extracts the data element sets that various data application scenarios depend on to provide training samples for the big model. It uses the sample training self-learning technology of the big model to accurately and efficiently extract the data elements that best match the data in the input data set corresponding to the specific data application scenario, and reversely optimizes the model through dynamic quality assessment to improve model performance, and ultimately achieves the purpose of accurate and efficient data element extraction and analysis based on data application scenarios, helping data application parties to more economically and efficiently convert multi-source heterogeneous data into the most matching data elements, and then form data resources, and ultimately form data assets, helping enterprises to add more new development momentum based on data elements, a new type of production factor.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data element extraction and analysis system based on artificial intelligence, characterized in that: It includes an application scenario management unit, a data element feature training unit, a data element identification unit, a data element management unit and a quality assessment unit; The application scenario management unit classifies and manages the data application scenarios belonging to each specific application field, extracts the data elements that each data application scenario relies on, establishes the association between the data elements and the data application scenarios, and provides training samples to the data element feature training unit; The data element feature training unit selects data element samples for model training from the training samples stored in the application scenario management unit, continuously improves the accuracy of the model in identifying data elements through the sample training self-learning process, and optimizes the model according to the comprehensive efficiency value of the model provided by the quality assessment unit; The data element recognition unit deploys the model trained and optimized by the data element feature training unit to extract data elements from the input data set, complete the automatic recognition and sorting of multi-source heterogeneous data, and send the extracted data elements to the data element management unit; The data element management unit stores and manages the data elements sent by the data element identification unit according to preset rules, and outputs them in a targeted manner according to requirements; The quality assessment unit samples the data elements stored in the data element management unit and sends them to the application scenario management unit to verify the validity of the data application scenario corresponding to the data element samples to be tested, obtain the comprehensive efficiency value of the model corresponding to the specific data application scenario, and update it to the corresponding model.

2. The data element extraction and analysis system based on artificial intelligence according to claim 1 is characterized in that: The application scenario management unit includes an application scenario registration module, a data element feature extraction module and a data element verification module; The application scenario registration module classifies and registers typical data application scenarios in specific application fields and generates corresponding scenario identifiers; The data element feature extraction module extracts the data elements relied on in various data application scenarios, organizes them into data element sets corresponding to various data application scenarios, and associates the scenario identifier with the data element set of the corresponding category data application scenario; The data element verification module places the data element samples to be detected sent by the quality assessment unit into the associated data application scenario for in-scenario validity verification, and feeds back to the quality assessment unit the degree of fit between the data element samples to be detected and the associated data application scenario.

3. The data element extraction and analysis system based on artificial intelligence according to claim 2 is characterized in that: The data element feature training unit includes a data element sample extraction module, a sample training module and a training model generation module; A data element sample extraction module, which selects data element samples for model training from the training samples stored in the application scenario management unit; The sample training module uses the data element samples to extract the overall characteristics of the data element samples through model training; The training model generation module continuously optimizes the model parameters according to the model training results of the sample training module, and finally forms a model for data element extraction. Multiple models are trained for the same data application scenario. At the same time, the model comprehensive efficiency value provided by the quality assessment unit is compared with the preset threshold. When the model comprehensive efficiency value is less than the preset threshold, the model is trained for the second time, and the tracking detection is carried out until the model comprehensive efficiency value is not less than the preset threshold.

4. The data element extraction and analysis system based on artificial intelligence according to claim 3 is characterized in that: The data element identification unit includes a data pre-classification module, a data element extraction module and a data element marking module; The data pre-classification module divides the corresponding classification identification number according to the characteristic information of the input data set, including the source, storage format and overall size; A data element extraction module determines the optimal model according to the classification identification number and the comprehensive efficiency value of the model corresponding to the specific data application scenario, and uses the optimal model to extract data elements from the data in the input data set of the corresponding classification identification number; The data element marking module performs auxiliary attribute marking on the data elements extracted by the data element extraction module, provides auxiliary information for the subsequent processing module, and sends the marked data elements to the data element management unit.

5. The data element extraction and analysis system based on artificial intelligence according to claim 4 is characterized in that: The data element management unit includes a data element classification module, a data element storage module and a data element service module; The data element classification module calculates the classification identification value corresponding to the data element sent by the data element identification unit according to the corresponding attribute information; The data element storage module optimizes the storage and management of data elements according to their classification identification values ​​and comprehensive characteristics, so as to facilitate subsequent retrieval and update; The data element service module matches the target data elements in the data element storage area according to the data element feature description information contained in the external request, and packages and outputs it to the external requester.

6. The data element extraction and analysis system based on artificial intelligence according to claim 5 is characterized in that: The data element classification module calculates the classification identification value corresponding to the data element sent by the data element identification unit according to the corresponding attribute information, including: The classification identification value corresponding to the data element sent by the data element identification unit is calculated using the following formula: CID i =SNID i *(α*RI i +β*SRI i +γ*WI i +δ*TVI i ); Among them, CID i is the classification identification value of the i-th data element, SNID i is the scene identifier of the i-th data element, RI i is the identification reliability reference value of the i-th data element, SRI i is the scene relevance value of the i-th data element, WI i is the weight value of the i-th data element, TVI i is the timeliness value of the ith data element, α, β, γ, and δ are the weight coefficients of the data element's recognition reliability reference value, scene relevance value, weight value, and timeliness value, respectively. The four weight coefficients α, β, γ, and δ are adjusted as needed to optimize the rationality of the calculation of the data element's classification identification value.

7. The data element extraction and analysis system based on artificial intelligence according to claim 5 is characterized in that: The quality assessment unit includes a data element sampling detection module, a data element quality scoring module and a model efficiency assessment module; The data element sampling detection module samples the data elements stored in the data element storage area in the data element management unit to form a sample of the data elements to be detected, and sends the sample of the data elements to be detected to the application scenario management unit to perform in-scenario validity verification on the data application scenario corresponding to the sample of the data elements to be detected, and calculates the validity value of the data element according to the degree of fit fed back by the application scenario management unit; The data element quality scoring module calculates the quality score corresponding to the data element based on the validity value of the data element and the corresponding attribute information; The model efficiency evaluation module extracts the comprehensive efficiency value of the data element model based on the comprehensive calculation of the quality scores corresponding to the data elements, and updates it to the corresponding model.

8. The data element extraction and analysis system based on artificial intelligence according to claim 7 is characterized in that: The data element quality scoring module calculates the quality score corresponding to the data element according to the validity value of the data element and the corresponding attribute information, including: The quality score corresponding to the data element is calculated using the following formula: QS i =DSVI i *(a*RI i +β*SRI i +γ*WI i +δ*TVI i ); Among them, QS i is the quality score of the i-th data element, DSVI i is the validity value of the ith data element, RI i is the identification reliability reference value of the i-th data element, SRI i is the scene relevance value of the i-th data element, WI i is the weight value of the i-th data element, TVI i is the timeliness value of the ith data element, α, β, γ, and δ are the weight coefficients of the data element's recognition reliability reference value, scene relevance value, weight value, and timeliness value, respectively. The four weight coefficients α, β, γ, and δ are adjusted as needed to optimize the rationality of the quality score calculation of the data element.

Images