AI data intelligent analysis and processing method and device based on large model

Abstract

The invention relates to the technical field of data processing, and discloses an AI data intelligent analysis and processing method and device based on a large model, the method and device are applied to an AI data governance platform, and the method comprises the following steps: obtaining at least one piece of initial data to be processed; based on an AI data cleaning and integrating technology, cleaning and integrating the data to obtain target data; based on a data identification and classification technology, identifying and classifying the target data to obtain a data field and a data set of the target data; based on a data security and access control technology, matching the data set of each data field with a platform user to obtain a matched target user; and based on a data analysis and mining technology, analyzing the data set of each data field to obtain potential value information of the field, and outputting the potential value information to a data display interface of the target user for the target user to view the potential value information. Therefore, by implementing the method, the data processing efficiency and quality can be improved, and the data security can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for intelligent analysis and processing of AI data based on a large model. Background Art

[0002] With the rapid development of information technology, the generation, storage, processing and analysis of data have become increasingly important, and the complexity and challenges of data governance have increased.

[0003] At present, traditional data governance tools and methods are unable to cope with challenges such as inconsistent data quality, information islands, privacy protection, and compliance requirements. New technologies are urgently needed to improve the efficiency and quality of data processing. The solutions in existing technologies are mainly focused on data cleaning, desensitization, and archiving. However, practice has found that these traditional solutions still have many shortcomings when facing complex data types and business scenarios. For example, the more complex the data types and business scenarios, the lower the efficiency and quality of data processing, the higher the enterprise management costs, and the lower the data security. It can be seen that it is particularly important to propose a technical solution to improve the efficiency and quality of data processing in the case of complex data types and business scenarios. Summary of the invention The present invention provides a method and device for intelligent analysis and processing of AI data based on a large model, which can improve data processing efficiency and quality in complex data types and business scenarios, and is conducive to improving data security.

[0004] In order to solve the above technical problems, the first aspect of the present invention discloses an AI data intelligent analysis and processing method based on a large model, which is applied to an AI data governance platform and includes: Acquire at least one initial data to be processed, where the initial data includes structured data or unstructured data; Based on AI data cleaning and integration technology, each of the initial data is cleaned and integrated to obtain the target data corresponding to each of the initial data; Based on data recognition and classification technology, all the target data are identified and classified to obtain data recognition and classification results, wherein the data recognition and classification results include the data field of each target data and the data set corresponding to each data field, and all the target data in the same data set correspond to the same data field; Based on data security and access control technology, the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform to obtain a target user that matches each data field; Based on data analysis and mining technology, the data set corresponding to each of the data fields is analyzed to obtain the potential value information of each of the data fields. The potential value information is used to be output to the data display interface corresponding to the target user, so that the target user can view the potential value information according to the display results of the data display interface.

[0005] As an optional implementation, in the first aspect of the present invention, the AI-based data cleaning and integration technology is used to clean and integrate each of the initial data to obtain target data corresponding to each of the initial data, including: Input each of the initial data into the pre-trained data analysis model of the AI ​​data governance platform for analysis to obtain the data content of one or more sub-data contained in the initial data and the data format of each of the sub-data, wherein the data analysis model is obtained by training with a preset deep learning algorithm; For each of the initial data, judging whether there is any preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data; When it is determined that the abnormal sub-data does not exist in all the sub-data of the initial data, all the sub-data of the initial data are integrated based on the AI ​​data cleaning and integration technology to obtain the target data corresponding to the initial data; When it is determined that the abnormal sub-data exists in all the sub-data of the initial data, the abnormal sub-data of the initial data is cleaned based on the AI ​​data cleaning and integration technology, and the operation based on the AI ​​data cleaning and integration technology is triggered to integrate all the sub-data of the initial data to obtain the target data corresponding to the initial data, in which all the sub-data are the remaining sub-data except the abnormal sub-data.

[0006] As an optional implementation manner, in the first aspect of the present invention, for each of the initial data, judging whether there is preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data includes: For each of the initial data, judging whether the data content of each of the sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all the sub-data of the initial data, and / or judging whether the data format of each of the sub-data of the initial data is inconsistent with the data formats of other sub-data according to the data formats of all the sub-data of the initial data; When it is determined that the data format of the sub-data does not satisfy the data abnormality condition, and the data format of the sub-data is consistent with the data format of other sub-data, determining that the sub-data is normal sub-data; When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined as the first sub-data; When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined as the second sub-data; The abnormal sub-data includes the first sub-data and / or the second sub-data.

[0007] As an optional implementation manner, in the first aspect of the present invention, for each of the initial data, judging whether the data content of each of the sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all the sub-data of the initial data includes: For each sub-data of each of the initial data, judging whether the data content of the sub-data is erroneous according to the data content of the sub-data, which is recorded as the first condition; and / or, For each sub-data of each of the initial data, judging whether the sub-data belongs to the preset irrelevant data according to the data content of the sub-data, which is recorded as the second condition; and / or, For each of the initial data, according to the data contents of all the sub-data of the initial data, calculate the content similarity between each of the sub-data of the initial data, and determine whether the content similarity between each of the sub-data of the initial data is greater than or equal to a preset similarity, which is recorded as the third condition; When at least one of the first condition, the second condition and the third condition is satisfied, it is determined that the data content of the corresponding sub-data satisfies a preset data abnormality condition.

[0008] As an optional implementation, in the first aspect of the present invention, the AI ​​data cleaning and integration technology includes a technology for cleaning and integrating data based on natural language processing and / or image recognition, and the data recognition and classification technology includes a technology for recognizing and classifying data based on a convolutional neural network and / or a recurrent neural network; And, based on the data recognition and classification technology, all the target data are identified and classified to obtain data recognition and classification results, including: For each of the target data, analyzing the data source of the target data, wherein the data source includes natural language technology or image recognition technology; For the first target data group whose data source is the natural language technology, based on the technology of data recognition and classification by the convolutional neural network, each target data in the first target data group is recognized to obtain the data field of each target data in the first target data group; For the second target data group whose data source is the image recognition technology, based on the technology of identifying and classifying data by the recurrent neural network, each target data in the second target data group is identified to obtain the data field of each target data in the second target data group; According to the data fields of all target data within the first target data and the data fields of all target data within the second target data, all the target data are classified to obtain a data set corresponding to each data field, and all the first target data and all the second target data in the same data set correspond to the same data field.

[0009] As an optional implementation, in the first aspect of the present invention, the data analysis and mining technology includes cluster analysis technology and association rule mining technology; Furthermore, the data analysis and mining technology is used to analyze the data set corresponding to each data field to obtain the potential value information of each data field, including: Based on the cluster analysis technology, cluster analysis is performed on the data set corresponding to each of the data fields to obtain multiple data item sets corresponding to each of the data fields; For each of the data item sets corresponding to each of the data fields, calculating the support of the data item set according to the transaction data set corresponding to the data item set; For each of the data fields, the data item set whose support is greater than or equal to the preset support is selected from all the data item sets corresponding to the data field as the target data item set corresponding to the data field, and the association rule information corresponding to the data field is generated according to all the data item sets corresponding to the data field; Based on the association rule mining technology, according to the target data item set corresponding to each data field, analyzing the association relationship between the target data item sets corresponding to each data field; According to the association relationship corresponding to each of the data fields, the potential value information of each of the data fields is determined.

[0010] As an optional implementation, in the first aspect of the present invention, based on the data security and access control technology, matching the data set corresponding to each of the data fields with at least one user corresponding to the AI ​​data governance platform to obtain a target user matching each of the data fields includes: Determine first access control information for each of the data fields based on data security and access control technology; When the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, according to the data access request of each user, determine the second access control information of each user; According to the first access control information of all the data fields and the second access control information of all the users, all the data fields are matched with all the users to obtain one or more matching groups, each of which includes a data field and a target user matching the data field.

[0011] The second aspect of the present invention discloses an AI data intelligent analysis and processing device based on a large model, the device is applied to an AI data governance platform, and the device includes: A data acquisition module, used to acquire at least one initial data to be processed, wherein the initial data includes structured data or unstructured data; A data cleaning module, used to clean and integrate each of the initial data based on AI data cleaning and integration technology to obtain target data corresponding to each of the initial data; A data classification module, used to identify and classify all the target data based on data identification and classification technology to obtain data identification and classification results, wherein the data identification and classification results include the data field of each target data and the data set corresponding to each data field, and all target data in the same data set correspond to the same data field; An access control module, used to match the data set corresponding to each of the data fields with at least one user corresponding to the AI ​​data governance platform based on data security and access control technology, to obtain a target user that matches each of the data fields; The value analysis module is used to analyze the data set corresponding to each data field based on data analysis and mining technology to obtain the potential value information of each data field. The potential value information is used to output to the data display interface corresponding to the target user so that the target user can view the potential value information according to the display results of the data display interface.

[0012] As an optional implementation, in the second aspect of the present invention, the data cleaning module cleans and integrates each of the initial data based on AI data cleaning and integration technology, and the method of obtaining the target data corresponding to each of the initial data specifically includes: Input each of the initial data into the pre-trained data analysis model of the AI ​​data governance platform for analysis to obtain the data content of one or more sub-data contained in the initial data and the data format of each of the sub-data, wherein the data analysis model is obtained by training with a preset deep learning algorithm; For each of the initial data, judging whether there is any preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data; When it is determined that the abnormal sub-data does not exist in all the sub-data of the initial data, all the sub-data of the initial data are integrated based on the AI ​​data cleaning and integration technology to obtain the target data corresponding to the initial data; When it is determined that the abnormal sub-data exists in all the sub-data of the initial data, the abnormal sub-data of the initial data is cleaned based on the AI ​​data cleaning and integration technology, and the operation based on the AI ​​data cleaning and integration technology is triggered to integrate all the sub-data of the initial data to obtain the target data corresponding to the initial data, in which all the sub-data are the remaining sub-data except the abnormal sub-data.

[0013] As an optional implementation, in the second aspect of the present invention, for each of the initial data, the data cleaning module determines whether there is a preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data, specifically including: For each of the initial data, judging whether the data content of each of the sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all the sub-data of the initial data, and / or judging whether the data format of each of the sub-data of the initial data is inconsistent with the data formats of other sub-data according to the data formats of all the sub-data of the initial data; When it is determined that the data format of the sub-data does not satisfy the data abnormality condition, and the data format of the sub-data is consistent with the data format of other sub-data, determining that the sub-data is normal sub-data; When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined as the first sub-data; When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined as the second sub-data; The abnormal sub-data includes the first sub-data and / or the second sub-data.

[0014] As an optional implementation, in the second aspect of the present invention, for each of the initial data, the data cleaning module determines whether the data content of each of the sub-data of the initial data satisfies the preset data abnormality condition according to the data content of all the sub-data of the initial data, specifically including: For each sub-data of each of the initial data, judging whether the data content of the sub-data is erroneous according to the data content of the sub-data, which is recorded as the first condition; and / or, For each sub-data of each of the initial data, judging whether the sub-data belongs to the preset irrelevant data according to the data content of the sub-data, which is recorded as the second condition; and / or, For each of the initial data, according to the data contents of all the sub-data of the initial data, calculate the content similarity between each of the sub-data of the initial data, and determine whether the content similarity between each of the sub-data of the initial data is greater than or equal to a preset similarity, which is recorded as the third condition; When at least one of the first condition, the second condition and the third condition is satisfied, it is determined that the data content of the corresponding sub-data satisfies a preset data abnormality condition.

[0015] As an optional implementation, in the second aspect of the present invention, the AI ​​data cleaning and integration technology includes a technology for cleaning and integrating data based on natural language processing and / or image recognition, and the data recognition and classification technology includes a technology for recognizing and classifying data based on a convolutional neural network and / or a recurrent neural network; Furthermore, the data classification module identifies and classifies all the target data based on data identification and classification technology, and the method of obtaining the data identification and classification results specifically includes: For each of the target data, analyzing the data source of the target data, wherein the data source includes natural language technology or image recognition technology; For the first target data group whose data source is the natural language technology, based on the technology of data recognition and classification by the convolutional neural network, each target data in the first target data group is recognized to obtain the data field of each target data in the first target data group; For the second target data group whose data source is the image recognition technology, based on the technology of identifying and classifying data by the recurrent neural network, each target data in the second target data group is identified to obtain the data field of each target data in the second target data group; According to the data fields of all target data within the first target data and the data fields of all target data within the second target data, all the target data are classified to obtain a data set corresponding to each data field, and all the first target data and all the second target data in the same data set correspond to the same data field.

[0016] As an optional implementation, in the second aspect of the present invention, the data analysis and mining technology includes cluster analysis technology and association rule mining technology; Furthermore, the value analysis module analyzes the data set corresponding to each of the data fields based on data analysis and mining technology, and the method of obtaining the potential value information of each of the data fields specifically includes: Based on the cluster analysis technology, cluster analysis is performed on the data set corresponding to each of the data fields to obtain multiple data item sets corresponding to each of the data fields; For each of the data item sets corresponding to each of the data fields, calculating the support of the data item set according to the transaction data set corresponding to the data item set; For each of the data fields, the data item set whose support is greater than or equal to the preset support is selected from all the data item sets corresponding to the data field as the target data item set corresponding to the data field, and the association rule information corresponding to the data field is generated according to all the data item sets corresponding to the data field; Based on the association rule mining technology, according to the target data item set corresponding to each data field, analyzing the association relationship between the target data item sets corresponding to each data field; According to the association relationship corresponding to each of the data fields, the potential value information of each of the data fields is determined.

[0017] As an optional implementation, in the second aspect of the present invention, the access control module matches the data set corresponding to each data field with at least one user corresponding to the AI ​​data governance platform based on data security and access control technology, and the method of obtaining the target user matching each data field specifically includes: Determine first access control information for each of the data fields based on data security and access control technology; When the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, according to the data access request of each user, determine the second access control information of each user; According to the first access control information of all the data fields and the second access control information of all the users, all the data fields are matched with all the users to obtain one or more matching groups, each of which includes a data field and a target user matching the data field.

[0018] The third aspect of the present invention discloses another AI data intelligent analysis and processing device based on a large model, the device comprising: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the AI ​​data intelligent analysis and processing method based on a large model disclosed in the first aspect of the present invention.

[0019] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the AI ​​data intelligent analysis and processing method based on a large model disclosed in the first aspect of the present invention.

[0020] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In an embodiment of the present invention, at least one initial data to be processed is obtained, and the initial data includes structured data or unstructured data; based on the AI ​​data cleaning and integration technology of the AI ​​data governance platform, each initial data is cleaned and integrated to obtain the target data corresponding to each initial data; based on the data identification and classification technology of the AI ​​data governance platform, all target data are identified and classified to obtain data identification and classification results, and the data identification and classification results include the data field of each target data, and also include the data set corresponding to each data field, and all target data in the same data set correspond to the same data field; based on the data security and access control technology of the AI ​​data governance platform, the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform to obtain the target user matching each data field; based on the data analysis and mining technology of the AI ​​data governance platform, the data set corresponding to each data field is analyzed to obtain the potential value information of each data field, and the potential value information is used to output to the data display interface corresponding to the target user, so that the target user can view the potential value information according to the display result of the data display interface. It can be seen that the implementation of the present invention can build an AI data governance platform, and obtain at least one initial data to be processed, such as structured data or unstructured data, through the platform, and based on the AI ​​data cleaning and integration technology of the AI ​​data governance platform, clean and integrate each initial data to obtain the target data corresponding to each initial data, which can improve the cleaning accuracy and integration accuracy of the initial data, and is conducive to improving the accuracy and efficiency of the target data obtained by cleaning and integration. Subsequently, based on the data recognition and classification technology of the AI ​​data governance platform, all target data are identified and classified to obtain the data field of each target data and the data set corresponding to each data field, which can improve the recognition accuracy and classification accuracy of each target data. Then, based on the data security and access control technology of the AI ​​data governance platform, the data set corresponding to each data field and the AI ​​data At least one user corresponding to the governance platform is matched to obtain the target user matching each data field, which can improve the matching accuracy and efficiency between the data field and the platform users, and the data analysis and mining technology based on the AI ​​data governance platform is used to analyze the data set corresponding to each data field to obtain the potential value information of each data field. The data processing efficiency and quality can be improved through a series of data processing methods, thereby improving the analysis accuracy and efficiency of the potential value of each data field. By outputting the potential value of the data to the data display interface corresponding to the target user for the user to view, it is beneficial to improve data security, and to reduce the situation where users unrelated to the data access the data and cause the data to be leaked, thereby reducing data security, which is beneficial to improving users' trust, satisfaction and usage experience with the AI ​​data governance platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flowchart of a method for intelligent analysis and processing of AI data based on a large model disclosed in an embodiment of the present invention; Figure 2 It is a flowchart of another AI data intelligent analysis and processing method based on a large model disclosed in an embodiment of the present invention; Figure 3 It is a structural schematic diagram of an AI data intelligent analysis and processing device based on a large model disclosed in an embodiment of the present invention; Figure 4 It is a structural schematic diagram of another AI data intelligent analysis and processing device based on a large model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0025] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0026] The present invention discloses a method and device for intelligent analysis and processing of AI data based on a large model, which can build an AI data governance platform, and obtain at least one initial data to be processed, such as structured data or unstructured data, through the platform, and based on the AI ​​data cleaning and integration technology of the AI ​​data governance platform, clean and integrate each initial data to obtain the target data corresponding to each initial data, which can improve the cleaning accuracy and integration accuracy of the initial data, and is conducive to improving the accuracy and efficiency of the target data obtained by cleaning and integration. Subsequently, based on the data recognition and classification technology of the AI ​​data governance platform, all target data are identified and classified to obtain the data field of each target data and the data set corresponding to each data field, which can improve the recognition accuracy and classification accuracy of each target data, and then based on the data security and access control technology of the AI ​​data governance platform, each data field is classified into a corresponding group of data. The corresponding data set is matched with at least one user corresponding to the AI ​​data governance platform to obtain the target user that matches each data field, which can improve the matching accuracy and efficiency between the data field and the platform user, and the data analysis and mining technology based on the AI ​​data governance platform is used to analyze the data set corresponding to each data field to obtain the potential value information of each data field, which can improve the data processing efficiency and quality through a series of data processing methods, thereby improving the analysis accuracy and efficiency of the potential value of each data field, and outputting the potential value of the data to the data display interface corresponding to the target user for the user to view is conducive to improving data security, and is conducive to reducing the occurrence of data leakage caused by users unrelated to the data accessing the data, thereby reducing the occurrence of data security reduction, and thus helping to improve the user's trust, satisfaction and user experience of the AI ​​data governance platform. The following are detailed descriptions.

[0027] Embodiment 1 See also Figure 1 , Figure 1 This is a flow chart of a method for intelligent analysis and processing of AI data based on a large model disclosed in an embodiment of the present invention. Figure 1The described AI data intelligent analysis and processing method based on a large model can be applied to an AI data governance platform, wherein the AI ​​data governance platform can integrate multiple data processing technologies, including one or more combinations of AI data cleaning and integration technologies, data identification and classification technologies, data security and access control technologies, and data analysis and mining technologies, and the AI ​​data intelligent analysis and processing method based on a large model can also be applied to an AI data intelligent analysis and processing device based on a large model, wherein the device may include a data processing device or a data processing server, wherein the data processing server may include a cloud server or a local server, which is not limited in the embodiments of the present invention. Figure 1 As shown, the AI ​​data intelligent analysis and processing method based on the big model may include the following operations: 101. Obtain at least one initial data to be processed.

[0028] In the embodiment of the present invention, the initial data may include structured data or unstructured data. Among them, structured data has a fixed format and structure, and is usually stored in a relational database, such as an SQL database, and each piece of data has clear fields and definitions, and the format is fixed, which is easy to manage and analyze; unstructured data has no predefined data model or structure, and the format is diverse and irregular, and unstructured data may include data in various forms such as text, images, audio, and video, and such data is difficult to represent with a unified two-dimensional logical table, and is usually stored in a file system or a NoSQL database.

[0029] In an embodiment of the present invention, optionally, the initial data may be enterprise data uploaded to the platform by platform users of the AI ​​data governance platform, and the enterprise data is customer-related data collected by the enterprise where the corresponding platform user is located. Among them, the enterprise data may include one or more combinations of basic customer information (such as gender, age, etc.), purchasing behavior, preference information, etc., which are not limited in the embodiment of the present invention. In this way, through the analysis and mining of customer data uploaded by the enterprise by the AI ​​data governance platform, the enterprise can better understand customer needs, provide personalized products and services, and thus improve customer satisfaction.

[0030] 102. Based on AI data cleaning and integration technology, each initial data is cleaned and integrated to obtain the target data corresponding to each initial data.

[0031] In an embodiment of the present invention, the AI ​​data cleaning and integration technology may include a technology for cleaning and integrating data based on natural language processing (NLP) and / or image recognition. Specifically, based on the AI ​​data cleaning and integration technology, errors and / or inconsistencies in each initial data are corrected (such as removing duplicate, erroneous or irrelevant data), and then the corrected initial data is integrated to obtain the target data corresponding to the initial data.

[0032] 103. Based on data recognition and classification technology, all target data are identified and classified to obtain data recognition and classification results.

[0033] In an embodiment of the present invention, the data identification and classification technology may include a technology for identifying and classifying data based on a convolutional neural network (CNN) and / or a recurrent neural network (RNN). The data identification and classification results may include a data field for each target data, and may also include a data set corresponding to each data field. All target data in the same data set correspond to the same data field, wherein the data field is used to indicate the industry to which the corresponding target data belongs.

[0034] Specifically, based on data recognition and classification technology, each target data is identified to obtain the data field of each target data, and all target data are classified according to the data fields of all target data to obtain the data set corresponding to each data field.

[0035] 104. Based on data security and access control technology, the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform to obtain the target user matching each data field.

[0036] In an embodiment of the present invention, specifically, at least one user corresponding to the AI ​​data governance platform may be a user who has signed a contract with the AI ​​data governance platform, or may be a user who is new to the AI ​​data governance platform, and the embodiment of the present invention does not limit this.

[0037] 105. Based on data analysis and mining technology, analyze the data set corresponding to each data field to obtain the potential value information of each data field.

[0038] In the embodiment of the present invention, optionally, there is no order of precedence between step 105 and step 104, that is, step 205 may occur before step 204, may occur after step 204, or may occur simultaneously with step 204, which is not limited in the embodiment of the present invention.

[0039] In an embodiment of the present invention, data analysis and mining technology may include cluster analysis technology and association rule mining technology, and potential value information is used to output to the data display interface corresponding to the target user, so that the target user can view the potential value information according to the display results of the data display interface.

[0040] It can be seen that implementation Figure 1 The described AI data intelligent analysis and processing method based on large models can build an AI data governance platform, and obtain at least one initial data to be processed, such as structured data or unstructured data, through the platform, and based on the AI ​​data cleaning and integration technology of the AI ​​data governance platform, clean and integrate each initial data to obtain the target data corresponding to each initial data, which can improve the cleaning accuracy and integration accuracy of the initial data, and is conducive to improving the accuracy and efficiency of the target data obtained by cleaning and integration. Subsequently, based on the data recognition and classification technology of the AI ​​data governance platform, all target data are identified and classified to obtain the data field of each target data and the data set corresponding to each data field, which can improve the recognition accuracy and classification accuracy of each target data, and then based on the data security and access control technology of the AI ​​data governance platform, the data corresponding to each data field is classified. By matching the data set with at least one user corresponding to the AI ​​data governance platform to obtain the target user matching each data field, the matching accuracy and efficiency between the data field and the platform users can be improved. Based on the data analysis and mining technology of the AI ​​data governance platform, the data set corresponding to each data field is analyzed to obtain the potential value information of each data field. The data processing efficiency and quality can be improved through a series of data processing methods, thereby improving the analysis accuracy and efficiency of the potential value of each data field. By outputting the potential value of the data to the data display interface corresponding to the target user for the user to view, it is beneficial to improve data security, reduce the situation where users unrelated to the data access the data and cause the data to be leaked, thereby reducing data security, and further improve users' trust, satisfaction and usage experience with the AI ​​data governance platform.

[0041] In an optional embodiment, the above step 103, based on the data recognition and classification technology, identifies and classifies all target data to obtain the data recognition and classification results, which may include: For each target data, analyzing the data source of the target data; For the first target data group whose data source is natural language technology, each target data in the first target data group is identified based on the technology of data recognition and classification by convolutional neural network, and the data field of each target data in the first target data group is obtained; For the second target data group whose data source is the image recognition technology, each target data in the second target data group is identified based on the technology of identifying and classifying data by a recurrent neural network, and the data field of each target data in the second target data group is obtained; According to the data fields of all target data in the first target data and the data fields of all target data in the second target data, all target data are classified to obtain a data set corresponding to each data field.

[0042] In the embodiment of the present invention, optionally, the data source may include natural language technology or image recognition technology. Specifically, the technology of identifying and classifying data based on convolutional neural networks can identify target data whose data source is natural language technology, and the technology of identifying and classifying data based on recurrent neural networks can identify target data whose data source is image recognition technology. Optionally, the first target data group is empty or the data source of all target data in the first target data group is natural language technology, the second target data group is empty or the data source of all target data in the second target data group is image recognition technology, and the first target data group and the second target data group are not empty at the same time, that is, when the first target data group is empty, the second target data group is not empty, and when the second target data group is empty, the first target data group is not empty, and the embodiment of the present invention does not limit it.

[0043] It can be seen that this optional embodiment can analyze the data source of each target data, and for the first target data group whose data source is the natural language technology, based on the technology of identifying and classifying data using a convolutional neural network, identify each target data in the first target data group, and obtain the data field of each target data in the first target data group, which can improve the recognition accuracy and reliability of the data field of the target data obtained by processing with the natural language technology, and for the second target data group whose data source is the image recognition technology, based on the technology of identifying and classifying data using a recurrent neural network, identify each target data in the second target data group, and obtain the data field of each target data in the second target data group, which can improve the recognition accuracy and reliability of the data field of the target data obtained by processing with the image recognition technology, and can improve the diversity and flexibility of the identification method of the data field of the target data based on the diversified data sources, and then classify all the target data according to the data fields of all the target data in the first target data and the data fields of all the target data in the second target data, and obtain the data set corresponding to each data field, which can improve the classification accuracy and reliability of all the target data based on the data fields of all the target data obtained by accurate identification.

[0044] In another optional embodiment, the above step 105 analyzes the data set corresponding to each data field based on data analysis and mining technology to obtain the potential value information of each data field, which may include: Based on cluster analysis technology, cluster analysis is performed on the data set corresponding to each data field to obtain multiple data item sets corresponding to each data field; For each data item set corresponding to each data field, the support of the data item set is calculated according to the transaction data set corresponding to the data item set; For each data domain, a data item set whose support is greater than or equal to a preset support is selected from all data item sets corresponding to the data domain as a target data item set corresponding to the data domain, and association rule information corresponding to the data domain is generated based on all data item sets corresponding to the data domain; Based on the association rule mining technology, according to the target data item set corresponding to each data field, the association relationship between the target data item sets corresponding to each data field is analyzed; According to the corresponding association relationship of each data field, the potential value information of each data field is determined.

[0045] In the embodiment of the present invention, the target data item set is a frequent item set in association rule mining.

[0046] It can be seen that this optional embodiment can perform cluster analysis on the data set corresponding to each data field based on the cluster analysis technology, obtain multiple data item sets corresponding to each data field, and accurately calculate the support of the data item set according to the transaction data set corresponding to each data item set, and for each data field, screen out the data item set with a support greater than or equal to the preset support from all the data item sets corresponding to the data field as the target data item set corresponding to the data field, which can improve the screening accuracy and reliability of the target data item set, and accurately generate the association rule information corresponding to the data field according to all the data item sets corresponding to the data field, and then analyze the association relationship between the target data item sets corresponding to each data field based on the association rule mining technology according to the target data item set corresponding to each data field, which can improve the analysis accuracy of the association relationship between the target data item sets corresponding to each data field, and then determine the potential value information of each data field according to the association relationship corresponding to each data field, which is conducive to improving the analysis accuracy and reliability of the potential value of each data field.

[0047] In another optional embodiment, the above step 104 matches the data set corresponding to each data field with at least one user corresponding to the AI ​​data governance platform based on data security and access control technology to obtain the target user matching each data field, which may include: Based on data security and access control technology, determine the first access control information of each data field; When the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, the second access control information of each user is determined according to the data access request of each user; According to the first access control information of all data fields and the second access control information of all users, all data fields are matched with all users to obtain one or more matching groups, each of which includes a data field and a target user matching the data field.

[0048] In the embodiment of the present invention, the data security and access control technology may be a data processing technology including multiple encryption algorithms (such as symmetric encryption, asymmetric encryption, etc.) and access control strategies (such as role access control, attribute access control, etc.). Optionally, the first access control information may be an access control table, and the table stores multiple user identifiers and the permission status of each user identifier, and the permission status includes an authorized state or an unauthorized state; the second access control information may include the data that the user can access and the operations that the user can perform on the data. Optionally, based on the data security and access control technology, it is also possible to detect whether the compliance of each target data meets the standard (such as whether the compliance is greater than or equal to the preset compliance), and when it is detected that the compliance of each target data meets the standard, the above-mentioned data security and access control technology is executed, and the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform, and the operation of the target user matching each data field is obtained, which is not limited in the embodiment of the present invention.

[0049] It can be seen that this optional embodiment can accurately determine the first access control information of each data field based on data security and access control technology, and when the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, accurately determine the second access control information of each user according to the data access request of each user, and then match all data fields with all users according to the first access control information of all data fields and the second access control information of all users to obtain one or more matching groups, which can improve the matching accuracy and reliability between data fields and target users.

[0050] Embodiment 2 See also Figure 2 , Figure 2 This is a flow chart of a method for intelligent analysis and processing of AI data based on a large model disclosed in an embodiment of the present invention. Figure 2The described AI data intelligent analysis and processing method based on a large model can be applied to an AI data governance platform, wherein the AI ​​data governance platform can integrate multiple data processing technologies, including one or more combinations of AI data cleaning and integration technologies, data identification and classification technologies, data security and access control technologies, and data analysis and mining technologies, and the AI ​​data intelligent analysis and processing method based on a large model can also be applied to an AI data intelligent analysis and processing device based on a large model, wherein the device may include a data processing device or a data processing server, wherein the data processing server may include a cloud server or a local server, which is not limited in the embodiments of the present invention. Figure 2 As shown, the AI ​​data intelligent analysis and processing method based on the big model may include the following operations: 201. Obtain at least one initial data to be processed.

[0051] 202. Input each initial data into the data analysis model of the pre-trained AI data governance platform for analysis to obtain the data content of one or more sub-data contained in the initial data and the data format of each sub-data.

[0052] In an embodiment of the present invention, the data analysis model can be obtained through training with a preset deep learning algorithm. Specifically, the data analysis model is obtained by training the data samples collected by the AI ​​data governance platform with a deep learning algorithm.

[0053] 203. For each initial data, determine whether there is preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data.

[0054] In the embodiment of the present invention, optionally, the abnormal sub-data may include first sub-data whose data content satisfies a preset data abnormality condition, and / or second sub-data whose data format is inconsistent with that of other sub-data, which is not limited in the embodiment of the present invention.

[0055] In an embodiment of the present invention, when the judgment result of step 203 is no, that is, when it is judged that there is no abnormal sub-data in all the sub-data of the initial data, step 204 is triggered to be executed; when the judgment result of step 203 is yes, that is, when it is judged that there is abnormal sub-data in all the sub-data of the initial data, step 208 is triggered to be executed.

[0056] 204. Based on AI data cleaning and integration technology, all sub-data of the initial data are integrated to obtain target data corresponding to the initial data.

[0057] 205. Based on data recognition and classification technology, all target data are identified and classified to obtain data recognition and classification results.

[0058] 206. Based on data security and access control technology, the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform to obtain the target user matching each data field.

[0059] 207. Based on data analysis and mining technology, the data set corresponding to each data field is analyzed to obtain the potential value information of each data field.

[0060] 208. Based on AI data cleaning and integration technology, the abnormal sub-data of the initial data are cleaned.

[0061] In the embodiment of the present invention, after executing step 208, the AI ​​data cleaning and integration technology in step 204 may be triggered to integrate all sub-data of the initial data to obtain the target data corresponding to the initial data. In this operation, all sub-data are the remaining sub-data except the abnormal sub-data.

[0062] In the embodiment of the present invention, for other descriptions of step 201 to step 208, please refer to the detailed description of step 101 to step 105 in the first embodiment, and the embodiment of the present invention will not be repeated.

[0063] It can be seen that implementation Figure 2The described AI data intelligent analysis and processing method based on large models can build an AI data governance platform, and obtain at least one initial data to be processed, such as structured data or unstructured data, through the platform, and based on the AI ​​data cleaning and integration technology of the AI ​​data governance platform, clean and integrate each initial data to obtain the target data corresponding to each initial data, which can improve the cleaning accuracy and integration accuracy of the initial data, and is conducive to improving the accuracy and efficiency of the target data obtained by cleaning and integration. Subsequently, based on the data recognition and classification technology of the AI ​​data governance platform, all target data are identified and classified to obtain the data field of each target data and the data set corresponding to each data field, which can improve the recognition accuracy and classification accuracy of each target data, and then based on the data security and access control technology of the AI ​​data governance platform, the data corresponding to each data field is classified. By matching the data set with at least one user corresponding to the AI ​​data governance platform to obtain the target user matching each data field, the matching accuracy and efficiency between the data field and the platform users can be improved. Based on the data analysis and mining technology of the AI ​​data governance platform, the data set corresponding to each data field is analyzed to obtain the potential value information of each data field. The data processing efficiency and quality can be improved through a series of data processing methods, thereby improving the analysis accuracy and efficiency of the potential value of each data field. By outputting the potential value of the data to the data display interface corresponding to the target user for the user to view, it is beneficial to improve data security, reduce the situation where users unrelated to the data access the data and cause the data to be leaked, thereby reducing data security, and further improve users' trust, satisfaction and usage experience with the AI ​​data governance platform.In addition, each initial data can be input into the data analysis model of the pre-trained AI data governance platform for analysis to obtain the data content of one or more sub-data contained in the initial data and the data format of each sub-data, which can improve the analysis accuracy and efficiency of the content and format of the initial target data, and judge whether there are preset abnormal sub-data in all the sub-data of each initial data according to the data content and data format of all the sub-data of the initial data. When it is judged that there are no abnormal sub-data in all the sub-data of the initial data, all the sub-data of the initial data are directly integrated based on the AI ​​data cleaning and integration technology to obtain the target data corresponding to the initial data, which can In the absence of abnormal sub-data, the accuracy and reliability of the integration of initial data are improved. When it is determined that abnormal sub-data exists in all sub-data of the initial data, the abnormal sub-data of the initial data are cleaned based on AI data cleaning and integration technology, and the operation based on AI data cleaning and integration technology is triggered to integrate all sub-data of the initial data to obtain the target data corresponding to the initial data. All sub-data in the operation are the remaining sub-data except the abnormal sub-data. The accuracy of cleaning the abnormal sub-data can be improved in the presence of abnormal sub-data, thereby improving the accuracy and reliability of the processed target data by cleaning first and then integrating.

[0064] In an optional embodiment, for each initial data, judging whether there is a preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data in step 203 may include: For each initial data, judging whether the data content of each sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all sub-data of the initial data, and / or judging whether the data format of each sub-data of the initial data is inconsistent with the data formats of other sub-data according to the data formats of all sub-data of the initial data; When it is determined that the data format of the sub-data does not meet the data abnormality condition, and the data format of the sub-data is consistent with the data format of other sub-data, the sub-data is determined to be normal sub-data; When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined as the first sub-data; When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined as the second sub-data.

[0065] In an embodiment of the present invention, optionally, the data abnormality condition may include one or more conditions of an error in the data content, the data being irrelevant data, the data content similarity being greater than a preset similarity, etc., which is not limited in the embodiment of the present invention.

[0066] It can be seen that this optional embodiment can determine whether the data content of each sub-data of the initial data meets the preset data abnormality condition based on the data content of all sub-data of each initial data, and / or, based on the data format of all sub-data of the initial data, determine whether the data format of each sub-data of the initial data is inconsistent with the data format of other sub-data. When it is determined that the data format of the sub-data does not meet the data abnormality condition and the data format of the sub-data is consistent with the data format of other sub-data, the sub-data is determined to be normal sub-data, and the accuracy and reliability of determining whether the sub-data is normal can be improved through the data content and data format. When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined to be the first sub-data. When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined to be the second sub-data. Both the first sub-data and the second sub-data are abnormal sub-data, and the accuracy, diversity and flexibility of determining the abnormality of the sub-data can be improved through the data content and data format.

[0067] In this optional embodiment, as an optional implementation manner, for each initial data, judging whether the data content of each sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all sub-data of the initial data may include: For each sub-data of each initial data, judging whether the data content of the sub-data is erroneous according to the data content of the sub-data, which is recorded as the first condition; and / or, For each sub-data of each initial data, judging whether the sub-data belongs to the preset irrelevant data according to the data content of the sub-data, which is recorded as the second condition; and / or, For each initial data, according to the data content of all sub-data of the initial data, the content similarity between each sub-data of the initial data is calculated, and it is determined whether the content similarity between each sub-data of the initial data is greater than or equal to a preset similarity, which is recorded as the third condition; When at least one of the first condition, the second condition and the third condition is satisfied, it is determined that the data content of the corresponding sub-data satisfies a preset data abnormality condition.

[0068] In the embodiment of the present invention, optionally, the preset similarity may be 80%, or 65%, or any other preset value, which is not limited in the embodiment of the present invention.

[0069] Specifically, when each of the first condition and / or the second condition and / or the third condition included in the judgment condition is not satisfied, it is determined that the data content of the corresponding sub-data does not satisfy the preset data abnormality condition.

[0070] It can be seen that this optional implementation method can determine whether there is an error in the data content of each sub-data of each initial data based on the data content of the sub-data, which is recorded as the first condition; and / or, determine whether the sub-data belongs to preset irrelevant data based on the data content of the sub-data, which is recorded as the second condition; and / or, calculate the content similarity between each sub-data of the initial data based on the data content of all sub-data of the initial data, and determine whether the content similarity between each sub-data of the initial data is greater than or equal to the preset similarity, which is recorded as the third condition; wherein, when at least one of the first condition, the second condition and the third condition is met, it is determined that the data content of the corresponding sub-data meets the preset data abnormality condition, and the accuracy, diversity and flexibility of the judgment of whether the data content of each sub-data is abnormal can be improved by setting multiple judgment conditions.

[0071] Embodiment 3 See also Figure 3 , Figure 3 : is a schematic diagram of a structure of an AI data intelligent analysis and processing device based on a large model disclosed in an embodiment of the present invention. Figure 3 The AI ​​data intelligent analysis and processing device based on a large model described can be applied to an AI data governance platform, wherein the AI ​​data governance platform can integrate multiple data processing technologies, including one or more combinations of AI data cleaning and integration technologies, data identification and classification technologies, data security and access control technologies, and data analysis and mining technologies, and the AI ​​data intelligent analysis and processing device based on a large model can include a data processing device or a data processing server, wherein the data processing server can include a cloud server or a local server, which is not limited in the embodiments of the present invention. Figure 3 As shown, the AI ​​data intelligent analysis and processing device based on the big model may include: The data acquisition module 301 is used to acquire at least one initial data to be processed, where the initial data includes structured data or unstructured data; The data cleaning module 302 is used to clean and integrate each initial data based on AI data cleaning and integration technology to obtain target data corresponding to each initial data; The data classification module 303 is used to identify and classify all target data based on data identification and classification technology to obtain data identification and classification results. The data identification and classification results include the data field of each target data and the data set corresponding to each data field. All target data in the same data set correspond to the same data field. The access control module 304 is used to match the data set corresponding to each data field with at least one user corresponding to the AI ​​data governance platform based on data security and access control technology to obtain a target user matching each data field; The value analysis module 305 is used to analyze the data set corresponding to each data field based on data analysis and mining technology to obtain the potential value information of each data field. The potential value information is used to output to the data display interface corresponding to the target user, so that the target user can view the potential value information according to the display results of the data display interface.

[0072] It can be seen that implementation Figure 3 The described AI data intelligent analysis and processing device based on a large model can build an AI data governance platform, and obtain at least one initial data to be processed, such as structured data or unstructured data, through the platform, and based on the AI ​​data cleaning and integration technology of the AI ​​data governance platform, clean and integrate each initial data to obtain the target data corresponding to each initial data, which can improve the cleaning accuracy and integration accuracy of the initial data, and is conducive to improving the accuracy and efficiency of the target data obtained by cleaning and integration. Subsequently, based on the data recognition and classification technology of the AI ​​data governance platform, all target data are identified and classified to obtain the data field of each target data and the data set corresponding to each data field, which can improve the recognition accuracy and classification accuracy of each target data, and then based on the data security and access control technology of the AI ​​data governance platform, the corresponding The data set is matched with at least one user corresponding to the AI ​​data governance platform to obtain the target user matching each data field, which can improve the matching accuracy and efficiency between the data field and the platform users. Based on the data analysis and mining technology of the AI ​​data governance platform, the data set corresponding to each data field is analyzed to obtain the potential value information of each data field. The data processing efficiency and quality can be improved through a series of data processing methods, thereby improving the analysis accuracy and efficiency of the potential value of each data field. By outputting the potential value of the data to the data display interface corresponding to the target user for the user to view, it is beneficial to improve data security, reduce the occurrence of data leakage caused by users unrelated to the data accessing the data, and thus improve the user's trust, satisfaction and usage experience of the AI ​​data governance platform. In an optional embodiment, the data cleaning module 302 cleans and integrates each initial data based on AI data cleaning and integration technology, and the method of obtaining the target data corresponding to each initial data may specifically include: Input each initial data into the data analysis model of the pre-trained AI data governance platform for analysis to obtain the data content of one or more sub-data contained in the initial data and the data format of each sub-data. The data analysis model is trained by a preset deep learning algorithm. For each initial data, judging whether there is any preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data; When it is determined that there is no abnormal sub-data in all the sub-data of the initial data, all the sub-data of the initial data are integrated based on the AI ​​data cleaning and integration technology to obtain the target data corresponding to the initial data; When it is determined that abnormal sub-data exists in all the sub-data of the initial data, the abnormal sub-data of the initial data is cleaned based on the AI ​​data cleaning and integration technology, and the operation based on the AI ​​data cleaning and integration technology is triggered to integrate all the sub-data of the initial data to obtain the target data corresponding to the initial data, in which all the sub-data are the remaining sub-data except the abnormal sub-data.

[0073] It can be seen that this optional embodiment can input each initial data into the data analysis model of the pre-trained AI data governance platform for analysis, and obtain the data content of one or more sub-data contained in the initial data and the data format of each sub-data, which can improve the analysis accuracy and efficiency of the content and format of the initial target data, and judge whether there are preset abnormal sub-data in all the sub-data of each initial data according to the data content and data format of all the sub-data of the initial data. When it is judged that there are no abnormal sub-data in all the sub-data of the initial data, all the sub-data of the initial data are directly integrated based on the AI ​​data cleaning and integration technology to obtain the target data corresponding to the initial data. , which can improve the accuracy and reliability of the integration of initial data in the absence of abnormal sub-data. When it is determined that there is abnormal sub-data in all the sub-data of the initial data, the abnormal sub-data of the initial data is cleaned based on the AI ​​data cleaning and integration technology, and the operation based on the AI ​​data cleaning and integration technology is triggered to integrate all the sub-data of the initial data to obtain the target data corresponding to the initial data. All sub-data in the operation are the remaining sub-data except the abnormal sub-data. The cleaning accuracy of the abnormal sub-data can be improved in the presence of abnormal sub-data, thereby improving the accuracy and reliability of the processed target data by cleaning first and then integrating.

[0074] In this optional embodiment, as an optional implementation, for each initial data, the data cleaning module 302 determines whether there is a preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data, and the method may specifically include: For each initial data, judging whether the data content of each sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all sub-data of the initial data, and / or judging whether the data format of each sub-data of the initial data is inconsistent with the data formats of other sub-data according to the data formats of all sub-data of the initial data; When it is determined that the data format of the sub-data does not meet the data abnormality condition, and the data format of the sub-data is consistent with the data format of other sub-data, the sub-data is determined to be normal sub-data; When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined as the first sub-data; When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined as the second sub-data; The abnormal sub-data includes the first sub-data and / or the second sub-data.

[0075] It can be seen that this optional implementation can determine whether the data content of each sub-data of the initial data meets the preset data abnormality condition based on the data content of all sub-data of each initial data, and / or, based on the data format of all sub-data of the initial data, determine whether the data format of each sub-data of the initial data is inconsistent with the data format of other sub-data. When it is determined that the data format of the sub-data does not meet the data abnormality condition and the data format of the sub-data is consistent with the data format of other sub-data, the sub-data is determined to be normal sub-data, and the accuracy and reliability of determining whether the sub-data is normal can be improved through the data content and data format. When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined to be the first sub-data. When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined to be the second sub-data. Both the first sub-data and the second sub-data are abnormal sub-data, and the accuracy, diversity and flexibility of determining the abnormality of the sub-data can be improved through the data content and data format.

[0076] In this optional implementation, optionally, for each initial data, the data cleaning module 302 determines whether the data content of each sub-data of the initial data satisfies the preset data abnormality condition according to the data content of all sub-data of the initial data, which may specifically include: For each sub-data of each initial data, judging whether the data content of the sub-data is erroneous according to the data content of the sub-data, which is recorded as the first condition; and / or, For each sub-data of each initial data, judging whether the sub-data belongs to the preset irrelevant data according to the data content of the sub-data, which is recorded as the second condition; and / or, For each initial data, according to the data content of all sub-data of the initial data, the content similarity between each sub-data of the initial data is calculated, and it is determined whether the content similarity between each sub-data of the initial data is greater than or equal to a preset similarity, which is recorded as the third condition; When at least one of the first condition, the second condition and the third condition is satisfied, it is determined that the data content of the corresponding sub-data satisfies a preset data abnormality condition.

[0077] It can be seen that this optional implementation manner can also determine whether there is an error in the data content of each sub-data of each initial data based on the data content of the sub-data, which is recorded as the first condition; and / or, determine whether the sub-data belongs to preset irrelevant data based on the data content of the sub-data, which is recorded as the second condition; and / or, calculate the content similarity between each sub-data of the initial data based on the data content of all sub-data of the initial data, and determine whether the content similarity between each sub-data of the initial data is greater than or equal to the preset similarity, which is recorded as the third condition; wherein, when at least one of the first condition, the second condition and the third condition is met, it is determined that the data content of the corresponding sub-data meets the preset data abnormality condition, and the accuracy, diversity and flexibility of the judgment of whether the data content of each sub-data is abnormal can be improved by setting multiple judgment conditions.

[0078] In another optional embodiment, the AI ​​data cleaning and integration technology includes a technology for cleaning and integrating data based on natural language processing and / or image recognition, and the data identification and classification technology includes a technology for identifying and classifying data based on a convolutional neural network and / or a recurrent neural network. In addition, the data classification module 303 identifies and classifies all target data based on the data identification and classification technology, and the method for obtaining the data identification and classification results may specifically include: For each target data, analyzing the data source of the target data, the data source includes natural language technology or image recognition technology; For the first target data group whose data source is natural language technology, each target data in the first target data group is identified based on the technology of data recognition and classification by convolutional neural network, and the data field of each target data in the first target data group is obtained; For the second target data group whose data source is the image recognition technology, each target data in the second target data group is identified based on the technology of identifying and classifying data by a recurrent neural network, and the data field of each target data in the second target data group is obtained; According to the data fields of all target data in the first target data and the data fields of all target data in the second target data, all target data are classified to obtain a data set corresponding to each data field. All first target data and all second target data in the same data set correspond to the same data field.

[0079] It can be seen that this optional embodiment can analyze the data source of each target data, and for the first target data group whose data source is the natural language technology, based on the technology of identifying and classifying data using a convolutional neural network, identify each target data in the first target data group, and obtain the data field of each target data in the first target data group, which can improve the recognition accuracy and reliability of the data field of the target data obtained by processing with the natural language technology, and for the second target data group whose data source is the image recognition technology, based on the technology of identifying and classifying data using a recurrent neural network, identify each target data in the second target data group, and obtain the data field of each target data in the second target data group, which can improve the recognition accuracy and reliability of the data field of the target data obtained by processing with the image recognition technology, and can improve the diversity and flexibility of the identification method of the data field of the target data based on the diversified data sources, and then classify all the target data according to the data fields of all the target data in the first target data and the data fields of all the target data in the second target data, and obtain the data set corresponding to each data field, which can improve the classification accuracy and reliability of all the target data based on the data fields of all the target data obtained by accurate identification.

[0080] In another optional embodiment, the data analysis and mining technology includes cluster analysis technology and association rule mining technology. And, the value analysis module 305 analyzes the data set corresponding to each data field based on the data analysis and mining technology, and the method of obtaining the potential value information of each data field may specifically include: Based on cluster analysis technology, cluster analysis is performed on the data set corresponding to each data field to obtain multiple data item sets corresponding to each data field; For each data item set corresponding to each data field, the support of the data item set is calculated according to the transaction data set corresponding to the data item set; For each data domain, a data item set whose support is greater than or equal to a preset support is selected from all data item sets corresponding to the data domain as a target data item set corresponding to the data domain, and association rule information corresponding to the data domain is generated based on all data item sets corresponding to the data domain; Based on the association rule mining technology, according to the target data item set corresponding to each data field, the association relationship between the target data item sets corresponding to each data field is analyzed; According to the corresponding association relationship of each data field, the potential value information of each data field is determined.

[0081] It can be seen that this optional embodiment can perform cluster analysis on the data set corresponding to each data field based on the cluster analysis technology, obtain multiple data item sets corresponding to each data field, and accurately calculate the support of the data item set according to the transaction data set corresponding to each data item set, and for each data field, screen out the data item set with a support greater than or equal to the preset support from all the data item sets corresponding to the data field as the target data item set corresponding to the data field, which can improve the screening accuracy and reliability of the target data item set, and accurately generate the association rule information corresponding to the data field according to all the data item sets corresponding to the data field, and then analyze the association relationship between the target data item sets corresponding to each data field based on the association rule mining technology according to the target data item set corresponding to each data field, which can improve the analysis accuracy of the association relationship between the target data item sets corresponding to each data field, and then determine the potential value information of each data field according to the association relationship corresponding to each data field, which is conducive to improving the analysis accuracy and reliability of the potential value of each data field.

[0082] In another optional embodiment, the access control module 304 matches the data set corresponding to each data field with at least one user corresponding to the AI ​​data governance platform based on data security and access control technology, and the method of obtaining the target user matching each data field may specifically include: Based on data security and access control technology, determine the first access control information of each data field; When the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, the second access control information of each user is determined according to the data access request of each user; According to the first access control information of all data fields and the second access control information of all users, all data fields are matched with all users to obtain one or more matching groups, each of which includes a data field and a target user matching the data field.

[0083] It can be seen that this optional embodiment can accurately determine the first access control information of each data field based on data security and access control technology, and when the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, accurately determine the second access control information of each user according to the data access request of each user, and then match all data fields with all users according to the first access control information of all data fields and the second access control information of all users to obtain one or more matching groups, which can improve the matching accuracy and reliability between data fields and target users.

[0084] Embodiment 4 See also Figure 4 , Figure 4 Schematic diagram of another AI data intelligent analysis and processing device based on a large model disclosed in an embodiment of the present invention. Figure 4 As shown, the AI ​​data intelligent analysis and processing device based on the big model may include: A memory 401 storing executable program codes; a processor 402 coupled to the memory 401; The processor 402 calls the executable program code stored in the memory 401 to execute the steps of the AI ​​data intelligent analysis and processing method based on a large model described in the first embodiment of the present invention or the second embodiment of the present invention.

[0085] Embodiment 5 An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the AI ​​data intelligent analysis and processing method based on a large model described in Embodiment 1 or Embodiment 2 of the present invention.

[0086] Embodiment 6 An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the large model-based AI data intelligent analysis and processing method described in Example 1 or Example 2.

[0087] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0088] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0089] Finally, it should be noted that the AI ​​data intelligent analysis and processing method and device based on a large model disclosed in the embodiment of the present invention discloses only the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for intelligent analysis and processing of AI data based on a large model, characterized in that: The method is applied to an AI data governance platform, and the method includes: Acquire at least one initial data to be processed, where the initial data includes structured data or unstructured data; Based on AI data cleaning and integration technology, each of the initial data is cleaned and integrated to obtain the target data corresponding to each of the initial data; Based on data recognition and classification technology, all the target data are identified and classified to obtain data recognition and classification results, wherein the data recognition and classification results include the data field of each target data and the data set corresponding to each data field, and all the target data in the same data set correspond to the same data field; Based on data security and access control technology, the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform to obtain a target user that matches each data field; Based on data analysis and mining technology, the data set corresponding to each of the data fields is analyzed to obtain the potential value information of each of the data fields. The potential value information is used to be output to the data display interface corresponding to the target user, so that the target user can view the potential value information according to the display results of the data display interface.

2. The AI ​​data intelligent analysis and processing method based on a large model according to claim 1 is characterized in that: The AI-based data cleaning and integration technology cleans and integrates each of the initial data to obtain target data corresponding to each of the initial data, including: Input each of the initial data into the pre-trained data analysis model of the AI ​​data governance platform for analysis to obtain the data content of one or more sub-data contained in the initial data and the data format of each of the sub-data, wherein the data analysis model is obtained by training with a preset deep learning algorithm; For each of the initial data, judging whether there is any preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data; When it is determined that the abnormal sub-data does not exist in all the sub-data of the initial data, all the sub-data of the initial data are integrated based on the AI ​​data cleaning and integration technology to obtain the target data corresponding to the initial data; When it is determined that the abnormal sub-data exists in all the sub-data of the initial data, the abnormal sub-data of the initial data is cleaned based on the AI ​​data cleaning and integration technology, and the operation based on the AI ​​data cleaning and integration technology is triggered to integrate all the sub-data of the initial data to obtain the target data corresponding to the initial data, in which all the sub-data are the remaining sub-data except the abnormal sub-data.

3. The AI ​​data intelligent analysis and processing method based on a large model according to claim 2 is characterized in that: For each of the initial data, judging whether there is a preset abnormal sub-data in all the sub-data of the initial data according to the data content and data format of all the sub-data of the initial data, includes: For each of the initial data, judging whether the data content of each of the sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all the sub-data of the initial data, and / or judging whether the data format of each of the sub-data of the initial data is inconsistent with the data formats of other sub-data according to the data formats of all the sub-data of the initial data; When it is determined that the data format of the sub-data does not satisfy the data abnormality condition, and the data format of the sub-data is consistent with the data format of other sub-data, determining that the sub-data is normal sub-data; When it is determined that the data content of the sub-data meets the data abnormality condition, the sub-data is determined as the first sub-data; When it is determined that the data format of the sub-data is inconsistent with the data format of other sub-data, the sub-data is determined as the second sub-data; The abnormal sub-data includes the first sub-data and / or the second sub-data.

4. The AI ​​data intelligent analysis and processing method based on a large model according to claim 3 is characterized in that: For each of the initial data, judging whether the data content of each of the sub-data of the initial data satisfies a preset data abnormality condition according to the data content of all the sub-data of the initial data, includes: For each sub-data of each of the initial data, judging whether the data content of the sub-data is erroneous according to the data content of the sub-data, which is recorded as the first condition; and / or, For each sub-data of each of the initial data, judging whether the sub-data belongs to the preset irrelevant data according to the data content of the sub-data, which is recorded as the second condition; and / or, For each of the initial data, according to the data contents of all the sub-data of the initial data, calculate the content similarity between each of the sub-data of the initial data, and determine whether the content similarity between each of the sub-data of the initial data is greater than or equal to a preset similarity, which is recorded as the third condition; When at least one of the first condition, the second condition and the third condition is satisfied, it is determined that the data content of the corresponding sub-data satisfies a preset data abnormality condition.

5. The AI ​​data intelligent analysis and processing method based on a large model according to any one of claims 1 to 4, characterized in that: The AI ​​data cleaning and integration technology includes technologies for cleaning and integrating data based on natural language processing and / or image recognition, and the data identification and classification technology includes technologies for identifying and classifying data based on convolutional neural networks and / or recurrent neural networks; And, based on the data recognition and classification technology, all the target data are identified and classified to obtain data recognition and classification results, including: For each of the target data, analyzing the data source of the target data, wherein the data source includes natural language technology or image recognition technology; For the first target data group whose data source is the natural language technology, based on the technology of data recognition and classification by the convolutional neural network, each target data in the first target data group is recognized to obtain the data field of each target data in the first target data group; For the second target data group whose data source is the image recognition technology, based on the technology of identifying and classifying data by the recurrent neural network, each target data in the second target data group is identified to obtain the data field of each target data in the second target data group; According to the data fields of all target data within the first target data and the data fields of all target data within the second target data, all the target data are classified to obtain a data set corresponding to each data field, and all first target data and all second target data in the same data set correspond to the same data field.

6. The AI ​​data intelligent analysis and processing method based on a large model according to claim 5 is characterized in that: The data analysis and mining technology includes cluster analysis technology and association rule mining technology; Furthermore, the data analysis and mining technology is used to analyze the data set corresponding to each data field to obtain the potential value information of each data field, including: Based on the cluster analysis technology, cluster analysis is performed on the data set corresponding to each of the data fields to obtain multiple data item sets corresponding to each of the data fields; For each of the data item sets corresponding to each of the data fields, calculating the support of the data item set according to the transaction data set corresponding to the data item set; For each of the data fields, the data item set whose support is greater than or equal to the preset support is selected from all the data item sets corresponding to the data field as the target data item set corresponding to the data field, and the association rule information corresponding to the data field is generated according to all the data item sets corresponding to the data field; Based on the association rule mining technology, according to the target data item set corresponding to each data field, analyzing the association relationship between the target data item sets corresponding to each data field; According to the association relationship corresponding to each of the data fields, the potential value information of each of the data fields is determined.

7. The AI ​​data intelligent analysis and processing method based on a large model according to claim 5 is characterized in that: Based on the data security and access control technology, the data set corresponding to each data field is matched with at least one user corresponding to the AI ​​data governance platform to obtain a target user matching each data field, including: Determine first access control information for each of the data fields based on data security and access control technology; When the AI ​​data governance platform receives data access requests from one or more users for the AI ​​data governance platform, based on data security and access control technology, according to the data access request of each user, determine the second access control information of each user; According to the first access control information of all the data fields and the second access control information of all the users, all the data fields are matched with all the users to obtain one or more matching groups, each of which includes a data field and a target user matching the data field.

8. An AI data intelligent analysis and processing device based on a large model, characterized in that: The device is applied to an AI data governance platform, and the device includes: A data acquisition module, used to acquire at least one initial data to be processed, wherein the initial data includes structured data or unstructured data; A data cleaning module, used to clean and integrate each of the initial data based on AI data cleaning and integration technology to obtain target data corresponding to each of the initial data; A data classification module, used to identify and classify all the target data based on data identification and classification technology to obtain data identification and classification results, wherein the data identification and classification results include the data field of each target data and the data set corresponding to each data field, and all target data in the same data set correspond to the same data field; An access control module, used to match the data set corresponding to each of the data fields with at least one user corresponding to the AI ​​data governance platform based on data security and access control technology, to obtain a target user that matches each of the data fields; The value analysis module is used to analyze the data set corresponding to each data field based on data analysis and mining technology to obtain the potential value information of each data field. The potential value information is used to output to the data display interface corresponding to the target user so that the target user can view the potential value information according to the display results of the data display interface.

9. An AI data intelligent analysis and processing device based on a large model, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the large model-based AI data intelligent analysis and processing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when called, are used to execute the large model-based AI data intelligent analysis and processing method as described in any one of claims 1 to 7.

Images