Data specification checking method and device based on neural network, equipment and medium

Abstract

The application discloses a neural network-based data specification checking method and device, equipment and medium. The method comprises the following steps: obtaining data to be checked; inputting the data to be checked into a pre-trained data type identification model to obtain the data type of each column of data; inputting each column of data and the corresponding data type into a pre-trained specification checking model to obtain the specification checking result of each column of data. According to the data specification checking method provided in the application, the efficiency of data specification checking is greatly improved, the data type identification capability and generalization capability are increased, so that the data type identification omission caused by the limitation and strong matching of the regular expression is avoided, and the personnel contact with public data is reduced, so that the data sharing and use safety are ensured.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for data standardization checking based on neural networks. Background Technology

[0002] As government digital transformation continues to advance, the importance of data security to government work is becoming increasingly apparent. For government departments, protecting public data has become a key focus. As an important inspection item in the field of data security, data compliance inspection is an indispensable step in building secure public data, as it checks whether data is stored in accordance with business requirements and standards.

[0003] Existing data compliance checks often involve manual checks by data department staff. This basic manual method is time-consuming and inefficient, failing to meet actual business needs. Other methods use regular expressions for data matching. However, this approach suffers from poor generalization due to the large number and complexity of regular expressions required, leading to omissions. Furthermore, the sheer volume of data and the diverse data storage formats involved also negatively impact the efficiency and accuracy of the checks. Summary of the Invention

[0004] This application provides a data standardization checking method, apparatus, device, and medium based on neural networks. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] In a first aspect, embodiments of this application provide a data normalization checking method based on neural networks, including:

[0006] Obtain the data to be inspected;

[0007] The data to be checked is input into a pre-trained data type recognition model to obtain the data type of each column.

[0008] Each column of data and its corresponding data type is input into a pre-trained normative checking model to obtain the normative checking results for each column of data.

[0009] In an optional embodiment, before inputting the data to be inspected into a pre-trained data type recognition model, the method further includes:

[0010] Define the target data categories, which can be single-level or multi-level target data categories;

[0011] Obtain and label the data corresponding to each minimum level of target data category to obtain the first dataset;

[0012] The first dataset is divided into a training set and a test set, and a data type recognition model is trained based on the training set and the test set.

[0013] In an optional embodiment, the data to be inspected is input into a pre-trained data type recognition model to obtain the data type of each column, including:

[0014] The data to be inspected is input into a pre-trained data type recognition model to obtain the data type of each data in each column;

[0015] Voting is performed on the data types of all data in each column. The data type with the most proportion is used as the data type of that column. Columns with equal proportions of data types are defined as other types. Columns whose data types cannot be identified are also defined as other types.

[0016] The data types in other data columns are identified and labeled to obtain the updated training dataset;

[0017] Train the data type recognition model based on the updated training dataset to obtain the updated data type recognition model;

[0018] The updated data type identification model is used to identify the data types of other data columns again until the data types of all data columns are obtained.

[0019] In an optional embodiment, before inputting each column of data and its corresponding data type into the pre-trained prescriptive checking model, the method further includes:

[0020] Based on the departmental business corresponding to the data to be inspected, the data category to be inspected is set, and the data category to be inspected is a first-level data category;

[0021] Obtain the data corresponding to the data category to be inspected and label it to obtain the second dataset, wherein the data format in the second dataset meets the preset normative inspection rules;

[0022] The second dataset is divided into a training set and a test set, and a normative checking model is trained based on the training set and the test set.

[0023] In an optional embodiment, each column of data and its corresponding data type are input into a pre-trained normalization checking model to obtain the normalization checking result for each column of data, including:

[0024] Input each column of data and its corresponding data type into the pre-trained normative checking model;

[0025] Determine whether the data type of each column belongs to the data category to be checked. If it does not, then the data format of that column is determined to be non-standard.

[0026] In an optional embodiment, it further includes:

[0027] If the data type of a certain column belongs to the preset data category to be checked, then output the data type of each data in that column;

[0028] Determine if all data in this column are of the same data type. If they are all the same, then determine the format specification of this column.

[0029] If not all are identical, then the data in that column is not formatted correctly.

[0030] In an optional embodiment, the neural network structure of the data type identification model and the normativity check model is an improved GAE network structure. The improved GAE network structure includes an encoding layer and a decoding layer. The encoding layer includes a GNN network and an FC network connected in sequence, and the decoding layer includes an FC network.

[0031] Secondly, embodiments of this application provide a data normalization checking device based on a neural network, comprising:

[0032] The acquisition module is used to acquire the data to be inspected.

[0033] The identification module is used to input the data to be inspected into a pre-trained data type identification model to obtain the data type of each column of data;

[0034] The inspection module is used to input each column of data and its corresponding data type into a pre-trained normative inspection model to obtain the normative inspection results for each column of data.

[0035] Thirdly, embodiments of this application provide a data standardization checking device based on a neural network, including a processor and a memory storing program instructions. The processor is configured to execute the data standardization checking method based on a neural network provided in the above embodiments when executing the program instructions.

[0036] Fourthly, embodiments of this application provide a computer-readable medium storing computer-readable instructions, which are executed by a processor to implement a data normalization checking method based on a neural network provided in the above embodiments.

[0037] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0038] The data standardization checking method based on neural networks provided in this application improves the efficiency of data standardization checking by first identifying data types and then checking data formats. It uses a pre-trained neural network model instead of regular expressions to identify data types, thus avoiding omissions in data type identification due to the limitations and strong matching of regular expressions. It uses a neural network model instead of manual data services, reducing personnel's contact with public data and ensuring data sharing and usage security. Furthermore, it uses neural networks instead of manual data standardization checks, reducing the time spent on manual processes and improving the efficiency of data standardization checking, thus adapting to business development requirements and performance needs.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] Figure 1 This is a flowchart illustrating a data normalization checking method based on a neural network according to an exemplary embodiment;

[0042] Figure 2 This is a schematic diagram illustrating a training method for a data recognition model according to an exemplary embodiment;

[0043] Figure 3 This is a schematic diagram illustrating a data identification model update method according to an exemplary embodiment;

[0044] Figure 4 This is a schematic diagram illustrating an improved GAE network structure according to an exemplary embodiment;

[0045] Figure 5 This is a schematic diagram illustrating the structure of a neural network-based data standardization checking device according to an exemplary embodiment;

[0046] Figure 6 This is a schematic diagram illustrating the structure of a neural network-based data standardization checking device according to an exemplary embodiment;

[0047] Figure 7 This is a schematic diagram illustrating a computer storage medium according to an exemplary embodiment. Detailed Implementation

[0048] The following description and accompanying drawings fully illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0049] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0050] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems and methods consistent with some aspects of the invention as detailed in the appended claims.

[0051] Existing data compliance checks often involve manual checks by data department staff. This basic manual method is time-consuming and inefficient, failing to meet actual business needs. Other methods use regular expressions for data matching. However, this approach suffers from poor generalization due to the large number and complexity of regular expressions required, leading to omissions. Furthermore, the sheer volume of data and the diverse data storage formats involved also negatively impact the efficiency and accuracy of the checks.

[0052] Based on this, this application provides a data standardization inspection method based on a neural network model. For government public data, it adopts a method of first identifying data types and then inspecting them, improving the efficiency of data standardization inspection. Using model recognition to replace manual classification improves the efficiency of data classification and reduces the demand for government human resources. Using a GAE neural network instead of regular expressions to identify data types improves the generalization ability and accuracy of the identification.

[0053] The data normalization checking method based on neural networks provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. See also Figure 1 The method specifically includes the following steps.

[0054] Step S101: Obtain the data to be inspected.

[0055] In one possible implementation, the data inspection method in this application embodiment can conduct standardized inspections of public data from government departments. Public data mainly refers to information generated by the government during administrative law enforcement processes, such as information arising from administrative licensing, court litigation, and other activities. Since this information is generated by the coercive power of the government and the law, it has a certain impact on the production, operation, and performance of enterprises and individuals, and also involves the interests of the public and others. For government departments, data security protection for public data has become a key focus. Therefore, public data generated by government departments can be obtained as the data to be inspected, and standardized inspections can be conducted on this public data.

[0056] Optionally, the scope of application of the embodiments of this application includes, but is not limited to, government public data scenarios, and is also applicable to enterprise industry data.

[0057] Step S102: Input the data to be checked into the pre-trained data type recognition model to obtain the data type of each column.

[0058] In one possible implementation, the data type recognition model is trained before the data to be inspected is input into the pre-trained data type recognition model.

[0059] Specifically, the first step is to define the target data categories, which can be single-level or multi-level. Common data types are designated as the target data categories that the model needs to identify. For example, phone numbers, names, addresses, and times can be used as single-level categories. Depending on the requirements, second-level categories can be created under the single-level categories; for example, under the phone number category, second-level categories could include mobile phone numbers, landline numbers, and internal network short numbers.

[0060] Further, the data corresponding to each smallest-level target data category is obtained and labeled to obtain the first dataset. Generally, second-level target categories are obtained. When no second-level target categories exist, the first-level target category is used as the smallest category. At least 300 data points are selected as labeled data. The selected data should have as many different formats as possible; for example, mobile phone numbers can be displayed in different formats such as 133-****-**** or 133********, resulting in the final first dataset. Noisy data, i.e., data whose format does not conform to the category, is allowed in the labeled data.

[0061] Furthermore, the first dataset is divided into a training set and a test set, and a data type recognition model is trained based on the training set and the test set. The data type recognition model includes, but is not limited to, machine learning models such as KNN, SVM, and neural network models such as CNN, RNN, and GNN.

[0062] In one possible implementation, the data type identification model employs an improved GAE network structure, such as... Figure 4 As shown, the improved GAE network structure includes an encoding layer and a decoding layer. The encoding layer consists of a GNN network and an FC network connected in sequence, and the decoding layer consists of an FC network. By adopting the improved GAE network structure, the model training time can be shortened, while also having the advantages of better generalization performance, higher robustness, and faster inference speed.

[0063] Specifically, the model is trained based on the improved GAE neural network structure. Pattern encoding is used to encode the regular features of the data, generating an initial feature vector X1; pattern encoding is also used to encode the machine learning statistical features, generating an initial feature vector X2; pattern encoding is then used to encode the semantic features of the data, generating an initial feature vector X3. The X1, X2, and X3 vectors are re-encoded and then standardized to obtain the standard vector X. A random 80 / 20 split is used for the test and training sets.

[0064] Further, model evaluation metrics are constructed, the constructed Graph Autoencoder Neural Network (GAE) is obtained, and default initial parameters are set. The training set is input into the constructed GAE model to update the parameters. The test set is input into the model. If the evaluation metrics are not met, the process is returned and the parameters are adjusted according to the manual. If the evaluation metrics are met, the data recognition model construction is complete.

[0065] Figure 2 This is a schematic diagram illustrating a training method for a data recognition model according to an exemplary embodiment, such as... Figure 2 As shown, first define the target data categories. You can create a primary target category and then create secondary target categories as needed.

[0066] Further, training data is selected, typically from secondary target categories. If no secondary target categories exist, the primary target category is used as the minimum category, and at least 300 data points are selected as labeled data. Next, feature extraction and quantization are performed. Pattern coding is used to encode the regular features of the data, generating an initial feature vector X1; pattern coding is also used to encode the machine learning statistical features, generating an initial feature vector X2; pattern coding is used to encode the semantic features of the data, generating an initial feature vector X3; vectors X1, X2, and X3 are re-encoded and then standardized to obtain the standard vector X. A random 80 / 20 split is used between the test and training sets.

[0067] The model is trained using the training and test sets, and the parameters are updated until the model evaluation metrics are met, resulting in a well-trained data type recognition model.

[0068] Furthermore, the data to be inspected is input into the trained data type recognition model to obtain the data type of each data point in each column. A vote is then taken on the data type of all data in each column, and the data type with the highest proportion is designated as the data type for that column. Columns with an equal proportion of data types are defined as other types, and columns whose data types cannot be identified are also defined as other types.

[0069] In an exemplary scenario, suppose a column has only 3 values, and the model identifies them as a, a, and b respectively. Then, through voting, the data type of this column is determined to be 'a'. Suppose a column has only 4 values, and the model identifies them as a, a, b, and b respectively. Then, through voting, the data type of this column is determined to be "other". Suppose a column has only 3 values, and none of them can be identified as having a data type. Then, the data type of this column is determined to be "other". In summary, the data type of all values ​​in the column is voted on, and the type with the highest percentage is the data type of this column.

[0070] After identifying the data type of each column, it is determined whether the data type of all columns can be determined. If the data type of each column can be determined, step S103 is executed, and the data of each column and its corresponding data type are input into the pre-trained normalization check model. If the data type of some columns is other types, the data type identification model is updated according to the data columns of other types.

[0071] Specifically, the data type of each data item in other data columns is identified and labeled to obtain the updated training dataset.

[0072] If other types of datasets, such as Dother, contain data, then the data type of each data item in the other data type columns is identified and labeled, resulting in a new data type set, Supdate. Similarly, a primary category needs to be defined, and secondary categories as needed. For each minimal category in the data type set Supdate (usually a secondary category; if no secondary category exists, the primary category is used as the minimal category), at least 300 data items are selected as labeled data. The selected data should have as many different representations as possible, resulting in an updated training dataset, Dupdate. Noisy data, i.e., data whose format does not conform to the specified category, is allowed in the labeled data.

[0073] Train the data type recognition model using the updated training dataset to obtain the updated data type recognition model. The training method is the same as that used for the original data type recognition model, until the evaluation criteria are met. This yields the updated data type recognition model.

[0074] The updated data type identification model is used to re-identify the data types of other data columns until the data types of all data columns are identified. If, after this step, there are still data columns whose categories cannot be distinguished, the above steps of updating the model are repeated, or the types are manually labeled. Finally, the data type of each column is obtained.

[0075] Figure 3 This is a schematic diagram illustrating a data recognition model update method according to an exemplary embodiment, such as... Figure 3 As shown, a data type identification model is used to traverse all data columns in the target database (the database to be inspected), determining whether the data in a column is identified as another type. If so, the data in that column is manually labeled, and the data type identification model is updated based on the labeled data. The updated data type identification model is then used to classify other data types. If no data is identified as another type, the data type of each column is determined based on the classification result of each value, and the data types of all data columns are summarized.

[0076] By identifying the data type of each column in the above manner, the efficiency of standardization checks can be improved.

[0077] Step S103: Input each column of data and its corresponding data type into the pre-trained normative checking model to obtain the normative checking results for each column of data.

[0078] In one possible implementation, before inputting each column of data and its corresponding data type into the pre-trained normative checking model, the process also includes: training the normative checking model.

[0079] Specifically, based on the departmental business corresponding to the data to be inspected, the data categories to be inspected are set, and these categories are the first-level data categories. Data categories are then organized according to the department's business situation. Unlike step S102, the data categories for the standardization inspection are only set up to the first-level category, such as telephone number, mobile phone number, name, and time.

[0080] Obtain and label the data corresponding to the categories to be inspected to obtain a second dataset. The data format in the second dataset must meet the preset normative inspection rules. For example, select at least 300 data points for each category as labeled data. The selected data should have the same format and conform to the normative inspection rules of this department to obtain the second dataset.

[0081] The second dataset is divided into a training set and a test set, and a normative checking model is trained based on the training set and the test set.

[0082] Specifically, features are extracted and quantized from the data in the second dataset. Pattern coding is used to encode the regular features of the data, generating an initial feature vector X1; pattern coding is also used to encode the machine learning statistical features, generating an initial feature vector X2; pattern coding is then used to encode the semantic features of the data, generating an initial feature vector X3; vectors X1, X2, and X3 are re-encoded and then standardized to obtain a standard vector X. The test and training sets are randomly divided into 2 / 8 sets.

[0083] An improved GAE network is trained using training and test sets, and its parameters are updated until the model evaluation metrics are met, resulting in a well-trained canonicality checking model. By employing the improved GAE network model, training time can be shortened, while also offering advantages such as better generalization performance, higher robustness, and faster inference speed.

[0084] Furthermore, each column of data and its corresponding data type are input into a pre-trained normative checking model.

[0085] First, the model determines whether the data type of each column belongs to the category of data to be inspected within the department. If it does not, the data format of that column is determined to be non-standard. If the data type of a column belongs to the preset category of data to be inspected, the model outputs the data type of each data item in that column.

[0086] Furthermore, determine if all data in the column are of the same data type. If they are all the same, the column's data format is considered correct. If they are not all the same—for example, if the classification results contain two different categories or other unidentifiable types—the column's data format is considered incorrect. The final data format check result is then obtained.

[0087] The data standardization checking method based on neural networks provided in this application improves the efficiency of data standardization checking by first identifying data types and then checking data formats. It uses a pre-trained neural network model instead of regular expressions to identify data types, thus avoiding omissions in data type identification due to the limitations and strong matching of regular expressions. Furthermore, it uses a neural network model instead of manual data services, reducing personnel contact with public data, thereby ensuring data sharing and usage security, reducing the time spent on manual processes, and ultimately improving the efficiency of data standardization checking to meet business development requirements and performance needs.

[0088] This application also provides a neural network-based data standardization checking device, which is used to execute the neural network-based data standardization checking method described in the above embodiments, such as... Figure 5 As shown, the device includes:

[0089] Module 501 is used to acquire the data to be inspected;

[0090] The recognition module 502 is used to input the data to be checked into a pre-trained data type recognition model to obtain the data type of each column of data;

[0091] The inspection module 503 is used to input each column of data and its corresponding data type into a pre-trained normative inspection model to obtain the normative inspection results for each column of data.

[0092] It should be noted that the data standardization checking device based on neural networks provided in the above embodiments is only illustrated by the division of the above functional modules when executing the data standardization checking method based on neural networks. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data standardization checking device based on neural networks provided in the above embodiments and the data standardization checking method embodiments based on neural networks belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0093] This application also provides an electronic device corresponding to the neural network-based data standardization checking method provided in the foregoing embodiments, for executing the aforementioned neural network-based data standardization checking method.

[0094] Please refer to Figure 6 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device includes: a processor 600, a memory 601, a bus 602, and a communication interface 603. The processor 600, the communication interface 603, and the memory 601 are connected via the bus 602. The memory 601 stores a computer program that can run on the processor 600. When the processor 600 runs the computer program, it executes the data normalization checking method based on neural networks provided in any of the foregoing embodiments of this application.

[0095] The memory 601 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 603 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0096] Bus 602 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 601 is used to store programs. After receiving execution instructions, processor 600 executes the program. The data normalization checking method based on neural networks disclosed in any of the aforementioned embodiments of this application can be applied to processor 600, or implemented by processor 600.

[0097] The processor 600 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 600 or by instructions in software form. The processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 601. Processor 600 reads the information in memory 601 and, in conjunction with its hardware, completes the steps of the above method.

[0098] The electronic device provided in this application embodiment and the data standardization checking method based on neural networks provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0099] This application also provides a computer-readable storage medium corresponding to the neural network-based data normalization checking method provided in the foregoing embodiments. Please refer to... Figure 7 The computer-readable storage medium shown is an optical disc 700, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the data normalization check method based on neural networks provided in any of the foregoing embodiments.

[0100] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0101] The computer-readable storage medium provided in the above embodiments of this application and the data normalization checking method based on neural networks provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A data normalization checking method based on neural networks, characterized in that, include: Obtain the data to be inspected; The data to be checked is input into a pre-trained data type recognition model to obtain the data type of each column. Input each column of data and its corresponding data type into the pre-trained normative checking model to obtain the normative checking results for each column of data. This includes inputting each column of data and its corresponding data type into a pre-trained standardization inspection model; determining whether the data type of each column belongs to the category of data to be inspected; if not, determining that the format of the data in that column is non-standard; further, if the data type of a column belongs to the preset category of data to be inspected, outputting the data type of each data in that column; determining whether the data types of each data in that column are all the same; if all are the same, determining that the format of the data in that column is standard; if not all are the same, determining that the format of the data in that column is non-standard. The neural network structure of the data type recognition model and the normativity check model is an improved GAE network structure. The improved GAE network structure includes an encoding layer and a decoding layer. The encoding layer includes a GNN network and an FC network connected in sequence, and the decoding layer includes an FC network.

2. The method according to claim 1, characterized in that, Before inputting the data to be inspected into the pre-trained data type recognition model, the following steps are also included: Define target data categories, which may be single-level or multi-level target data categories; Obtain and label the data corresponding to each minimum level of target data category to obtain the first dataset; The first dataset is divided into a training set and a test set, and the data type recognition model is trained based on the training set and the test set.

3. The method according to claim 1, characterized in that, The data to be checked is input into a pre-trained data type recognition model to obtain the data type of each column, including: The data to be inspected is input into a pre-trained data type recognition model to obtain the data type of each data in each column; Voting is performed on the data types of all data in each column. The data type with the most proportion is used as the data type of that column. Columns with equal proportions of data types are defined as other types. Columns whose data types cannot be identified are also defined as other types. The data types in other data columns are identified and labeled to obtain the updated training dataset; The data type recognition model is trained based on the updated training dataset to obtain the updated data type recognition model; The updated data type identification model is used to identify the data types of other data columns again until the data types of all data columns are obtained.

4. The method according to claim 1, characterized in that, Before inputting each column of data and its corresponding data type into the pre-trained normative checking model, the following steps are also included: Based on the departmental business corresponding to the data to be inspected, the data category to be inspected is set, and the data category to be inspected is a first-level data category; Obtain the data corresponding to the data category to be inspected and label it to obtain a second dataset, wherein the data format in the second dataset meets the preset normative inspection rules; The second dataset is divided into a training set and a test set, and the normative inspection model is trained based on the training set and the test set.

5. A data standardization checking device based on neural networks, characterized in that, include: The acquisition module is used to acquire the data to be inspected. The identification module is used to input the data to be inspected into a pre-trained data type identification model to obtain the data type of each column of data; The inspection module is used to input each column of data and its corresponding data type into a pre-trained normative inspection model to obtain the normative inspection results for each column of data. This includes inputting each column of data and its corresponding data type into a pre-trained standardization inspection model; determining whether the data type of each column belongs to the category of data to be inspected; if not, determining that the format of the data in that column is non-standard; further, if the data type of a column belongs to the preset category of data to be inspected, outputting the data type of each data in that column; determining whether the data types of each data in that column are all the same; if all are the same, determining that the format of the data in that column is standard; if not all are the same, determining that the format of the data in that column is non-standard. The neural network structure of the data type recognition model and the normativity check model is an improved GAE network structure. The improved GAE network structure includes an encoding layer and a decoding layer. The encoding layer includes a GNN network and an FC network connected in sequence, and the decoding layer includes an FC network.

6. A data standardization checking device based on neural networks, characterized in that, It includes a processor and a memory storing program instructions, the processor being configured to perform, when executing the program instructions, the neural network-based data normalization checking method as described in any one of claims 1 to 4.

7. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement a neural network-based data normalization checking method as described in any one of claims 1 to 4.

Images