Data differentiation detection method and device, medium and equipment

Abstract

The invention provides a data differentiation detection method and device, a medium and equipment, which are used for detecting the data difference between a first file in a first data format and a second file in a second data format, and comprises the following steps: respectively carrying out analysis and structure conversion on the first file and the second file, selecting a first processing library of a java language for the first data format, and performing java object conversion on a character string of the first file based on a tree model provided by the first processing library so as to construct a first file hierarchical tree based on the tree model; selecting a java language second processing library to read the second file, obtaining a worksheet object of the second file, creating a map structure to store the worksheet object, and constructing a second file dictionary; and performing field matching and content differentiation detection on the first file hierarchy tree and the second file dictionary, and performing differentiation marking according to a comparison result. According to the invention, the efficiency and accuracy of file differentiation detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data differentiation detection method, device, medium and equipment. Background Art

[0002] In the process of software development, interfaces are usually developed through governance files (such as Excel files). After development is completed, they are tested in the user test environment. After testing is completed, the corresponding developed interface information (such as JSON files) needs to be exported and deployed to the production environment. In this process, it is often necessary to convert the governance files of business logic into interface information for the convenience of online use of the system. For example, in the software development online stage, the governance files are converted into interface information for use in the online material review. This process may cause inconsistencies between interface data and governance files during the online process, which may include: missing data: when exporting interface information (such as JSON) from the user test environment, interfaces are omitted; data differences: the values ​​of the same field in the two files are different; format errors: the data format does not meet the expected requirements; structural differences: the data structure is inconsistent in the two files.

[0003] In view of the above situation, in the past, data differentiation checks were generally performed manually during the software launch process. For example, the differences between the interface information (JSON) and the governance file (Excel) were manually checked. However, this method may introduce many problems, such as (1) poor performance: manually checking the differences between the interface information (JSON) and the governance file (Excel) is very time-consuming; (2) high error rate: human errors are prone to cause inspection errors, resulting in online problems, such as missing metadata and interface data. Summary of the invention

[0004] In view of this, the present application provides a data differentiation detection method, device, medium and electronic device, the main purpose of which is to overcome the drawbacks of existing manual inspection of data differentiation and improve the efficiency and accuracy of data differentiation detection.

[0005] According to one aspect of the present application, a data difference detection method is provided, for detecting a data difference between a first file in a first data format and a second file in a second data format, comprising the following steps:

[0006] Read the first file and the second file respectively;

[0007] The first file and the second file are parsed and structurally converted respectively: a first processing library of the Java language for the first data format is selected, and based on the tree model provided by the first processing library, the character string of the first file is converted into a Java object to construct a first file hierarchy tree based on the tree model; and a second processing library of the Java language is selected to read the second file, obtain a worksheet object of the second file, and create a map structure to store the worksheet object, and construct a second file dictionary;

[0008] Performing field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extracting field names from the first file hierarchy tree and the second file dictionary respectively, identifying fields with similar semantics using natural language processing technology, and performing field matching based on a similarity algorithm, and further performing content difference detection on fields with successful field matching;

[0009] Differentiation marking is performed based on the comparison results: marking the fields where field matching fails and the field contents where content differences are detected by content differentiation.

[0010] In one implementation, the selecting a first processing library for the first data format in Java language and converting the character string of the first file into a Java object based on a tree model provided by the first processing library to construct a first file hierarchy tree based on the tree model includes:

[0011] Select any processing library among Jackson, Gson and JsonSimple provided by Java language as the first processing library;

[0012] Calling the Parser or ObjectMapper model class provided by the first processing library as a tree model;

[0013] The Java object conversion method provided by the tree model is called to parse the character string of the first file into a Java object, and obtain a first file hierarchy tree based on the tree model.

[0014] In one implementation, the method further includes:

[0015] Determining whether the first file contains nested objects and / or arrays;

[0016] If the first file contains a nested object, converting the nested object into a Java object instance;

[0017] If the first file contains an array, the array is converted into a Java collection.

[0018] In one implementation, calling the second processing library of the Java language to read the second file and obtain the worksheet object of the second file includes: calling the Apache POI library of the Java language to load the second file, obtain the worksheet object of the second file, and traverse each row and each column in the worksheet;

[0019] The step of creating a map structure to store the worksheet object and constructing a second file dictionary includes: creating a Map for each row in the worksheet, wherein the key is the column title and the value is the cell value of the column corresponding to the row, and storing the converted Map of each row into a set to obtain a second file dictionary consisting of multiple Map sets.

[0020] In one implementation, performing field matching according to a similarity algorithm includes:

[0021] Get the first field of the first file hierarchy tree and the second field of the second file dictionary;

[0022] Perform word segmentation and text vectorization processing on the first field and the second field respectively, and obtain the word segmentation vector of the first field and the word segmentation vector of the second field respectively;

[0023] Calculate the similarity between the first field word segmentation vector and the second field word segmentation vector using a similarity algorithm;

[0024] The calculated similarity value is compared with the preset similarity threshold to determine the field matching result.

[0025] In one implementation, further performing content difference detection on the field for which the field matching is successful includes:

[0026] For a first field and a second field that have consistent field matching, further comparing a first data value of the first field with a second data value of the second field;

[0027] Determine the content differentiation detection results based on fuzzy matching, exact matching or priority matching rules.

[0028] In one implementation, the reading of the first file and the second file respectively includes:

[0029] The first file and the second file are uploaded through the file selector, and the file formats of the first file and the second file are identified according to the file suffix matching algorithm, and other format files that are not in the first data format and the second data format are filtered out.

[0030] According to one aspect of the present application, there is provided a data difference detection device for detecting a data difference between a first file in a first data format and a second file in a second data format, comprising:

[0031] A file reading unit, used to read a first file and a second file respectively;

[0032] The parsing and conversion unit is used to parse and convert the structure of the first file and the second file respectively: select a first processing library of the Java language for the first data format, and based on the tree model provided by the first processing library, perform Java object conversion on the character string of the first file to construct a first file hierarchy tree based on the tree model; and call the second processing library of the Java language to read the second file, obtain the worksheet object of the second file, and create a map structure to store the worksheet object, and construct a second file dictionary;

[0033] A detection execution unit, configured to perform field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extract field names from the first file hierarchy tree and the second file dictionary respectively, identify fields with similar semantics using natural language processing technology, perform field matching based on a similarity algorithm, and further perform content difference detection on fields with successful field matching;

[0034] The result processing unit performs differential marking according to the comparison result: marking the fields where the field matching fails and the field contents where the content difference is detected by content differentiation.

[0035] According to one aspect of the present application, a storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data difference detection method when running.

[0036] According to one aspect of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned data difference detection method.

[0037] Through the above-mentioned technical scheme, the present application provides a data differentiation detection method, device, medium and equipment, which are different from the previous manual comparison method. It uses automated means such as file parsing, field matching, content detection, etc. to compare the first file (such as interface information JSON file) with the second file (such as governance file Excel file) to quickly identify the differences between the two. Moreover, it is no longer limited to simple field full matching. Instead, it uses the file tree structure to more deeply parse and compare data structures, including nested JSON objects and arrays, providing more accurate difference detection capabilities.

[0038] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A flow chart of a data difference detection method provided in an embodiment of the present application is shown;

[0041] Figure 2 An example flow chart of a data difference detection method provided in an embodiment of the present application for detecting differences between a JSON file and an Excel file is shown;

[0042] Figure 3 A JSON parsing flow chart in an example of a data difference detection method provided by an embodiment of the present application for detecting differences between a JSON file and an Excel file is shown;

[0043] Figure 4 A schematic diagram of a JSON hierarchical tree structure in an example of a data difference detection method provided in an embodiment of the present application for detecting differences between a JSON file and an Excel file is shown;

[0044] Figure 5 An Excel reading flow chart in an example of a data difference detection method provided by an embodiment of the present application for detecting differences between a JSON file and an Excel file is shown;

[0045] Figure 6 A diagram showing a dictionary of a MAP structure corresponding to Excel in an example of a data difference detection method provided by an embodiment of the present application for detecting differences between a JSON file and an Excel file;

[0046] Figure 7 A field matching flow chart in an example of a data difference detection method provided by an embodiment of the present application for detecting differences between a JSON file and an Excel file is shown;

[0047] Figure 8 A schematic diagram of the structure of a data difference detection device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0049] As analyzed above, the existing manual method of checking the differences between the interface information (JSON) and the governance file (Excel) will bring about problems such as long time consumption and low accuracy. In order to solve the limitations of the existing technical solutions, the embodiment of the present application proposes an intelligent data differentiation detection method, which can automatically identify the differences between the interface information (JSON) and the governance file (Excel) content, and mark possible problem areas, thereby improving the review efficiency and accuracy, thereby greatly reducing software launch problems and improving software release progress.

[0050] See also Figure 1 , shows a flow chart of a data difference detection method provided by an embodiment of the present application. The data difference detection method is used to detect the data difference between a first file in a first data format and a second file in a second data format, and includes the following steps:

[0051] S101: Read a first file and a second file respectively.

[0052] In the embodiment of the present application, the first file and the second file are used to distinguish two types of files with different data formats. For example, the first file is a JSON type file and the second file is an Excel type file. The application scenario can be a scenario in which the online materials are reviewed during the online stage of software development. However, those skilled in the art should understand that the embodiments of the present application are not limited to the above-mentioned file type examples and scenario examples. The embodiments of the present application are also applicable to the data differentiation comparison between other types of files in other scenarios. As long as the specific data of the file can be read, the solution provided by the embodiments of the present application can be adopted. For example, the first file is a txt file, the second file is a csv file, and so on. For the convenience of explanation and understanding, the embodiments of the present application are mainly explained by taking the first file as a JSON type file and the second file as an Excel type file as an example.

[0053] The above-mentioned reading of the first file and the second file separately may include: uploading the first file and the second file through a file selector, identifying the file format of the first file and the second file according to a file suffix matching algorithm, and filtering out other format files that are not in the first data format and the second data format. For example, in the scenario of reviewing the online materials during the software development online stage, the JavaFX file selector can be used to upload the interface information (JSON) file and the governance document (Excel) file that need to be reviewed, and the documents are filtered according to the document suffix matching algorithm during uploading to identify the files that need to be reviewed.

[0054] S102: parsing and structural conversion are performed on the first file and the second file respectively.

[0055] Among them, parsing and structural conversion of the first file may include: selecting a first processing library of Java language for the first data format, and based on the tree model provided by the first processing library, converting the string of the first file into a Java object to construct a first file hierarchy tree based on the tree model; parsing and structural conversion of the second file may include: calling a second processing library of Java language to read the second file, obtaining a worksheet object of the second file, and creating a map structure to store the worksheet object, and constructing a second file dictionary.

[0056] The following description takes a JSON file as the first file as an example.

[0057] JSON is a lightweight data format. Java, as a popular programming language, provides a variety of libraries to process JSON data. However, when it comes to fast retrieval of large amounts of JSON data, traditional processing methods will encounter bottlenecks. Therefore, using Java to build an efficient JSON tree structure can achieve fast retrieval. JSON data is usually stored in a nested structure, which makes data organization and access complicated. Using a JSON tree can simplify the data organization and access process and improve retrieval efficiency. JSON tree is a method of storing JSON data in a tree structure, where each node represents a JSON object or array. In the tree structure representing JSON data, each node represents an element of JSON, so that the content of the JSON object can be easily accessed through the tree path.

[0058] For example, Java provides processing libraries such as Jackson, Gson, and JsonSimple that can process JSON. Each processing library further provides a tree model. For example, the Jackson processing library provides Parser and ObjectMapper tree models. In actual operation, you can choose any processing library and use the tree model provided by it to parse JSON. For example, first, select any processing library among Jackson, Gson, and JsonSimple provided by the Java language as the first processing library; then, call the Parser or ObjectMapper model class provided by the first processing library as the tree model; finally, call the Java object conversion method provided by the tree model (such as the readValue method of the ObjectMapper class) to parse the JSON string into a Java object to obtain a JSON tree based on the tree model.

[0059] An Excel file is taken as an example for description as the second file.

[0060] In an embodiment of the present application, an Excel file can be parsed based on a Map structure. Map is an abstract data structure consisting of a series of key-value pairs, and each key appears at most once. The keys in a Map can be of any type, but each key can only be mapped to one value, and the value mapped to each key is also unique. Map is unordered, that is, the storage order of elements is not necessarily arranged in the order of insertion. Map is a reference data type, which means that if two variables reference the same Map object, operations on one of the variables will affect the other variable.

[0061] In the specific operation, first, call the Apache POI library of the Java language to load the second file, obtain the worksheet object of the second file, and traverse each row and each column in the worksheet; then, create a Map for each row in the worksheet, in which the key is the column title and the value is the cell value of the column corresponding to the row, and store the converted Map of each row into a collection to obtain a second file dictionary composed of multiple Map collections.

[0062] S103: Perform field matching and content difference detection on the first file hierarchy tree and the second file dictionary.

[0063] Among them, field names are extracted from the first file hierarchy tree and the second file dictionary respectively, and natural language processing (NLP) technology is used to identify semantically similar fields, and field matching is performed based on a similarity algorithm. For fields with successful field matching, content differentiation detection is further performed.

[0064] NLP is a tool that uses computer technology to analyze, understand and process natural language with language as the object, including natural language understanding (NLU) and natural language generation (NLG). NLP is an important direction in the field of artificial intelligence, aiming to enable computers to understand and generate natural language. NLP algorithms include tasks such as lexical analysis, syntactic analysis, and semantic analysis, with the goal of enabling computers to process and understand human language. The main tasks of NLP algorithms include lexical analysis, syntactic analysis, and semantic analysis. Lexical analysis involves breaking down text into basic units such as words and punctuation; syntactic analysis focuses on the analysis of sentence structure; and semantic analysis involves understanding the meaning of sentences or texts. In addition, NLP technology also includes basic tasks such as word segmentation, named entity recognition, and word vectors, as well as application tasks such as text generation and text classification. NLP algorithms include but are not limited to maximum matching algorithms, shortest path algorithms, HMM hidden Markov models, etc.

[0065] In an embodiment of the present application, after obtaining the first field of the first file hierarchy tree and the second field of the second file dictionary according to the NLP algorithm, the similarity between the two fields can be calculated by a similarity calculation method, and the field matching results of the two can be determined according to pre-set matching rules.

[0066] In one implementation, the specific process of performing field matching according to a similarity algorithm may include: obtaining a first field from a first file hierarchy tree and a second field from a second file dictionary; performing word segmentation and text vectorization on the first field and the second field, respectively, to obtain a first field word segmentation vector and a second field word segmentation vector, respectively; using a similarity algorithm to calculate the similarity between the first field word segmentation vector and the second field word segmentation vector; comparing the calculated similarity value with a pre-set similarity threshold to determine a field matching result.

[0067] After the field matching is performed, the field with successful field matching is further subjected to content differentiation detection, that is, the content differentiation detection result is determined by judging whether the specific content of the field is consistent. In one implementation, the specific process of further performing content differentiation detection on the field with successful field matching may include: for the first field and the second field with consistent field matching, further comparing the first data value of the first field with the second data value of the second field; and determining the content differentiation detection result according to fuzzy matching, complete matching or priority matching rules.

[0068] S104: Perform differential marking according to the comparison result.

[0069] Among them, the fields where field matching fails and the field contents where content differences are detected by content differentiation are marked.

[0070] It can be understood that if the field match fails, it is determined that there is a difference, and if the field match succeeds, it is necessary to further determine the content difference, and if the content is different, it is also determined that there is a difference. Therefore, when marking, it is necessary to mark both the field match failure and the content difference detection, so as to fully and completely reflect the data difference between the two files.

[0071] The following is an example of detecting the difference between a JSON file and an Excel file to illustrate the data difference detection method provided in the embodiment of the present application.

[0072] See also Figure 2 , shows an example flow chart of a data difference detection method provided in an embodiment of the present application for detecting differences between JSON files and Excel files.

[0073] S200: Review begins;

[0074] S201: Upload JSON file;

[0075] S202: The document suffix matching algorithm verifies whether it is a JSON file. If so, execute S203; otherwise, return to S201;

[0076] S203: JSON parsing;

[0077] S204: Upload the Excel file;

[0078] S205: The document suffix matching algorithm verifies whether it is an Excel file. If so, execute S206; otherwise, return to S205;

[0079] S206: Excel read;

[0080] S207: Field name resolution;

[0081] S208: Field comparison;

[0082] S209: Create a rule base;

[0083] S210: data value comparison;

[0084] S211: Marking differences;

[0085] S212: Report output;

[0086] S213: Does the report output meet the forecast? If not, execute S214; if yes, execute S215;

[0087] S214: the rule base is updated, and then the process returns to S210;

[0088] S215: Review completed.

[0089] In summary, the above-mentioned whole process includes five aspects: reading files (S201-S202, S204-S205), file parsing (S203, S206), field matching (S207-S208), content difference detection (S209-S210), difference marking and report generation (S211-S214). These aspects are described in detail below.

[0090] 1. File reading

[0091] For example, a JavaFX file selector is used to upload interface information (JSON) files and governance document (Excel) files that need to be reviewed. When uploading, the files are filtered according to the document suffix matching algorithm to identify the files that need to be reviewed. In addition, the reading method is not limited to the above JavaFX file selector, and other reading methods can be used as long as the file can be read.

[0092] 2. File Analysis

[0093] (1) JSON parsing

[0094] JSON parsing can be understood as converting JSON data into a hierarchical structure of an object tree to facilitate subsequent field matching.

[0095] See also Figure 3 , JSON parsing flowchart, including the following steps:

[0096] S300: JSON parsing starts;

[0097] S301: select library;

[0098] S302: Create a model class;

[0099] S303: Parse the JSON string;

[0100] S304: Is the analysis successful? If not, execute S305-S306, otherwise execute S307-S309;

[0101] S305: Exception handling, printing exception information;

[0102] S306: Re-upload the JSON after adjusting according to the exception;

[0103] S307: Processing nested objects and arrays;

[0104] S308: Is the processing successful? If not, execute SS305; if yes, execute S309;

[0105] S309: JSON parsing is completed.

[0106] The above process is described as follows:

[0107] First, you need to choose a JSON processing library. For example, in Java, there are multiple processing libraries that can process JSON, such as Jackson, Gson, JsonSimple, etc. These libraries provide methods to convert JSON into Java objects. Next, you need to create model classes, such as defining Java model classes (POJO-Plain Old Java Object). These classes can be mapped to the structure in JSON, including attribute names and types. If JSON contains nested objects or arrays, then a corresponding model class is required for each type of object. Then, parse the JSON string, that is, use the method of the selected library to parse the JSON string into a Java object. For example, if you use the Jackson library, you can use the readValue method of the ObjectMapper class to complete the parsing of the JSON string. If you use the Gson processing library, you can use the fromJson method; if you use the JsonSimple processing library, you can use the parse method of JSONParser. Among them, you need to process nested objects and arrays. If the file contains nested objects, they will be converted into corresponding Java object instances. If JSON contains arrays, they will be converted into Java collections (such as List). Accessing and manipulating objects: Once the JSON conversion is complete, you can manipulate these objects like normal Java objects, for example, you can access fields, call methods, or modify values. In addition, exceptions need to be handled during the JSON parsing process. For example, during the parsing process, you may encounter various errors such as incorrect JSON format or mismatched field types. Therefore, exception handling should be considered during the conversion.

[0108] The process of parsing a JSON file is the process of building a hierarchical JSON tree.

[0109] JSON is a lightweight data format. Java, as a popular programming language, provides a variety of libraries to process JSON data. However, when it comes to fast retrieval of large amounts of JSON data, traditional processing methods will encounter bottlenecks. Therefore, using Java to build an efficient JSON tree structure can achieve fast retrieval. JSON data is usually stored in a nested structure, which makes data organization and access complicated. Using JSON tree can simplify the data organization and access process and improve retrieval efficiency. JSON tree is a method of storing JSON data in a tree structure, where each node represents a JSON object or array.

[0110] In the tree structure representing JSON data, each node represents an element of JSON, so the content of the JSON object can be easily accessed through the tree path. Figure 4 , is a schematic diagram of the JSON hierarchical tree structure. The root node S includes child nodes A1, A2, and A3. Further, A1 includes B1, B2, and so on. Figure 4 For understanding, assume that the JSON file contains the following content:

[0111] {

[0112] "methods":[{},{},{}],

[0113] "protocols":[},{},{}],

[0114] "outPaths":[},{},{}],

[0115] “products”:[},{},{}]

[0116] }

[0117] In the above JSON example, "services", "protocols", "outPaths", "products", etc. represent Figure 4 Corresponding to A1, A2, A3, A4... etc.

[0118] (2) Excel Reading

[0119] Excel reading is the process of reading an Excel table and converting it into a key-value data frame structure that is easy to process.

[0120] See also Figure 5 , is an Excel reading flow chart, including the following steps S501-S504.

[0121] S501: Select a Java library.

[0122] Choose a suitable Java library to read Excel files. For example, Apache POI is a widely used library that supports multiple Excel file formats, including .xls and .xlsx.

[0123] S502: Load a workbook worksheet.

[0124] For example, load an Excel file using the selected library and obtain the Workbook and Worksheet objects. This allows access to all the data in the Excel file.

[0125] S503: Traverse rows and columns.

[0126] Iterate through each row and column in the worksheet. Typically, the first row might contain column headers, which can serve as keys, and the following rows contain the values ​​associated with those keys.

[0127] S504: Create and store a Map.

[0128] Create a Map structure: For each row, create a Map where the key is the column header and the value is the cell value of the corresponding column of the row. In this way, the data of each row is converted into a Map.

[0129] Store Map: Store each row of converted Map in a List or other collection, so that there is a collection of multiple Maps, each Map represents a row of data in Excel.

[0130] At this point, you can use this Map collection to query and process data. Since Map is stored in the form of key-value pairs, you can quickly find values ​​by key.

[0131] See also Figure 6 , which is a dictionary diagram of the MAP structure corresponding to Excel. Take a specific Excel table as an example Figure 6 To understand.

[0132] Table 1

[0133] method protocol Access path product ……

[0134] In the Excel file shown in Table 1, "method", "protocol", "access path", "product", etc. (which may be in Chinese or English) can represent Figure 6 The key1, key2, key3, key4... shown in the figure are the specific contents under each key, i.e., the values.

[0135] 3. Field Matching

[0136] (1) Feature extraction and preliminary matching

[0137] This step is field name parsing. Field names are extracted from JSON and Excel respectively, and natural language processing technology is used to identify fields with similar semantics.

[0138] For example, in the above example, it is identified that the "methods" field in the JSON file is similar to the "method" in the Excel file, "protocols" is similar to the "protocol", and so on.

[0139] (2) Text preprocessing

[0140] Unify the case of all nodes in the JSON tree, such as A1, A2, A3... field names, remove special characters, and perform stemming or lemmatization to reduce synonym variations.

[0141] Similarly, unify all string keys in the Map after Excel processing, such as Key1, Key2, Key3…

[0142] (3) Field Matching

[0143] Compare the fields in the interface information (JSON) and the governance file (Excel) one by one, fuzzy match similar fields, and mark missing or newly added fields.

[0144] For the field matching process, see Figure 7 . The specific instructions are as follows.

[0145] (1) Data input: Convert the interface information (JSON) into a JSON tree and the management file (Excel) into dictionary elements, such as inputting the key value (A1, A2, A3...) in JSON and the value (Key1, Key2, Key3...) in Excel.

[0146] (2) Text preprocessing: Segment the input text and add special tags [CLS] and [SEP].

[0147] (3) Text vectorization: Use the BERT model to convert the segmented text into a vector of fixed length. Here, the vector expression of [CLS] is used as the vector representation of sentence.

[0148] (4) Similarity calculation: By converting the text into vector representation (vec1 and vec2), we can use methods such as cosine similarity to calculate similarity. For example,

[0149]

[0150] For two vectors vec1 and vec2, cosine similarity measures the angle between them, and its value range is between [-1,1].

[0151] (5) Matching rule (name resolution): Set a threshold τ. If the similarity between two vectors is greater than the threshold, the two items are considered to be matched, otherwise they are not matched.

[0152] For example, the matching rules in the previous name resolution calculate the similarity between key1, key2... and A1, A2... For example, the "method" field name in Excel is calculated similarly to "methods" in Json. If the similarity is greater than the threshold (0.9), the two fields are considered to match successfully.

[0153] 4. Content Difference Detection

[0154] Content differentiation detection, for fields with consistent field matching, further compare their data values ​​to see if they are consistent. After natural language processing, for example, if the similarity between the "method" field name in Excel and the "methods" in Json is greater than the threshold (0.9), further specific matching of the data value (Value) inside is performed. Generally, fuzzy matching or full matching can be used. There are also some special cases where specific matching rules need to be formulated, such as priority. The specific values ​​in Excel may be "high priority", "medium priority", "low priority"; corresponding to "1", "2", "3" in Json.

[0155] 5. Difference marking and report generation

[0156] This part includes the following three aspects:

[0157] (1) Marking differences: During the review process, fields that are inconsistent after feature extraction and preliminary matching, and inconsistent data values ​​after content difference detection are automatically marked.

[0158] (2) Report output: Generate a detailed review report, including a summary of differences and location information of specific differences.

[0159] (3) Rule base update: According to the feedback of matching results, adjust the matching rules to improve the matching accuracy.

[0160] For example, for field matching differences, if the column in Excel does not match the field name in Json, it is equivalent to missing data in the field in Json, and a prompt is given to supplement the field and check the online materials. For field content matching differences, for example, according to the matching rules in the creation rule library, fuzzy matching, or full matching, there are some special needs to formulate specific matching rules. If there is no match, the difference in the field content is prompted and the online materials are checked.

[0161] In summary, the embodiment of the present application proposes a data differentiation detection method, aiming to provide an intelligent detection method. Different from the previous manual comparison method, it uses automated means to compare interface information (JSON) and governance files (Excel) to quickly identify the differences between the two; and, it is no longer limited to simple field full matching, but uses the file tree structure to more deeply parse and compare data structures, including nested JSON objects and arrays, to provide more accurate difference detection capabilities, including data missing, data redundancy, data differences and format errors, etc.; and, for any differences or errors found, natural language processing can be used in combination with matching rules to accurately locate the specific location.

[0162] See also Figure 8 , shows a schematic diagram of the structure of a data difference detection device provided in an embodiment of the present application. The data difference detection device is used to detect the data difference between a first file in a first data format and a second file in a second data format, including:

[0163] A file reading unit 801, used to read a first file and a second file respectively;

[0164] The parsing and conversion unit 802 is used to parse and convert the structure of the first file and the second file respectively: select a first processing library of the Java language for the first data format, and based on the tree model provided by the first processing library, perform Java object conversion on the character string of the first file to construct a first file hierarchy tree based on the tree model; and call a second processing library of the Java language to read the second file, obtain a worksheet object of the second file, create a map structure to store the worksheet object, and construct a second file dictionary;

[0165] The detection execution unit 803 is used to perform field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extract field names from the first file hierarchy tree and the second file dictionary respectively, identify fields with similar semantics using natural language processing technology, perform field matching according to a similarity algorithm, and further perform content difference detection on the fields with successful field matching;

[0166] The result processing unit 804 performs differential marking according to the comparison result: marking the fields where the field matching fails and the field contents where the content difference is detected by content differentiation.

[0167] In one implementation, the parsing and conversion unit 802 is specifically used to select any one of the processing libraries Jackson, Gson, and JsonSimple provided by the Java language as the first processing library; call the Parser or ObjectMapper model class provided by the first processing library as a tree model; call the Java object conversion method provided by the tree model to parse the string of the first file into a Java object, and obtain a first file hierarchy tree based on the tree model.

[0168] In one implementation, the parsing and conversion unit 802 is further used to determine whether the first file contains nested objects and / or arrays; if the first file contains nested objects, convert the nested objects into Java object instances; if the first file contains arrays, convert the arrays into Java collections.

[0169] In one implementation, the parsing and conversion unit 802 is specifically used to call the Apache POI library of the Java language to load the second file, obtain the worksheet object of the second file, and traverse each row and each column in the worksheet; the creation of a map structure to store the worksheet object and construct a second file dictionary includes: creating a Map for each row in the worksheet, where the key is the column title and the value is the cell value of the column corresponding to the row, and storing the converted Map of each row into a set to obtain a second file dictionary consisting of multiple Map sets.

[0170] In one implementation, the detection execution unit 803 is specifically used to obtain the first field of the first file hierarchy tree and the second field of the second file dictionary; perform word segmentation and text vectorization on the first field and the second field, respectively, to obtain the first field word segmentation vector and the second field word segmentation vector, respectively; use a similarity algorithm to calculate the similarity between the first field word segmentation vector and the second field word segmentation vector; compare the calculated similarity value with a pre-set similarity threshold to determine the field matching result.

[0171] In one implementation, the detection execution unit 803 is specifically used to further compare the first data value of the first field with the second data value of the second field for a first field and a second field with consistent field matching; and determine the content differentiation detection result according to fuzzy matching, complete matching or priority matching rules.

[0172] In one implementation, the file reading unit 801 is specifically used to upload the first file and the second file through a file selector, identify the file format of the first file and the second file according to a file suffix matching algorithm, and filter out other format files that are not in the first data format and the second data format.

[0173] An embodiment of the present application further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0174] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0175] Read the first file and the second file respectively;

[0176] The first file and the second file are parsed and structurally converted respectively: a first processing library of the Java language for the first data format is selected, and based on the tree model provided by the first processing library, the character string of the first file is converted into a Java object to construct a first file hierarchy tree based on the tree model; and a second processing library of the Java language is selected to read the second file, obtain a worksheet object of the second file, and create a map structure to store the worksheet object, and construct a second file dictionary;

[0177] Performing field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extracting field names from the first file hierarchy tree and the second file dictionary respectively, identifying fields with similar semantics using natural language processing technology, and performing field matching based on a similarity algorithm, and further performing content difference detection on fields with successful field matching;

[0178] Differentiation marking is performed based on the comparison results: marking the fields where field matching fails and the field contents where content differences are detected by content differentiation.

[0179] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0180] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0181] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0182] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0183] Read the first file and the second file respectively;

[0184] The first file and the second file are parsed and structurally converted respectively: a first processing library of the Java language for the first data format is selected, and based on the tree model provided by the first processing library, the character string of the first file is converted into a Java object to construct a first file hierarchy tree based on the tree model; and a second processing library of the Java language is selected to read the second file, obtain a worksheet object of the second file, and create a map structure to store the worksheet object, and construct a second file dictionary;

[0185] Performing field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extracting field names from the first file hierarchy tree and the second file dictionary respectively, identifying fields with similar semantics using natural language processing technology, and performing field matching based on a similarity algorithm, and further performing content difference detection on fields with successful field matching;

[0186] Differentiation marking is performed based on the comparison results: marking the fields where field matching fails and the field contents where content differences are detected by content differentiation.

[0187] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0188] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0189] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0194] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data differentiation detection method, characterized in that: The method for detecting a data difference between a first file in a first data format and a second file in a second data format comprises the following steps: Read the first file and the second file respectively; Parsing and structurally converting the first file and the second file respectively: selecting a first processing library for the first data format in Java language, and converting the character string of the first file into a Java object based on a tree model provided by the first processing library, so as to construct a first file hierarchy tree based on the tree model; And, selecting a second processing library in Java language to read the second file, obtain a worksheet object of the second file, create a map structure to store the worksheet object, and construct a second file dictionary; Performing field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extracting field names from the first file hierarchy tree and the second file dictionary respectively, identifying fields with similar semantics using natural language processing technology, and performing field matching based on a similarity algorithm, and further performing content difference detection on fields with successful field matching; Differentiation marking is performed based on the comparison results: marking the fields where field matching fails and the field contents where content differences are detected by content differentiation.

2. The method according to claim 1, characterized in that The step of selecting a first processing library in Java language for the first data format and converting the character string of the first file into a Java object based on a tree model provided by the first processing library to construct a first file hierarchy tree based on the tree model includes: Select any processing library among Jackson, Gson and JsonSimple provided by Java language as the first processing library; Calling the Parser or ObjectMapper model class provided by the first processing library as a tree model; The Java object conversion method provided by the tree model is called to parse the character string of the first file into a Java object, and obtain a first file hierarchy tree based on the tree model.

3. The method according to claim 2, characterized in that Also includes: Determining whether the first file contains nested objects and / or arrays; If the first file contains a nested object, converting the nested object into a Java object instance; If the first file contains an array, the array is converted into a Java collection.

4. The method according to claim 1, characterized in that: The calling of the second processing library of the Java language to read the second file and obtain the worksheet object of the second file includes: calling the Apache POI library of the Java language to load the second file, obtain the worksheet object of the second file, and traverse each row and each column in the worksheet; The step of creating a map structure to store the worksheet object and constructing a second file dictionary includes: creating a Map for each row in the worksheet, wherein the key is the column title and the value is the cell value of the column corresponding to the row, and storing the converted Map of each row into a set to obtain a second file dictionary consisting of multiple Map sets.

5. The method according to claim 1, characterized in that: The field matching according to the similarity algorithm includes: Get the first field of the first file hierarchy tree and the second field of the second file dictionary; Perform word segmentation and text vectorization processing on the first field and the second field respectively, and obtain a word segmentation vector of the first field and a word segmentation vector of the second field respectively; Calculate the similarity between the first field word segmentation vector and the second field word segmentation vector using a similarity algorithm; The calculated similarity value is compared with the preset similarity threshold to determine the field matching result.

6. The method according to claim 1, characterized in that The further performing of content difference detection on the field with successful field matching includes: For a first field and a second field that have consistent field matching, further comparing a first data value of the first field with a second data value of the second field; Determine the content differentiation detection results based on fuzzy matching, exact matching or priority matching rules.

7. The method according to claim 1, characterized in that The reading of the first file and the second file respectively comprises: The first file and the second file are uploaded through the file selector, and the file formats of the first file and the second file are identified according to the file suffix matching algorithm, and other format files that are not in the first data format and the second data format are filtered out.

8. A data differentiation detection device, characterized in that: The method is used to detect a data difference between a first file in a first data format and a second file in a second data format, comprising: A file reading unit, used to read a first file and a second file respectively; The parsing and conversion unit is used to parse and convert the structure of the first file and the second file respectively: select a first processing library of the Java language for the first data format, and based on the tree model provided by the first processing library, perform Java object conversion on the character string of the first file to construct a first file hierarchy tree based on the tree model; and call the second processing library of the Java language to read the second file, obtain the worksheet object of the second file, and create a map structure to store the worksheet object, and construct a second file dictionary; A detection execution unit, configured to perform field matching and content difference detection on the first file hierarchy tree and the second file dictionary: extract field names from the first file hierarchy tree and the second file dictionary respectively, identify fields with similar semantics using natural language processing technology, perform field matching based on a similarity algorithm, and further perform content difference detection on fields with successful field matching; The result processing unit performs differential marking according to the comparison result: marking the fields where the field matching fails and the field contents where the content difference is detected by content differentiation.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images