Refined data problem analysis method and system

Abstract

The embodiment of the present application provides a fine data problem analysis method and system, and belongs to the technical field of computer data screening. The computer data screening technology comprises the following steps: obtaining input analysis constraints, wherein the analysis constraints comprise the types of data elements, the data volume of data elements and the data constraints between data elements; obtaining data text to be analyzed; searching the data elements in the data text; and searching and analyzing defects of the data text according to the types, data volume and data constraints of the data elements. The analysis method and system decompose the analysis constraints input by the user, then search the data text according to the decomposed data elements, and obtain the defects of the data text. Compared with the existing technology which relies on artificial targeted coding rules and artificial screening methods, the analysis method and system provided by the present application have higher freedom and universality because they directly use the input analysis constraints for screening.

Description

Technical Field

[0001] This invention relates to the field of computer data screening technology, and more specifically to a refined data problem analysis method and system. Background Technology

[0002] The research and application of data quality management and data problem analysis have gradually become a focus of attention. Currently, data quality management and data problem analysis have become key links in data-driven decision-making and business optimization. Various industries and fields require reliable and efficient methods and tools to handle and resolve data quality issues. Commonly used data management methods in existing technologies generally fall into two categories: one is direct manual verification and analysis, which is inefficient and time-consuming; the other is direct customization through algorithmic rules, which is costly and lacks versatility because it requires customization for each rule. Summary of the Invention

[0003] The purpose of this invention is to provide a refined data problem analysis method and system that can accurately screen for defects in data text according to user needs.

[0004] To achieve the above objectives, embodiments of the present invention provide a refined data problem analysis method, including:

[0005] Obtain the input analysis constraints, wherein the analysis constraints include the types of data elements, the amount of data elements, and the data constraints between data elements;

[0006] Obtain the text data to be analyzed;

[0007] Retrieve the data element from the data text;

[0008] Based on the type of data element, the amount of data, and the data constraints, the defects of the data text are retrieved and analyzed.

[0009] Optionally, retrieving the data element from the data text includes:

[0010] Define valid and invalid regions in the data text;

[0011] The valid area is selected using a rectangular bounding box to obtain the corresponding valid data area;

[0012] Obtain the data fields of the first two rows or first two columns of the valid data area;

[0013] The data field is matched with the keywords of the data element to determine the position of the data element in the data text.

[0014] Optionally, the data element types include keyword sequences, which include multiple keywords connected by a preset connector.

[0015] Optionally, the method further includes:

[0016] Determine whether there are consecutive sequence numbers for multiple keywords near the end of the keyword sequence;

[0017] If it is determined that multiple keywords near the end of the keyword sequence have consecutive sequence number values, an amplification operation is performed on the keywords with multiple consecutive sequence number values ​​to obtain a new keyword sequence.

[0018] Optionally, the data fields of the first two rows or the first two columns of the valid data area are obtained, including:

[0019] Retrieve the data from the first two rows or the first two columns;

[0020] Determine whether purely encoded data exists in the data field;

[0021] If it is determined that the pure encoded data does not exist in the data field, the data field is used as the field to be matched.

[0022] Optionally, matching the data field with the keywords of the data element includes:

[0023] The fields to be matched are converted into corresponding word vectors based on a preset corpus;

[0024] The field to be matched is matched with each keyword of the data element using word vector matching to obtain the corresponding semantic distance;

[0025] The matching result between the data element and the field to be matched is determined based on the semantic distance.

[0026] Optionally, word vector matching is performed between the field to be matched and each keyword of the data element to obtain the corresponding semantic distance, including:

[0027] Calculate the semantic distance between the field to be matched and the category keyword;

[0028] Determining the matching result between the data element and the field to be matched based on the semantic distance includes:

[0029] Determine if the same field to be matched has a semantic distance of 0;

[0030] If the same field to be matched has a semantic distance of 0, the data element and the field to be matched are determined to be matched.

[0031] Optionally, determining the matching result between the data element and the field to be matched based on the semantic distance includes:

[0032] If it is determined that there is no semantic distance of 0 for the same field to be matched, the data elements are numbered in ascending order of semantic distance to obtain the first number of the data element;

[0033] Obtain the data fields that are not in the first two rows or the first two columns corresponding to the field to be matched;

[0034] Calculate the average distance between the acquired data field and the serial number keyword of each data element;

[0035] The data elements are numbered in ascending order of average distance to obtain the second number of the data elements;

[0036] Calculate the comprehensive correlation coefficient between the data element and the field to be matched based on the first number and the second number;

[0037] The data element with the lowest correlation coefficient is selected to be matched with the field to be matched.

[0038] Optionally, the comprehensive correlation coefficient between the data element and the field to be matched is calculated based on the first number and the second number, including:

[0039] The comprehensive correlation coefficient is calculated according to formula (1).

[0040] θ i =αN1 i +βN2 i (1)

[0041] Where, θ i N1 is the comprehensive correlation coefficient of the i-th data element. i N2 is the first number of the i-th data element. i It is the second number of the i-th data element.

[0042] On the other hand, the present invention also provides a sophisticated data problem analysis system, the analysis system including a processor configured to execute the analysis methods described above.

[0043] Through the above technical solution, the refined data problem analysis method and system provided by this invention decomposes the analytical constraints obtained from user input, and then retrieves data text based on the decomposed data elements, thereby obtaining the defects in the data text. Compared with existing technologies that rely on human-specific coding rules and manual screening methods, the analysis method and system provided by this invention have higher freedom and versatility because they directly use the input analytical constraints for screening.

[0044] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart of a refined data problem analysis method according to one embodiment of the present invention;

[0047] Figure 2 This is a partial flowchart of a refined data problem analysis method according to one embodiment of the present invention;

[0048] Figure 3 This is a flowchart of a method for filtering data fields in the first two rows or the first two columns according to an embodiment of the present invention;

[0049] Figure 4 This is a flowchart of a method for matching a field to be matched with keywords corresponding to data elements according to an embodiment of the present invention. Detailed Implementation

[0050] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0051] like Figure 1 The diagram shows a flowchart of a refined data problem analysis method according to an embodiment of the present invention. Figure 1 In this analysis, the method may include the following steps:

[0052] In step S10, the input analysis constraints are obtained, wherein the analysis constraints include the type of data element, the amount of data element, and the data constraints between data elements;

[0053] In step S11, the data text to be analyzed is obtained;

[0054] In step S12, data elements are retrieved from the data text;

[0055] In step S13, the defects of the data text are retrieved and analyzed based on the type of data element, the amount of data, and the data constraints.

[0056] In such Figure 1In the method shown, step S10 is used to obtain the input analysis constraints. In this embodiment, the analysis constraints may include the type of data element, the data volume of the data element, and the data constraints between data elements. The data volume of the data element may further include the numerical range and number of data elements. The data constraints may further include the magnitude relationship between the corresponding values ​​in each data element.

[0057] Step S11 is used to obtain the data text to be analyzed. This data text may consist of a list of cells consisting of index numbers and specific values ​​arranged after the index numbers.

[0058] Based on the data text from step S11, step S12 can be used to retrieve data elements from the data text. Since the data text may contain data of multiple categories, and since the data text is generally arranged in an index-value format, the arrangement of data of a single category will ultimately form a rectangular array. Therefore, in this embodiment, before performing the retrieval, the analysis method may further include, for example... Figure 2 The steps shown are described. Figure 2 In addition, the analytical method may also include the following steps:

[0059] In step S20, valid and invalid regions are defined in the data text;

[0060] In step S21, a rectangular frame is used to select the valid area to obtain the corresponding valid data area;

[0061] In step S22, the data fields of the first two rows or the first two columns of the valid data area are obtained;

[0062] In step S23, the data fields are matched with the keywords of the data elements to determine the position of the data elements in the data text.

[0063] In such Figure 2 In the method shown, step S20 is used to determine the valid and invalid regions. The valid region can be used to represent cells in the data text containing characters. Conversely, the invalid region represents cells in the data text where no characters are present.

[0064] Because data text may contain missing or blank data, meaning that different index values ​​within the same index class may correspond to different amounts of data, the resulting valid regions may be irregular polygons. However, due to the index-value arrangement of the data text, the complete valid region must be rectangular. Therefore, to include missing data within the valid region, this method can determine the valid data region through step S21: selecting the valid region with a rectangular frame to obtain the corresponding valid data area.

[0065] After identifying the valid data area, matching needs to be performed on this area. Since the typical index arrangement of data elements is in the form of "data element type - index number 1 - index number 2 - ..." or "index number - data element type - index number 1 - index number 2 ...", the first two rows or columns of data fields from the valid data area need to be extracted during matching, i.e., step S22. Finally, matching calculations are performed in step S23 to determine the position of the data element within the data text.

[0066] In this implementation, since the types of data elements are included in the index content, to better match data elements and valid data areas, the types of data elements can include keyword sequences. These keyword sequences can further include multiple keywords connected by preset connectors, and these keywords can be divided into category keywords representing data element types and sequence keywords representing index numbers. Furthermore, considering that the number of consecutive sequence keywords such as "Sequence Number 1, Sequence Number 2, Sequence Number 3" in the index cannot be directly determined, the keyword sequence can be expanded when obtaining it. Specifically, this expansion process can be an amplification operation on the sequence keywords to obtain a new keyword sequence. The specific method for this amplification operation can be, for example, adding keywords such as "Sequence Number 5, Sequence Number 6" in sequential increments.

[0067] After extracting the first two rows or columns of valid data from the data range, further filtering of these data is needed to confirm their validity. Since indexed data is generally composed of Chinese characters or a combination of Chinese characters and codes (numbers, letters, etc.), and the corresponding numerical values ​​are generally composed of codes (numbers, letters, etc.), the method for filtering the first two rows or columns of data in this implementation could include, for example... Figure 3 The steps shown are described. Figure 3 In this context, the method may include:

[0068] In step S30, the data fields of the first two rows or the first two columns are obtained;

[0069] In step S31, it is determined whether there is purely encoded data in the data field;

[0070] In step S32, if it is determined that there is no pure encoded data in the data field, the data field is used as the field to be matched.

[0071] After filtering out the fields to be matched, the methods for matching these fields with the keywords corresponding to the data elements can include, for example: Figure 4The method shown. Specifically, in this Figure 4 In this method, the steps may include:

[0072] In step S40, the field to be matched is converted into the corresponding word vector based on the preset corpus;

[0073] In step S41, word vector matching is performed between the field to be matched and each keyword of the data element to obtain the corresponding semantic distance;

[0074] In step S42, the matching result between the data element and the field to be matched is determined based on the semantic distance.

[0075] Furthermore, since the keyword sequence of a data element is divided into category keywords and sequence keywords, and category keywords best represent the characteristics of the data element, the semantic distance calculated in step S41 can be the semantic distance between the field to be matched and the category keywords.

[0076] The specific process of matching based on this semantic distance can be as follows: First, determine whether there exists a semantic distance of 0 for the same field to be matched. If there is a semantic distance of 0 for the same field to be matched, it means that the field to be matched is highly matched with the data element, so the two can be directly determined to be matched.

[0077] Conversely, it indicates that the user's input keywords are relatively vague, and further fuzzy search should be performed. Specifically, one can first obtain the data fields that are not in the first two rows or the first two columns corresponding to the field to be matched, then obtain the average distance between the data fields and the serial number keywords of each data element, and then number the data elements in ascending order of the average distance to obtain the second number of the data element. Next, calculate the comprehensive correlation coefficient between the data element and the field to be matched based on the first number and the second number. Finally, select the data element with the smallest correlation coefficient to match the field to be matched. The specific calculation method for the comprehensive correlation coefficient can be determined according to the following formula (1).

[0078] θ i =αN1 i +βN2 i (1)

[0079] Where, θ i N1 is the comprehensive correlation coefficient of the i-th data element. i N2 is the first number of the i-th data element. i It is the second number of the i-th data element.

[0080] On the other hand, the present invention also provides a sophisticated data problem analysis system, the analysis system including a processor configured to execute the analysis methods described above.

[0081] Through the above technical solution, the refined data problem analysis method and system provided by this invention decomposes the analytical constraints obtained from user input, and then retrieves data text based on the decomposed data elements, thereby obtaining the defects in the data text. Compared with existing technologies that rely on human-specific coding rules and manual screening methods, the analysis method and system provided by this invention have higher freedom and versatility because they directly use the input analytical constraints for screening.

[0082] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0087] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0088] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0090] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A refined data problem analysis method, characterized in that, The analytical method includes: Obtain the input analysis constraints, wherein the analysis constraints include the types of data elements, the amount of data elements, and the data constraints between data elements; Obtain the text data to be analyzed; Retrieve the data element from the data text; Based on the type of data element, the amount of data, and the data constraints, the defects of the data text are retrieved and analyzed; Retrieving the data element from the data text includes: Define valid and invalid regions in the data text; The valid area is selected using a rectangular bounding box to obtain the corresponding valid data area; Obtain the data fields of the first two rows or first two columns of the valid data area; The data fields are matched with the keywords of the data elements to determine the position of the data elements in the data text; The matching calculation between the data field and the keywords of the data element includes: Based on a pre-defined corpus, the fields to be matched are converted into corresponding word vectors; The field to be matched is matched with each keyword of the data element using word vector matching to obtain the corresponding semantic distance; The matching result between the data element and the field to be matched is determined based on the semantic distance; Determining the matching result between the data element and the field to be matched based on the semantic distance includes: If it is determined that there is no semantic distance of 0 for the same field to be matched, the data elements are numbered in ascending order of semantic distance to obtain the first number of the data element; Obtain the data fields that are not in the first two rows or the first two columns corresponding to the field to be matched; Calculate the average distance between the acquired data field and the serial number keyword of each data element; The data elements are numbered in ascending order of average distance to obtain the second number of the data elements; Calculate the comprehensive correlation coefficient between the data element and the field to be matched based on the first number and the second number; The data element with the lowest correlation coefficient is selected to be matched with the field to be matched.

2. The method according to claim 1, characterized in that, The data element types include keyword sequences, which include multiple keywords connected by a preset connector. The keywords are divided into category keywords that represent the data element types and index number keywords that represent the index sequence number.

3. The method according to claim 2, characterized in that, The method further includes: The sequenced keywords are amplified to obtain a new keyword sequence.

4. The method according to claim 2, characterized in that, Retrieve the data fields of the first two rows or first two columns of the valid data area, including: Retrieve the data from the first two rows or the first two columns; Determine whether purely encoded data exists in the data field; If it is determined that the pure encoded data does not exist in the data field, the data field is used as the field to be matched.

5. The method according to claim 2, characterized in that, The field to be matched is matched with each keyword of the data element using word vector matching to obtain the corresponding semantic distance, including: Calculate the semantic distance between the field to be matched and the category keyword; Determining the matching result between the data element and the field to be matched based on the semantic distance includes: Determine if the same field to be matched has a semantic distance of 0; If the same field to be matched has a semantic distance of 0, the data element and the field to be matched are determined to be matched.

6. The method according to claim 1, characterized in that, The comprehensive correlation coefficient between the data element and the field to be matched is calculated based on the first number and the second number, including: The comprehensive correlation coefficient is calculated according to formula (1). ，（1） in, For the first The comprehensive correlation coefficient of each data element. For the first The first number of each data element For the first The second number of each data element.

7. A sophisticated data problem analysis system, characterized in that, The analysis system includes a processor configured to perform the method as described in any one of claims 1 to 6.

Images