Data quality inspection method and system and related equipment

Abstract

The embodiment of the invention provides a data quality inspection method and system and related equipment, and relates to the technical field of data processing, and the method comprises the steps: obtaining a data boundary and basic information corresponding to a target quality inspection object; according to the data boundary and the basic information, executing a target cutting operation on the target quality inspection object to generate a target data element; according to the target data element, a target quality inspection data set is constructed, and the granularity of the target data element is larger than or equal to the preset granularity; and executing a target quality inspection operation on the target quality inspection data set to determine target abnormal data. In this way, the manual participation degree in the data quality inspection process can be reduced, and the quality inspection efficiency and the quality inspection precision of the data are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular, to a data quality inspection method, system and related equipment. Background Art

[0002] The key to archival digital resources lies in data quality management, and in accordance with the inherent connections of the archives, the process of establishing the relationship between catalog data and digital images, as well as the process of establishing corresponding catalog and image databases.

[0003] In related technologies, data quality inspection mostly relies on manual work, and the above method has problems such as low quality inspection efficiency and poor accuracy.

[0004] Therefore, a new technical solution is urgently needed to solve the above technical problems. Summary of the invention

[0005] According to the embodiments of the present application, a data quality inspection method, system and related equipment are provided, which can reduce the manual participation in the data quality inspection process and improve the data quality inspection efficiency and quality inspection accuracy. In a first aspect of the present application, a data quality inspection method is provided, comprising: Obtain the data boundaries and basic information corresponding to the target quality inspection object; According to the data boundary and basic information, a target cutting operation is performed on the target quality inspection object to generate target data elements; According to the target data elements, a target quality inspection data group is constructed, wherein the granularity of the target data elements is greater than or equal to a preset granularity; For the target quality inspection data group, a target quality inspection operation is performed to determine the target abnormal data.

[0006] In some feasible implementations, the above basic information includes: data scale information, batch information, and / or time information.

[0007] In some feasible implementations, the above-mentioned target quality inspection operation includes: a directory quality inspection operation, an image quality inspection operation, and / or a directory and image quality inspection operation.

[0008] In some feasible implementations, the above-mentioned directory quality check operations include: required item quality check, length quality check, multi-level same field consistency quality check, multi-level entry matching quality check, multi-level start and end page number quality check, typo quality check, date quality check, digit quality check, recorded content quality check, serial number quality check, recorded dictionary item quality check, and / or repeated word quality check.

[0009] In some feasible implementations, the image quality inspection operation includes: image color mode quality inspection, image content quality inspection, image angle quality inspection, image defect quality inspection, image DPI quality inspection, image size quality inspection, format quality inspection, KB value quality inspection, metadata item missing quality inspection, readability quality inspection, and / or image storage structure quality inspection.

[0010] In some feasible implementations, the above directory and image quality inspection operations include: Determine the number of pictures to be recorded according to the target catalogue; Determine the actual number of frames according to the target image; When the recorded number of frames is inconsistent with the actual number of frames, it is determined that there is a data anomaly.

[0011] In some feasible implementations, the above-mentioned image storage structure quality inspection includes: When the mean square error of the target image is greater than a preset mean square error threshold, and / or the mean absolute error is greater than a preset mean absolute error threshold, it is determined that data anomaly exists; Where the mean square error is determined based on the following code: import numpy as np mse = np.mean((original_image - reconstructed_image) ** 2); The mean absolute error is determined based on the following code: mae = np.mean(np.abs(original_image - reconstructed_image)).

[0012] In a second aspect of the present application, a data quality inspection system is provided, comprising: An acquisition unit, used to acquire data boundaries and basic information corresponding to the target quality inspection object; A first execution unit, configured to perform a target cutting operation on a target quality inspection object according to data boundaries and basic information to generate target data elements; A construction unit, used to construct a target quality inspection data group according to the target data element, wherein the granularity of the target data element is greater than or equal to a preset granularity; The second execution unit is used to perform a target quality inspection operation on the target quality inspection data group to determine target abnormal data.

[0013] In a third aspect of the present application, an electronic device is provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the data quality inspection method as described in any one of the above items when the processor is running.

[0014] In a fourth aspect of the present application, a computer storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the data quality inspection method as described in any of the above items when running.

[0015] The embodiment of the present application provides a data quality inspection method, system and related equipment, wherein the method includes: obtaining the data boundary and basic information corresponding to the target quality inspection object; performing a target cutting operation on the target quality inspection object to generate a target data element according to the data boundary and basic information; constructing a target quality inspection data group according to the target data element, wherein the granularity of the target data element is greater than or equal to the preset granularity; performing a target quality inspection operation on the target quality inspection data group to determine the target abnormal data. The present application can reduce the degree of manual participation in the data quality inspection process and improve the data quality inspection efficiency and quality inspection accuracy.

[0016] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein: Figure 1 A schematic diagram of a process of data quality inspection method provided in an embodiment of the present application; Figure 2 A schematic block diagram of a target quality inspection operation provided in an embodiment of the present application; Figure 3 A structural schematic diagram of a data quality inspection system provided in an embodiment of the present application; Figure 4 A structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0019] In addition, the term "and / or" in this article is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0020] According to a first aspect of an embodiment of the present application, a data quality inspection method is provided, which is applicable to archival data with high requirements on data quality. Figure 1 A process diagram of a data quality inspection method 100 provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method 100 includes: Step S110, obtaining data boundaries and basic information corresponding to the target quality inspection object.

[0021] Exemplarily, before obtaining the data boundary corresponding to the target quality inspection object, a target import operation can be performed on the target quality inspection object to construct a target database. Specifically, the target import operation can be performed on the batch number information, batch name information, target project name information, import time information, and / or import quantity information of the target quality inspection object. The target import operation can display the tasks being imported in real time. The target database can include two types of databases: a database to be stored and a database that has been stored.

[0022] Exemplarily, the data boundary can be determined based on the target project information input by the target user based on a preset template. The target project information can include: item level information, time information, and / or name information, etc. The target project can be established based on the dimension of the archive data fonds and / or based on the dimension of the catalog of the archive data.

[0023] It should be noted that the above data boundaries include: project boundaries, file boundaries, and / or document boundaries.

[0024] Exemplarily, the above basic information includes: data scale information, batch information, and / or time information.

[0025] Step S120: performing a target cutting operation on the target quality inspection object according to the data boundary and the basic information to generate a target data element.

[0026] Exemplarily, a target cutting operation may be performed on the target quality inspection object according to the data boundary and the basic information to generate a target data element, wherein the granularity of the target data element is greater than or equal to a preset granularity.

[0027] Step S130: construct a target quality inspection data group according to the target data elements.

[0028] Exemplarily, a target quality inspection data group may be generated based on target data elements having a granularity greater than or equal to a preset granularity.

[0029] Exemplarily, when the target quality inspection object is stored according to the preset directory rules, the target directory element corresponding to the target cutting operation can be determined according to the preset granularity, and the target cutting operation is performed on the target quality inspection object according to the target directory element to generate the target data element. The preset directory rules can be defined according to the actual needs of the target user.

[0030] Specifically, you can set, edit, manage fields, add notes, etc. based on the N-level entry and the original directory, where the N-level entry can be associated with a form. You can choose to import a template to maintain consistency with the source directory, and you can save the imported template to the template library.

[0031] Specifically, the top-level directory can be used as the minimum granularity, and according to the preset hierarchical structure directory elements, the target cutting operation is performed on the target quality inspection object to generate the target data element, and the target data elements with granularity equal to or greater than the preset granularity are combined to generate the target quality inspection data group. Among them, the preset granularity is negatively correlated with the target user's quality inspection accuracy requirement for the target quality inspection object, that is, the higher the target user's quality inspection accuracy requirement for the target quality inspection object, the smaller the preset granularity.

[0032] Specifically, according to the batch number and primary key starting range corresponding to the target batch element, a target cutting operation can be performed on the target quality inspection object to generate a target data element, and the target data elements with a granularity greater than or equal to a preset granularity can be combined to generate a target quality inspection data group.

[0033] Exemplarily, the scope of executing the target cutting operation may be determined according to a time attribute element, wherein the time attribute element includes: a timestamp element or other time identification element.

[0034] Exemplarily, a target cutting operation may be performed on the target quality inspection object according to the time attribute element to generate a target data element. The preset granularity is negatively correlated with the target user's quality inspection accuracy requirement for the target quality inspection object, that is, the higher the target user's quality inspection accuracy requirement for the target quality inspection object, the smaller the preset granularity.

[0035] Specifically, according to the time attribute corresponding to the time attribute element, a target cutting operation can be performed on the target quality inspection object to generate a target data element, and the target data elements with a granularity greater than or equal to a preset granularity can be combined to generate a target quality inspection data group.

[0036] For example, the target cutting operation can be performed on the target quality inspection object according to time attributes such as day, week, month, etc. to generate target data elements, and the target data elements with granularity greater than or equal to the preset granularity can be combined to generate a target quality inspection data group.

[0037] Exemplarily, the target data type element and / or format element corresponding to the target cutting operation can be determined according to the preset granularity, wherein the target data type element and / or format element include: text data, image data, audio data, etc., or database tables in different formats. The preset granularity is negatively correlated with the target user's quality inspection accuracy requirement for the target quality inspection object, that is, the higher the target user's quality inspection accuracy requirement for the target quality inspection object, the smaller the preset granularity.

[0038] Exemplarily, a target cutting operation can be performed on the above-mentioned target quality inspection object according to the target data type, and / or the data type and / or format corresponding to the format element to generate target data elements, and the target data elements with a granularity greater than or equal to a preset granularity can be combined to generate a target quality inspection data group.

[0039] Therefore, the above method is conducive to accurately selecting the target cutting operation to perform on the target quality inspection object according to the corresponding data boundary, batch information, and / or time information to generate the target data element, so as to accurately generate the target data element, and accurately construct the target quality inspection data group according to the target data element, thereby improving the quality inspection efficiency and quality inspection accuracy of the target abnormal data.

[0040] Step S140: performing a target quality inspection operation on the target quality inspection data group to determine target abnormal data.

[0041] It should be noted that target quality inspection tasks can be established according to the actual needs of target users, and / or historical and ongoing target quality inspection records can be viewed.

[0042] For example, the target user can select to perform a target quality inspection operation on the object in the target database through the existing options. The target database may include: a database to be stored and a formal database.

[0043] For example, Figure 2 A schematic block diagram of a target quality inspection operation provided in an embodiment of the present application. Figure 2 As shown, in some feasible implementations, the above-mentioned target quality inspection operation includes: directory quality inspection operation, image quality inspection operation, and / or directory and image quality inspection operation.

[0044] It should be noted that, thus, the above method performs directory quality inspection operations, image quality inspection operations, and / or directory and image quality inspection operations on the above-mentioned target quality inspection objects, thereby achieving all-round quality inspection of the data of the target quality inspection objects based on directory dimensions, image dimensions, and / or directory and image consistency dimensions, so as to improve the quality inspection accuracy of abnormal data.

[0045] In some feasible implementations, the above-mentioned catalog quality check may include: required item quality check, length quality check, multi-level same field consistency quality check, multi-level entry matching quality check, multi-level start and end page number quality check, typo quality check, date quality check, digit quality check, recorded content quality check, serial number quality check, recorded dictionary item quality check, and / or repeated word quality check.

[0046] Exemplarily, the above-mentioned necessary quality inspection can be performed according to the rules and / or standards pre-set by the target user. Alternatively, the above-mentioned necessary quality inspection can also be performed based on industry specifications, file management requirements, and / or the needs of a specific project.

[0047] Specifically, in the process of performing the above-mentioned mandatory field quality check, the corresponding mandatory fields in each data entry can be identified based on data parsing and extraction technology, and quality checks can be performed to see whether these fields have values ​​and comply with the specified format or content requirements.

[0048] For example, for personnel file data, "name", "ID number", "date of employment", etc. can be set as mandatory items. Each file record data can be traversed to check whether these fields are complete and the data is valid. For book file data, mandatory items can include: "book title", "author", "publisher", "publication year", "ISBN number", etc. For corporate contract file data, mandatory items can include: contract number, names of both parties, signing date, contract amount, summary of main terms, etc.

[0049] For example, the length quality check may include: according to the field type of character type or overlong character type in the field setting, checking whether the character length of the recorded content of the field of the checked data is within the set length range, and if the character is overlong, marking it as abnormal data. The setting rule of the character length is: one Chinese character occupies 2 characters and one English character occupies 1 character.

[0050] Specifically, during the quality inspection, the actual length of the field to be inspected can be obtained based on the string processing function, and the actual length can be compared with the preset length to implement the length quality inspection.

[0051] For example, for the "Product Name" field, set its length to no more than 50 characters, calculate the actual character length of each product name, and determine whether it exceeds 50 characters. It is understandable that in addition to the above "Product Name" field, it can also include: "Remarks" field, "Username field", "Password" and so on.

[0052] Exemplarily, the above-mentioned multi-level same field consistency quality check may include: for the multi-level directory, checking the retention level and / or the retention period.

[0053] Specifically, character-by-character comparison or a hash algorithm may be used to achieve the above-mentioned consistency quality check of the same fields.

[0054] For example: For two adjacent levels of a multi-level directory, if the retention level is marked in the field tag of the field settings, the levels of the adjacent levels need to be checked, and the level of the previous level must be higher than or equal to the highest level of the next level, otherwise it is marked as abnormal data. If there is no level, it is empty. The retention level is determined according to the range of values ​​from high to low in the settings. For two adjacent levels of a multi-level directory, if the retention period is marked in the field tag of the field settings, the retention period of the adjacent levels needs to be checked, and the retention period of the previous level must be higher than or equal to the highest retention period of the next level, otherwise it is marked as abnormal data. If there is no retention period, it is empty. The retention period level is determined according to the range of values ​​from high to low in the settings.

[0055] Exemplarily, the above-mentioned multi-level item matching quality inspection may include: for the multi-level directory, checking according to the checked items. If the corresponding upper directory cannot be found in the lower directory: for example, some directories in the second level are not associated with the first level directory, they are marked as abnormal data. If the corresponding lower directory cannot be found in the upper directory: for example, some directories in the first level have no lower directories, they are marked as abnormal data.

[0056] Exemplarily, the above-mentioned multi-level start and end page number quality inspection may include: for data containing page number attributes, the start and end page number information in the record may be automatically extracted, and its compliance may be verified.

[0057] Specifically, based on digital comparison and logic judgment algorithms, the actual extracted start and end page number information can be compared with the expected page number range information to achieve quality inspection for the continuity and logic of the page numbers. For multi-level directories, non-first-level entries are marked with the start and end page numbers or the items marked with the start and end page numbers in the field settings for inspection.

[0058] Among them, the end page number shall not be less than the start page number. For example, the end page number in a directory cannot be less than the start page number, otherwise it will be marked as abnormal data.

[0059] The continuity check of the start and end page numbers includes: whether the start and end page numbers of all items in the same volume are continuous. If they are not continuous, they are marked as abnormal data. For example, if the start and end page numbers of the first item are 1-3, and the second item is 5-6, 3 and 5 are not continuous, then they are marked as abnormal data. The start and end page numbers of the first item are 1-3, and the second item is 4-6. The start and end page numbers are continuous and correct.

[0060] Among them, the page number duplication in the start and end page numbers of the whole volume includes: query whether there are duplicate page numbers in the whole volume. For example: the start and end page numbers of the first item are 1-3, and the second item is 2-6. The 2nd and 3rd pages of the first item are the same as the 2nd and 3rd pages of the first item, and then it is marked as abnormal data.

[0061] Among them, the missing pages of the start and end page numbers of the whole volume include: checking whether there are page numbers in the whole volume that are not assigned to the items. For example: a volume has three file directories, the first item has the start and end page numbers 1-3, the second item 5-6, the third item 7-8, and page 4 is not assigned to the item, which is marked as abnormal data.

[0062] Exemplarily, the above-mentioned typo quality check may include: AI typo quality check.

[0063] Specifically, a preset AI model can be generated based on natural language processing (NLP) and pre-training to achieve the above-mentioned: AI typo quality inspection. Data can be input into the above-mentioned preset AI model, which analyzes and judges each word or character based on a large amount of text corpus and language knowledge to identify possible typos or spelling errors. Among them, the above-mentioned preset AI model may include: a neural network model based on deep learning, such as: a recurrent neural network (RNN) or a Transformer architecture, which is used to sequence the text to mark possible erroneous locations and / or correct the correct spelling.

[0064] Specifically, the data can be semantically analyzed based on the natural language processing model, the target fields in the text can be traversed, the typos in the target fields can be quality checked based on the check method of the language_tool_python library, and the quality check results can be stored in the target sequence.

[0065] It should be noted that the above-mentioned target fields may include: fields corresponding to industry standards and specifications, fields corresponding to business needs and processes, fields for sensitive data, fields corresponding to historical data and common errors, etc.

[0066] The fields corresponding to the above industry standards and specifications correspond to specific key information fields or terminology fields in different industry fields, which need to pay special attention to the writing accuracy, such as disease name fields, drug name fields, medical operation terminology fields, etc. in the medical industry.

[0067] The fields corresponding to the above business requirements and processes can be defined according to specific business processes and requirements. For example, in the financial field, fields such as amount, interest rate, and financial product name; in the bank's loan contract text, fields such as loan amount, repayment interest rate, and loan term.

[0068] Sensitive data fields can be set based on the importance and sensitivity of the data as a whole. For example, in legal documents, the names of the contracting parties, legal terms in key clauses, the amounts involved and time limits, etc.; in official documents, the policy name, the name of the executive department, key indicators, etc.

[0069] Among them, the fields corresponding to historical data and common errors can be determined based on the analysis of historical data, where the error rate is high and the probability of typos or irregular writing is high. For example: in enterprise archive management, the fields such as employee names, department names, project numbers, etc. that are found in quality inspection often have spelling errors or irregular capitalization.

[0070] It should be noted that the above-mentioned quality inspection results may include: location information of the typo, error type, correction suggestions, etc.

[0071] The position information may include the starting position and ending position index of each wrong character in the data. For example, if a word is "recieve", and its corresponding correct writing should be "receive", the starting position of the position information may be recorded as the 4th character and the ending position as the 5th character.

[0072] It should be noted that the above position information can be determined based on index slicing. For example, in Python, when the original data is stored in a string variable text, the error starting position detected by quality inspection is start_index and the end position is end_index, then the typo text segment can be directly extracted and determined through text[start_index:end_index].

[0073] The above error types may include spelling errors, grammatical errors, inappropriate wording, etc. For example, when the word is written as "recieve", the corresponding error type is a spelling error.

[0074] The correction suggestion may include: correct spelling content or correct grammatical form. For example, when the word is written as "recieve", "receive" may be given as the correct correction suggestion.

[0075] Exemplarily, the date quality check may include: implementing the date quality check according to a date format specification and a preset value range.

[0076] Specifically, the date field in the data can be identified and the date format used can be determined. Based on the date parsing function, the string can be converted into a date object and verified whether it is within the preset value range.

[0077] For example: The end time cannot be less than the start time: According to the field settings, when the same level has both the start time and the end time tags, compare the contents of these two fields. If both fields are not empty at the same time, and the end time is less than the start time, it is marked as abnormal data.

[0078] Exemplarily, the above-mentioned digit quality inspection can be implemented according to a preset digit rule. The above-mentioned preset digit rule can include: checking the field with a zero-padding digit greater than 0 according to the preset number of zero-padding digits in the field tag. If the content length in the field is not equal to the number of zero-padding digits, it is marked as abnormal data. Based on mathematical operations and string processing functions, the value can be extracted and compared with the preset number of digits, and the extraction and verification of the number of digits can be implemented based on regular expressions or value processing functions.

[0079] Exemplarily, the above-mentioned quality inspection of the bibliographic content may include: checking according to the settings of punctuation, allowed Chinese characters, allowed English, and allowed numbers in the field settings. It is also possible to perform semantic quality inspection on the bibliographic content in the data based on text analysis technology, keyword extraction, and semantic understanding algorithms, according to bibliographic rules and standards, and verify the bibliographic content in the data based on relevant knowledge bases or professional dictionaries to check the integrity, logic, accuracy, and compliance of the data.

[0080] Exemplarily, the serial number quality check may include: for data having a serial number attribute, the continuity and uniqueness of the serial number may be quality checked.

[0081] Specifically, according to the field settings, for the field with "number continuity check" set to "yes", and the field operation is set to auto-increment the serial number or the field is set to a combination field and the combination rule can achieve automatic auto-increment of the field, a check is performed.

[0082] For example, the serial number skip check may include: according to the auto-increment rule, according to the primary key sort number, judging whether there is a skipped number in the serial number of the field.

[0083] For example, checking the serial number starting from 1 may include: according to the auto-increment rule, after sorting by the primary key, determining that the first directory serial number of the field starts from 1.

[0084] Specifically, a counter or index mechanism can be constructed to compare the extracted serial number sequence with the existing serial number sequence, and the serial number sequence can be tracked and verified based on the database's auto-increment field, hash table or other data structure. If the serial number is found to be interrupted, repeated or does not conform to the predetermined numbering rules, the above data is marked as abnormal data.

[0085] Exemplarily, a catalog dictionary may be maintained in advance to implement the above-mentioned catalog dictionary item quality inspection. It should be noted that the above-mentioned catalog dictionary items include: catalog items, and standard values ​​or value ranges corresponding to the above-mentioned catalog items. The above-mentioned catalog dictionary items need to be regularly maintained and updated according to the development and changes of the business. In the event of new business types, product categories or catalog names, they can be added to the above-mentioned catalog dictionary items in a timely manner to ensure the accuracy and effectiveness of the catalog dictionary item quality inspection. At the same time, for some dictionary items in outdated or no longer used catalog dictionary items, cleanup and deletion operations can be performed.

[0086] Specifically, for the field set as a dictionary, it is determined whether the recorded content is empty or whether it is the content of the dictionary item. The recorded field in the data can be compared with the recorded dictionary item. If the recorded content is not empty and the field value corresponding to the above recorded field is not within the dictionary range or does not conform to the specified format, the above data is marked as abnormal data.

[0087] Exemplarily, the words in the text data may be counted and compared based on a text processing algorithm and a data structure, such as a hash table or a dictionary, to implement the above-mentioned duplicate word quality check.

[0088] Specifically, based on the lexical analysis and statistical functions in the natural language processing toolkit, the directory text can be scanned word by word, each word can be added to the data structure, and the number of times it appears can be recorded. If the number of times a word appears is greater than a preset number threshold, the target data is marked as duplicate data. The above-mentioned preset number threshold can be set according to actual needs, for example, the above-mentioned preset number threshold is set to 1 time.

[0089] Therefore, by performing quality checks on required items, length, multi-level consistency of identical fields, multi-level entry matching, multi-level start and end page numbers, typos, dates, digits, content, serial numbers, dictionary items, and repeated words on the directory, we can achieve comprehensive and accurate quality checks on the target quality inspection objects, thereby improving the efficiency and accuracy of quality inspection on the target quality inspection objects.

[0090] In some feasible implementations, the above-mentioned image quality inspection operations include: image color mode quality inspection, image content quality inspection, image angle quality inspection, image defect quality inspection, image DPI quality inspection, image size quality inspection, format quality inspection, KB value quality inspection, metadata item missing quality inspection, readability quality inspection, and / or image storage structure quality inspection.

[0091] It should be noted that machine learning can be performed based on a neural network algorithm model to achieve the above-mentioned image color mode quality inspection, image content quality inspection, image angle quality inspection, image defect quality inspection, image DPI quality inspection, image size quality inspection, format quality inspection, KB value quality inspection, metadata item missing quality inspection, readability quality inspection, and / or image storage structure quality inspection.

[0092] Exemplarily, the image color mode quality inspection can be based on a preset deep learning model, image processing library or software tool, read the metadata information of the target image, extract features of the target image to identify the image color mode, and compare the image color mode actually identified and determined with the preset image color mode to determine whether there is a data anomaly based on the image color mode quality inspection. Wherein, if there is a difference between the image color mode and the preset image color mode, it is determined that there is a data anomaly.

[0093] Specifically, the above model can be trained based on a preset image data set to learn different color mode features for feature extraction and classification of target images, identification of image color modes, and image color mode quality inspection to determine whether there are data anomalies. The above preset deep learning model can include: a convolutional neural network (CNN) model.

[0094] The above-mentioned color mode features may include: color distribution features, color histogram features, color space conversion features, etc.

[0095] Specifically, the color distribution feature is used to determine the spatial distribution of different colors in the image. For example, for an RGB image, the model can learn the pixel value distribution of each of the red, green and blue channels, as well as their combination and proportional relationship in different regions. The color histogram feature corresponds to the histogram formed by the distribution of the number of pixels in different color intervals of the image, which is used to reflect the overall color composition of the image. The color space conversion feature is used to determine the parameter changes in the new color space during the process of converting an image from one color space to another.

[0096] The color modes may include: RGB, CMYK, grayscale, HSV, Lab, indexed color mode, bitmap mode, etc. Specifically, the RGB mode is used to determine the distribution of independent pixel values ​​of three color channels, namely, the red channel, the green channel, and the blue channel, and the relationship between the channels. The CMYK mode is used to determine the distribution and proportion of the four colors of cyan (C), magenta (M), yellow (Y), and black (K). The grayscale mode is used to determine brightness information.

[0097] Specifically, the corresponding image color mode may be determined based on the above color features based on labels, feature similarities and differences, feature matching, etc. of the training data.

[0098] Specifically, during the model training phase, based on the color mode categories and corresponding features annotated for each image in the standard image dataset, the correspondence between the features of the annotated images and the color modes can be learned to determine the corresponding image color mode.

[0099] Specifically, the above model can also learn the similarities and differences between the features of different color mode images to determine the relationship between the features and the corresponding image color mode. For example, the color distribution of grayscale mode images is relatively simple, and the color histogram is concentrated in the low brightness and low saturation areas; while RGB mode images have richer color changes and combinations. The above model can determine the color mode category to which it belongs by comparing the features of the input image with the feature templates or distribution laws of different color modes that have been learned.

[0100] Specifically, the above model can extract features from the image based on a structure such as a convolutional layer to obtain a feature vector of the image. Then, the feature vector is matched and compared with feature templates or classifiers of different color modes learned during the training process to determine the color mode of the image. For example, a classification model based on a neural network can output a probability value corresponding to each color mode based on the feature vector based on a fully connected layer and a softmax function, and determine the color mode with the highest probability value as the corresponding image color mode.

[0101] Specifically, the above model can also perform multi-scale feature extraction operations to determine the above image features based on dimensions such as resolution and image block size. Alternatively, the image can also be analyzed based on multiple angles or rotation directions to prevent feature omissions caused by image angles and / or scaling, and to achieve omnidirectional extraction of image features to accurately determine the corresponding image color mode.

[0102] It should be noted that the above model can also be optimized based on fine-tuning training, increasing the diversity of samples, and / or adjusting parameters and model structure.

[0103] The fine-tuning training may include: when the recognition accuracy of the model for certain specific types of color patterns is low, collecting corresponding image data, and fine-tuning the model based on the image data. It should be noted that during the fine-tuning process, only specific parameters are updated to improve the recognition accuracy of the model for the specific types of color patterns.

[0104] The increasing of sample diversity may include: expanding the training data set so that the training set contains data that is more applicable to actual application scenarios.

[0105] The above-mentioned parameter and model structure adjustment may include: adjusting the parameters of the model according to the performance requirements and hardware conditions of the actual application. For example, if the model recognition speed needs to be improved, the scale of the model can be reduced or the parameters such as the step size of the convolution layer can be adjusted. If the model recognition accuracy needs to be improved, the model structure can be modified to add new layers or modules, for example, introducing an attention mechanism to improve the model recognition accuracy.

[0106] Exemplarily, the above-mentioned image content quality inspection can be implemented based on deep learning image recognition technology or traditional image processing algorithms.

[0107] Specifically, the objects, scenes, texts, etc. in the target image can be identified based on the convolutional neural network to determine whether the identified contents of the above objects, scenes, texts, etc. meet the preset standards. For example: when the above target image is a medical image, the above target image is quality inspected to determine whether the above target image contains: target organ features, and / or pathological features; when the above target image is a document image, the above target image is quality inspected to determine whether the above target image contains: clear text features, and / or chart features. When the above target image contains a target object, the above target image is quality inspected to determine whether the above target image contains: complete features of the target object; when the above target image is an image of an artwork, the above target image is quality inspected to determine whether the features contained in the above target image are complete, whether the color features are coordinated, whether the brushstroke features are clear, etc.

[0108] Alternatively, the text information extracted by OCR can be associated with the information in the target image, and the target association result can be quality checked to achieve the above image content quality check. The above target association can include: spatial position association, semantic association, logical association, etc.

[0109] For example, the position information of the text in the target image and the position information of the key visual elements in the target image can be obtained based on image processing technology and OCR technology to achieve spatial position association of the text information and the target image information, and the image content quality inspection can be achieved based on the spatial position association result. The position information can include: coordinate information, and / or area range information, etc., and the key visual elements can include: chart elements, graphic elements, and / or symbol elements, etc.

[0110] Specifically, when the target image is an engineering drawing file image, the corresponding text content can be determined based on OCR recognition. When the spatial position of each part graphic in the above text content, and / or the relative position of each part and the spatial position of each part graphic in the above target image, and / or the relative position of each part have a deviation greater than a preset deviation, it can be determined that there is a data anomaly. Among them, the deviation of the spatial position of each part graphic in the above text content, and / or the relative position of each part and the spatial position of each part graphic in the above target image, and / or the relative position of each part can be determined based on the center distance, overlapping area, etc.

[0111] Exemplarily, semantic analysis can be performed on the text content extracted by OCR, and the target image can be parsed to determine the semantic information of the subject or scene expressed by the target image. When the difference between the semantic parsed content of the above text content and the image parsed content of the target image is greater than a preset difference, it can be determined that there is a data anomaly.

[0112] Specifically, when the target image is a historical document archive image, the location and characters of the historical event extracted by OCR can be compared with the characters, scenes, etc. generated by analyzing the target image. When the difference between the location and main characters of the historical event extracted by OCR and the characters, scenes generated by analyzing the target image is greater than the preset difference, it can be determined that there is a data anomaly.

[0113] Exemplarily, whether there is a logical contradiction can be determined based on the matching degree between the text information and the target image information to achieve the above-mentioned image content quality inspection.

[0114] Specifically, when the target image is a financial file image, the values ​​and calculation relationships of various financial indicators can be determined based on the text information extracted by OCR, and the data and calculation relationships of various financial indicators in the target image can be determined based on the target image. If the values ​​and calculation relationships of various financial indicators determined by the above text information are inconsistent with the data and calculation relationships of various financial indicators determined by the above target image, it can be determined that there is a data anomaly. For example, if the text information indicates that the proportion of income to total income is 30%, and the proportion shown in the pie chart in the target image does not match the above 30%, it can be determined that there is a data anomaly.

[0115] It should be noted that, when the target image is a scanned image of a thread-bound book, the font, font size, relative position of illustrations and text, and / or the relative position of the title and text can also be identified to achieve the above-mentioned image content quality inspection.

[0116] Exemplarily, when there is a difference between the above-mentioned font and the preset font, and / or the font size and the preset font size, it is determined that there is a data anomaly.

[0117] Exemplarily, when the difference between the relative position of the illustration and the text and the preset relative position is greater than the preset difference, it is determined that there is a data anomaly.

[0118] Exemplarily, when the difference between the relative position of the title and the text and the preset relative position is greater than the preset difference, it is determined that there is a data anomaly.

[0119] Exemplarily, the above-mentioned image angle quality inspection can be implemented based on the use of image feature extraction and geometric transformation algorithms.

[0120] Exemplarily, key feature points of the target image can be extracted to determine geometric relationship information between the key feature points and / or angle information of a preset direction. Based on the geometric relationship information and / or angle information of a preset direction, the rotation angle of the target image can be determined to determine the tilt direction of the target image, wherein the key feature points can include: edge points and / or corner points.

[0121] Specifically, the rotation angle of the target image may be determined based on the straight line in the Hough transform quality inspection image and the slope or angle of the straight line.

[0122] Alternatively, the target image may be matched with a preset template rotated by a preset angle, and the rotation angle of the target image may be determined according to the rotation angle corresponding to the preset template that has the highest fit with the target image.

[0123] It should be noted that the above rotation angle is used to represent the rotation angle of the target image relative to a preset direction, wherein the above preset direction includes: a horizontal direction and / or a vertical direction.

[0124] Alternatively, the tilt category of the target image may also be identified based on a preset convolutional neural network classification model.

[0125] Exemplarily, a large number of images with rotation angles can be collected in advance as training samples, and the rotation angles corresponding to the training samples can be labeled to train and generate the preset convolutional neural network classification model to learn the rotation features of the training samples.

[0126] Exemplarily, the preset convolutional neural network classification model can determine the rotation angle of the target image based on the similarity between the rotation features of the training samples and the corresponding features of the target image, and determine the corresponding tilt category according to the rotation angle.

[0127] Specifically, the training sample rotation features may include: edge rotation features, texture rotation features, and / or grayscale or color distribution rotation features. The edge rotation features may be determined based on the changes in the edge of the target image at different scanning angles. The texture rotation features may be determined based on the changes in the texture direction of the target image at different scanning angles. The grayscale or color distribution rotation features may be determined based on the changes in the distribution of grayscale or color of the target image at different scanning angles.

[0128] Specifically, the similarity between the rotation feature of the training sample and the corresponding feature of the target image may be determined based on the Euclidean distance and / or the cosine similarity.

[0129] Specifically, the above tilt categories may include: horizontal tilt, vertical tilt, and / or compound tilt. Horizontal tilt corresponds to the target image tilting in the horizontal direction in a clockwise or counterclockwise direction, for example: 5 degrees clockwise tilt, 10 degrees counterclockwise tilt, etc. Vertical tilt corresponds to the image tilting in the vertical direction, for example: the target image tilts upward or downward at a certain angle as a whole. Compound tilt corresponds to the target image tilting in both horizontal and vertical directions, that is, oblique tilt.

[0130] It should be noted that if data anomalies are determined based on image angle quality inspection, a corresponding anomaly report can be generated to facilitate subsequent manual inspections, wherein the anomaly report may include: rotation angle, tilt category, and / or tilt degree, etc.

[0131] Exemplarily, the image defects of the original image and / or the target image can be determined based on image filtering, edge quality inspection and region growing, etc. to achieve the above-mentioned image defect quality inspection. The above-mentioned defects may include: ink spots, scratches, stains, noise points, bad points, blurred areas, cracks, depressions, protrusions, etc.

[0132] Specifically, the target image may be subjected to noise reduction processing based on median filtering, and the edge discontinuity or mutation region in the target image may be inspected based on a Sobel operator or a Canny edge inspection algorithm to determine that the region is an image defect.

[0133] It should be noted that the above defects may be identified and located based on a morphological image processing method, wherein the above morphological image processing method may include an erosion operation, a dilation operation, and the like.

[0134] In some feasible implementations, the abnormal area in the target image may be separated from the normal background based on a threshold segmentation technique to determine the location and size of the defect.

[0135] It should be noted that the type, location, and / or degree of the above-mentioned defects may be recorded. For example, a defect label may be added to indicate that the defect type is a scratch defect, the location is the upper left corner of the image, and the degree of the defect is medium.

[0136] It should be noted that corresponding repair operations may also be performed according to the type of defect.

[0137] Exemplarily, when the defect type is a noise defect, the image noise may be removed based on an image filtering technology, wherein the image filtering technology may include: median filtering, Gaussian filtering, and the like.

[0138] Exemplarily, when the defect type content is missing or blurred, repair can be achieved based on context information or reference images using an image repair model based on texture synthesis or deep learning.

[0139] Exemplarily, the above-mentioned image DPI quality inspection can obtain the corresponding DPI value by reading the metadata information of the image based on the preset model, and compare the actually read DPI value with the preset DPI value. When the actually read DPI value and the preset DPI value are inconsistent, it is determined that there is a data anomaly. It should be noted that the above-mentioned DPI value can be determined based on the header of the data or a specific metadata field. For example: in a TIFF image, the above-mentioned DPI value can be determined by parsing the specific tags of the file. Among them, the above-mentioned preset DPI value can be set according to actual needs, and can be specifically set to 300 dpi.

[0140] Exemplarily, the image format size quality inspection may include: obtaining the image format size, comparing the image format size actually obtained with the preset size, and determining that there is a data anomaly when the image format size actually obtained is inconsistent with the preset size. The size may include: width and height.

[0141] Specifically, the width of the image format actually obtained can be compared with the preset width, and when the width of the image format actually obtained is inconsistent with the preset width, it is determined that there is a data anomaly. And / or, the height of the image format actually obtained can be compared with the preset height, and when the height of the image format actually obtained is inconsistent with the preset height, it is determined that there is a data anomaly.

[0142] It should be noted that the image size can be obtained based on a preset function or attribute, wherein the preset function or attribute may include: getWidth() and getHeight() in the Java BufferedImage class.

[0143] Exemplarily, the format quality check may include: checking whether the image format is within a selected format range, wherein the image format may include: JPEG, PNG, TIFF, etc.

[0144] Specifically, for JPEG format images, the corresponding header information includes: "FFD8". The abnormal data can be located by checking whether the bytes at the beginning of the data contain the above "FFD8".

[0145] Exemplarily, the above-mentioned KB value quality inspection may include: checking whether the image size is within the preset maximum and minimum ranges. In the process of checking whether the KB value of the image is within the preset range, an efficient inspection mechanism based on cache and pre-reading is adopted. By building an image information cache library at the bottom of the system, when the image enters the quality inspection process, the system first quickly obtains the approximate KB value information of the image from the cache library for preliminary judgment. If there is no relevant information in the cache, the pre-reading program is started, and only the key part of the image file is read to estimate the file size, rather than fully loading the entire image, which greatly shortens the inspection time, improves quality inspection efficiency, and reduces system resource usage. It has significant advantages for mass archive image quality inspection scenarios.

[0146] Exemplarily, the above-mentioned metadata item missing quality inspection may include: checking whether the checked metadata item exists, and marking it as abnormal data if it does not exist. When it is found that the checked metadata item does not exist, an intelligent prompt and repair mechanism is used to prompt the user that the metadata item is missing, and based on its semantic understanding model and historical data, provide the user with possible correct metadata item suggestions. For example, if the "archive number" checked by the user does not exist, the system will prompt "ArchiveNumber" and other possible correct options based on semantic similarity and common data patterns. In addition, for some metadata items that can be supplemented by data mining and reasoning, the system will automatically use existing data for deduction and repair. This intelligent prompt and repair mechanism can quickly correct metadata item errors and further improve the integrity of metadata.

[0147] Exemplarily, the readability inspection may include: checking whether the image is readable. After completing the readability inspection of the archive image, the evaluation result of the image readability is fed back in real time, and a detailed analysis of the reasons for the unreadable parts is also given, such as "the text is fuzzy, which may be caused by the low scanning resolution" or "the image has serious noise interference, which affects the readability". This real-time feedback mechanism can quickly understand the readability of the archive image and improve the usability of the archive image.

[0148] Exemplarily, the image storage structure quality inspection may include: checking image serial number jumps, checking images mounted in the same directory, excluding specially named images, and checking whether the serial numbers of images named with serial numbers are continuous.

[0149] It should be noted that the target image can also be constructed based on the target image data based on the deep learning algorithm. According to the difference between the original image corresponding to the target image data and the above target image, the image storage structure quality inspection is realized. Among them, the above difference can be determined according to the pixel value difference, structure and texture difference, and / or spatial layout difference. Among them, the above pixel value difference is used to determine the difference in pixel value of each corresponding pixel point between the above target image and the above original image. Among them, the above structure and texture difference is used to determine the difference in edge, line or texture pattern between the above target image and the above original image. Among them, the above spatial layout difference: is used to determine the difference in spatial layout of objects or elements in the above target image and the above original image.

[0150] Specifically, when the difference between the original image and the target image is greater than the preset difference, it is determined that the image storage structure is abnormal and the target image data is partially missing. The preset difference can be determined based on the standard deviation of the difference between the original image and the target image. For example, the preset difference can be 3 times the standard deviation of the difference.

[0151] Specifically, the difference between the original image and the target image can be determined based on the difference between the features of the original image and the features of the corresponding target image. For example, the feature difference can be determined based on the difference between the features corresponding to the pixels of the original image and the features corresponding to the pixels of the target image. The feature difference can be determined based on the number of pixels whose matching degree between the pixels of the original image and the pixels of the target image is less than a preset matching degree, and / or the number of pixels whose distance between the pixels of the original image and the pixels of the target image is greater than a preset distance.

[0152] It should be noted that, when it is determined that the target image has noise and / or artifacts, it can also be determined that the image storage structure has an abnormality.

[0153] Therefore, the above method is conducive to realizing multi-dimensional quality inspection of image data anomalies by performing image color mode quality inspection, image content quality inspection, image angle quality inspection, image defect quality inspection, image DPI quality inspection, image size quality inspection, format quality inspection, KB value quality inspection, metadata item missing quality inspection, readability quality inspection, and / or image storage structure quality inspection on the target quality inspection object, thereby improving the quality inspection accuracy of image data.

[0154] In some feasible implementations, the image storage structure quality inspection includes: determining that data anomaly exists when the mean square error of the target image is greater than a preset mean square error threshold, and / or the mean absolute error is greater than a preset mean absolute error threshold; Where the mean square error is determined based on the following code: import numpy as np mse = np.mean((original_image - reconstructed_image) ** 2); The mean absolute error is determined based on the following code: mae = np.mean(np.abs(original_image - reconstructed_image)).

[0155] It should be noted that the above-mentioned preset mean square error threshold and / or the above-mentioned preset mean absolute error threshold can be determined according to the quality inspection accuracy of the abnormal data, wherein the above-mentioned preset mean square error threshold and / or the above-mentioned preset mean absolute error threshold are negatively correlated with the quality inspection accuracy of the above-mentioned abnormal data, that is, the higher the quality inspection accuracy of the abnormal data, the smaller the above-mentioned preset mean square error threshold and / or the above-mentioned preset mean absolute error threshold.

[0156] Therefore, the difference between the original image and the target image can be accurately determined based on the above code. When the difference between the target image and the original image is greater than the preset difference, the existence of data anomalies can be accurately determined, which is conducive to further improving the degree of automation of data quality inspection, reducing the degree of manual participation in the data quality inspection process, and improving the quality inspection accuracy and efficiency of abnormal data.

[0157] In some feasible implementations, the above-mentioned directory and image quality inspection includes: determining the recorded number of frames according to the target directory; determining the actual number of frames according to the target image; and determining that there is a data anomaly when the recorded number of frames and the actual number of frames are inconsistent.

[0158] Exemplarily, the number of recorded frames can be determined by parsing the metadata field corresponding to the catalog information.

[0159] Specifically, the key-value pair corresponding to the "number of frames" can be determined by parsing the XML information and / or JSON metadata in the directory information, and the number of recorded frames can be determined based on the key-value pair.

[0160] Exemplarily, the target image may be analyzed based on image recognition technology to determine the actual number of frames.

[0161] Specifically, the image of multiple pages of the document may be scanned, and corresponding boundary information may be determined to determine the actual number of frames.

[0162] Exemplarily, when the absolute value of the difference between the recorded frame number and the actual frame number is greater than 0, it is determined that there is a data anomaly.

[0163] It should be noted that when it is determined that there is a data anomaly, a corresponding anomaly report can be generated to facilitate subsequent manual inspections, wherein the above-mentioned anomaly report may include: image name, directory path, recorded number of frames, actual number of frames, and / or corresponding cause analysis, etc.

[0164] Therefore, the above method can accurately determine the number of recorded frames according to the target directory; accurately determine the actual number of frames according to the target image; and accurately determine the existence of data anomalies when the recorded number of frames and the actual number of frames are inconsistent, which is conducive to further improving the quality inspection efficiency of abnormal data.

[0165] It should be noted that you can set and select the corresponding target quality inspection operation according to actual needs. After completing the corresponding target quality inspection operation, you can also generate a corresponding target quality inspection report, where the above-mentioned target quality inspection report may include: the name of the target quality inspection operation, the quality inspection status of each operation, the total number of inspections required, the total number of inspections, the total number of errors, and / or the corresponding quality inspection details.

[0166] Among them, the corresponding quality inspection details may include: file level, total number of pages, title, case file number, file number, serial number, quality inspection status, error details, etc.

[0167] It should be noted that the above error details may include: abnormal descriptions based on the above target quality inspection operation content, such as "Archive level: the recorded content is not in the configuration dictionary", "Serial number: the content length in the field is not equal to the number of zero-padding digits", etc.

[0168] It should be noted that after the quality inspection is completed, the item details of the quality inspection task can be viewed on the progress page, the "All" tab displays all quality inspection items, the "Abnormal" tab displays abnormal items, and the "Classification Display" tab displays all abnormal quality inspection items by quality inspection item dimension. Furthermore, it is divided into three quality inspection classification tabs: "Directory Inspection", "Directory and Image Inspection", and "Image Inspection", which respectively display the items participating in the corresponding quality inspection classification under the "All", "Abnormal", and "Classification Display" tab conditions, and display them by "File" and "Document" classification, showing the specific archival information details of the directory and image.

[0169] Based on this, the data quality inspection method provided by the embodiment of the present application includes: obtaining the data boundary and basic information corresponding to the target quality inspection object; performing a target cutting operation on the target quality inspection object to generate a target data element according to the data boundary and basic information; constructing a target quality inspection data group according to the target data element, wherein the granularity of the target data element is greater than or equal to the preset granularity; performing a target quality inspection operation on the target quality inspection data group to determine the target abnormal data. The present application can reduce the degree of manual participation in the data quality inspection process and improve the quality inspection efficiency and quality inspection accuracy of abnormal data.

[0170] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0171] The above is an introduction to the method embodiment. The following is a further explanation of the scheme described in this application through an apparatus embodiment.

[0172] According to a second aspect of the embodiments of the present application, a data quality inspection system is provided. Figure 3 Schematic diagram of a data quality inspection system 200 provided in an embodiment of the present application. Figure 3 The system 200 shown includes: an acquisition unit 210 , a first execution unit 220 , a construction unit 230 , and a second execution unit 240 .

[0173] An acquisition unit 210 is used to acquire data boundaries and basic information corresponding to a target quality inspection object; A first execution unit 220, configured to perform a target cutting operation on a target quality inspection object according to data boundaries and basic information to generate target data elements; A construction unit 230, configured to construct a target quality inspection data group according to the target data element, wherein the granularity of the target data element is greater than or equal to a preset granularity; The second execution unit 240 is used to perform a target quality inspection operation on the target quality inspection data group to determine target abnormal data.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described method can refer to the corresponding process in the aforementioned system embodiment, and will not be repeated here.

[0175] Figure 4 Schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present application. Figure 4 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 into the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the terminal device or the server are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0176] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0177] In particular, according to an embodiment of the present application, the above method flow steps can be implemented as a computer software program. For example: an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the system of the present application are executed.

[0178] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0179] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example: two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0181] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A data quality inspection method, characterized in that: Applicable to archival data, the method comprises: Obtain the data boundaries and basic information corresponding to the target quality inspection object; According to the data boundary and the basic information, performing a target cutting operation on the target quality inspection object to generate a target data element; According to the target data element, construct the target quality inspection data group, wherein the granularity of the target data element is greater than or equal to a preset granularity; For the target quality inspection data group, a target quality inspection operation is performed to determine target abnormal data.

2. The data quality inspection method according to claim 1, characterized in that: The basic information includes: data scale information, batch information, and / or time information.

3. The data quality inspection method according to claim 2, characterized in that: The target quality inspection operation includes: a directory quality inspection operation, an image quality inspection operation, and / or a directory and image quality inspection operation.

4. The data quality inspection method according to claim 3, characterized in that: The directory quality inspection operations include: required item quality inspection, length quality inspection, multi-level same field consistency quality inspection, multi-level entry matching quality inspection, multi-level start and end page number quality inspection, typo quality inspection, date quality inspection, digit quality inspection, recorded content quality inspection, serial number quality inspection, recorded dictionary item quality inspection, and / or repeated word quality inspection.

5. The data quality inspection method according to claim 3, characterized in that: The image quality inspection operation includes: image color mode quality inspection, image content quality inspection, image angle quality inspection, image defect quality inspection, image DPI quality inspection, image size quality inspection, format quality inspection, KB value quality inspection, metadata item missing quality inspection, readability quality inspection, and / or image storage structure quality inspection.

6. The data quality inspection method according to claim 3, characterized in that: The directory and image quality inspection operations include: Determine the number of pictures to be recorded based on the target catalog; Determine the actual number of frames according to the target image; When the recorded number of frames and the actual number of frames are inconsistent, it is determined that there is a data anomaly.

7. The data quality inspection method according to claim 5, characterized in that: The image storage structure quality inspection includes: When the mean square error of the target image is greater than a preset mean square error threshold, and / or the mean absolute error is greater than a preset mean absolute error threshold, determining that data anomaly exists; Wherein, the mean square error is determined based on the following code: import numpy as np mse = np.mean((original_image - reconstructed_image) ** 2); The mean absolute error is determined based on the following code: mae = np.mean(np.abs(original_image - reconstructed_image)).

8. A data quality inspection system, characterized in that: include: An acquisition unit, used to acquire data boundaries and basic information corresponding to the target quality inspection object; A first execution unit, configured to perform a target cutting operation on the target quality inspection object according to the data boundary and the basic information to generate a target data element; A construction unit, configured to construct the target quality inspection data group according to the target data element, wherein the granularity of the target data element is greater than or equal to a preset granularity; The second execution unit is used to perform a target quality inspection operation on the target quality inspection data group to determine target abnormal data.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the data quality inspection method according to any one of claims 1 to 7 when the processor is running.

10. A storage medium having program instructions stored thereon, wherein the program instructions are used to execute the data quality inspection method according to any one of claims 1 to 7 when running.

Images