Material data cleaning method and device, computer equipment and storage medium

Abstract

The invention relates to a material data cleaning method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed material data set, and carrying out the data standardization processing of the to-be-processed material data set, and obtaining a standardized material data set; converting data in the standardized material data set into a numerical value feature vector, and performing similarity judgment based on the numerical value feature vector so as to screen initial similar material data from the standardized material data set; acquiring an image corresponding to the initial similar material data to obtain a target image, and performing similarity judgment based on the target image to divide the initial similar material data into target similar material data and non-similar material data; and performing data merging on the target similar material data, and performing data completion on the non-similar material data to obtain cleaned material data. According to the method and the device, similar data can be accurately identified from the material data, and the material cleaning accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a material data cleaning method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of enterprises, warehouse material control has become a key factor affecting the production efficiency and cost control of enterprises. Warehouse managers must avoid inventory backlogs while ensuring that production needs are met. Under traditional management methods, the storage location of materials depends on the manual memory of warehouse managers, which makes it difficult and time-consuming to find materials. For materials that have been stored for a longer time, it is easy to fail to find them in time. Duplication of materials in the material management system and errors in material coding may lead to the generation of obsolete materials and waste materials, resulting in unnecessary waste.

[0003] The accuracy of material data is crucial for material management. However, due to data input errors, transmission problems or human errors, material data often contains errors and inconsistencies, and material data needs to be cleaned. The existing material cleaning method is to clean by manual verification. However, during the material cleaning process, there are a large number of material specification codes that are highly similar. The cleaning personnel alone cannot accurately check whether the similar materials are duplicate materials, resulting in low accuracy of material cleaning. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a material data cleaning method, device, computer equipment and storage medium to improve the accuracy of material data cleaning.

[0005] In order to solve the above technical problems, the present application embodiment provides a material data cleaning method, including:

[0006] Acquire a data set of materials to be processed, and perform data standardization processing on the data set of materials to be processed to obtain a standardized material data set;

[0007] Converting the data in the standardized material data set into a numerical feature vector, and performing similarity determination based on the numerical feature vector to filter out initial similar material data from the standardized material data set;

[0008] Acquire an image corresponding to the initial similar material data to obtain a target image, and perform similarity determination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data;

[0009] The target similar material data are merged, and the non-similar material data are completed to obtain cleaned material data.

[0010] In order to solve the above technical problems, the present application provides a material data cleaning device, including:

[0011] A material data acquisition unit, used to acquire a material data set to be processed, and perform data standardization processing on the material data set to be processed to obtain a standardized material data set;

[0012] a first similar data identification unit, configured to convert the data in the standardized material data set into a numerical feature vector, and perform similarity determination based on the numerical feature vector, so as to screen out initial similar material data from the standardized material data set;

[0013] a second similar data identification unit, configured to obtain an image corresponding to the initial similar material data, obtain a target image, and perform similarity determination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data;

[0014] The material data cleaning unit is used to merge the target similar material data and complete the dissimilar material data to obtain cleaned material data.

[0015] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that one or more processors can implement any one of the material data cleaning methods described above.

[0016] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the material data cleaning methods described above is implemented.

[0017] The embodiment of the present invention provides a material data cleaning method, device, computer equipment and storage medium. The method includes: obtaining a data set of materials to be processed, and performing data standardization on the data set of materials to be processed to obtain a standardized material data set; converting the data in the standardized material data set into a numerical feature vector, and performing similarity discrimination based on the numerical feature vector to filter out initial similar material data from the standardized material data set; obtaining an image corresponding to the initial similar material data to obtain a target image, and performing similarity discrimination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data; merging the target similar material data, and completing the unsimilar material data to obtain cleaned material data. The embodiment of the present invention facilitates cleaning of material data by standardizing the material data, and then performs similarity discrimination on the standardized material data to preliminarily filter out similar material data, and then determines the target similar material data according to the image recognition method for the similar material data, thereby realizing cleaning of the material data, which is conducive to accurately identifying similar data from the material data and improving the accuracy of material cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart for implementing the material data cleaning method provided in the embodiment of the present application;

[0020] Figure 2 It is a flowchart for implementing the first sub-process in the material data cleaning method provided in the embodiment of the present application;

[0021] Figure 3 It is a flowchart for implementing the second sub-process in the material data cleaning method provided in the embodiment of the present application;

[0022] Figure 4 It is a flowchart for implementing the third sub-process in the material data cleaning method provided in the embodiment of the present application;

[0023] Figure 5 It is a flowchart for implementing the fourth sub-process in the material data cleaning method provided in the embodiment of the present application;

[0024] Figure 6 It is a flowchart for implementing the fifth sub-process in the material data cleaning method provided in the embodiment of the present application;

[0025] Figure 7 It is a flowchart for implementing the sixth sub-process in the material data cleaning method provided in the embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of a material data cleaning device provided in an embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0031] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the material data cleaning method provided in the embodiment of the present application is generally executed by a server, and accordingly, the material data cleaning device is generally configured in the server.

[0033] See also Figure 1 , Figure 1 A specific implementation of the material data cleaning method is shown.

[0034] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0035] S1: Obtain a data set of materials to be processed, and perform data standardization processing on the data set of materials to be processed to obtain a standardized material data set.

[0036] Specifically, the material data set that needs data cleaning is obtained to obtain the material data set to be processed, wherein the material data set to be processed includes various types of material information, aiming to support master data cleaning, processing and analysis. Each data in the data set to be processed includes material name, specification, model and grade, etc. In data cleaning processing, standardization processing is crucial. Since the material information formats vary, it is necessary to first formulate unified specifications to ensure the quality and consistency of the data throughout the organization. Therefore, the material data set to be processed is subjected to data standardization processing to obtain a standardized material data set.

[0037] In a specific embodiment, the data composition of the to-be-processed material data set is shown in Table 1:

[0038] Table 1

[0039] Key Tags meaning Material Description Reflects material information. Material Name Unambiguously identify the common name of the material. Specifications The material's specification model or model code provided by the manufacturer. Drawing code Drawing or design code of the material. Material The main material composition of the material. brand The brand that produces or supplies the material.

[0040] See also Figure 2 , Figure 2 A specific implementation of step S1 is shown, which is described in detail as follows:

[0041] S11: Acquire the material data set to be processed, and convert the data in the material data set to be processed into a preset format to obtain a converted material data set.

[0042] S12: Standardizing the data in the converted material data set according to preset naming rules and preset data rules, and encoding each data in the converted material data set to obtain the standardized material data set.

[0043] Specifically, the data in the material data set to be processed is converted into a unified preset format. The preset format can be: material name\specification model\drawing number code\material\brand\implementation standard; the date should be in YYYY-MM-DD format, the currency should use a unified unit, the address format and text case should also be consistent, and the full-width and half-width characters should be unified. After the data format is unified, it is necessary to unify the expression in naming standardization. For example, the place name "New York" is standardized as "NewYork, NY". Then you can set data rules, which define the quality standards that the data must meet, such as completeness, validity, and consistency. Then, by encoding each data in the material data set, data duplication and confusion can be avoided, and the manageability and retrieval efficiency of the data can be improved.

[0044] As shown in Table 2 and Table 3, they are the comparison results of the material data to be processed and the material data after standardization:

[0045] Table 2

[0046]

[0047] Table 3

[0048]

[0049]

[0050] S2: Convert the data in the standardized material data set into a numerical feature vector, and perform similarity determination based on the numerical feature vector to filter out initial similar material data from the standardized material data set.

[0051] Specifically, after the material data is standardized, it is necessary to identify similar data from the standardized material data, so as to facilitate merging the similar data and complete the cleaning of the material data. In an embodiment of the present application, it is necessary to convert the data in the standardized material data set into a numerical feature vector, and perform similarity discrimination based on the numerical feature vector to filter out initial similar material data with high similarity from the standardized material data set.

[0052] See also Figure 3 , Figure 3 A specific implementation of step S2 is shown, which is described in detail as follows:

[0053] S21: classifying the standardized material data set to obtain material data of different categories.

[0054] Specifically, the data in the standardized material data set is classified to obtain material data of different categories. Among them, the material data can be classified according to the code type, specifically according to a type code among the sub-category code, the middle-category code and the large-category code. In a specific embodiment, according to the material coding, the material data can be classified according to the sub-category code, and the data classification is conducive to the acceleration effect of the subsequent similarity comparison operation. In a specific embodiment, the standardized material data set is classified according to the sub-category code to obtain material data of different categories. The material data set is classified as shown in Table 4:

[0055] Table 4

[0056]

[0057]

[0058] S22: performing word segmentation on the material data of different categories respectively, so as to convert the material data of the same category into a vocabulary sequence.

[0059] Specifically, the IK tokenizer is used to tokenize different categories of material data, and all material descriptions under the same category are converted into vocabulary sequences. For example, "high-strength bolts\m30×41010.9 grade" is tokenized into: ["high strength", "bolts", "m30", "×", "410", "10.9 grade"].

[0060] S23: Convert the vocabulary sequence into the numerical feature vector, and calculate the similarity between every two numerical feature vectors to obtain an initial similarity value.

[0061] See also Figure 4 , Figure 4 A specific implementation of step S23 is shown, which is described in detail as follows:

[0062] S231: extracting vocabulary information from each of the vocabulary sequences respectively, and constructing a vocabulary table according to the vocabulary information, wherein each piece of vocabulary information in the vocabulary table is mapped to a unique index.

[0063] Specifically, unique vocabulary information is extracted from each of the vocabulary sequences to construct a vocabulary table V = {v1, v2, ..., v n}, (n is the number of vocabulary information), each vocabulary information will be mapped to a unique index for subsequent vectorization (if there are 100 categories, 100 vocabularies will be generated).

[0064] S232: Using the TF-IDF algorithm to convert the vocabulary information into the numerical feature vector based on the vocabulary table.

[0065] Specifically, after completing the word segmentation, these word segmentations need to be converted into numerical feature vectors. The TF-IDF algorithm is used to convert the vocabulary information into the numerical feature vector. Among them, the core idea of ​​the TF-IDF algorithm is to measure the importance of a word in a document while considering the rarity of the word in the entire corpus.

[0066] See also Figure 5 , Figure 5 A specific implementation of step S232 is shown, which is described in detail as follows:

[0067] S2321: Obtain document information of each word information in the word table.

[0068] S2322: Calculate the word frequency and inverse document frequency of each word information according to the document information.

[0069] S2323: Convert the vocabulary information into the numerical feature vector according to the word frequency and the inverse document frequency.

[0070] Specifically, by calculating the TF-IDF value of each word, a numerical feature vector can be generated for each material description, and each dimension of the numerical feature vector corresponds to a word, and its value is the TF-IDF value of the word. Each material data description is converted into a vector in a high-dimensional space. In an embodiment of the present application, the document information of each of the vocabulary information in the vocabulary table is obtained, and the document information includes the number of documents, the number of times the word appears in the document, the number of documents including the word, and the total number of all words in the document. Then, the word frequency and inverse document frequency of each of the vocabulary information are calculated based on the document information. Then, the word frequency and inverse document frequency are multiplied to obtain a numerical feature vector for each vocabulary information.

[0071] In a specific embodiment, there are currently two material description data, namely "high-strength bolts\m30×41010.9 grade" and "high-strength bolts\m20×3008.8 grade". After TF-IDF calculation, the corresponding space vectors are obtained:

[0072] (Corresponding to: high strength, bolts, M30, 410, 10.9 grade);

[0073] (Corresponding to: high strength, bolts, M20, 300, grade 8.8).

[0074] S233: Calculate the similarity between every two of the numerical feature vectors to obtain the initial similarity value.

[0075] Specifically, after obtaining the numerical feature vector, the similarity between different material descriptions is measured by methods such as the cosine theorem and the Euclidean distance. In a specific embodiment, the cosine theorem is used as an example to describe the processing process in detail. The cosine similarity measures the similarity by calculating the angle between two vectors, and its formula is:

[0076]

[0077] Among them, A and B are numerical feature vectors describing two material data, A×B represents the dot product of the vectors, and |A| and |B| represent the modulus of the vectors.

[0078] S24: If the initial similarity value exceeds a preset threshold, the material data corresponding to the initial similarity value is determined to be similar material data. When all the initial similarity values ​​are determined, the initial similar material data is generated.

[0079] Specifically, if the initial similarity value exceeds the preset threshold, the material data corresponding to the initial similarity value is determined to be similar material data; if the initial similarity value does not exceed the preset threshold, the material data corresponding to the initial similarity value is determined to be dissimilar material data; when all initial similarity values ​​are judged, the initial similar material data is generated.

[0080] It should be noted that the preset threshold is set according to actual conditions and is not limited here.

[0081] S3: Acquire an image corresponding to the initial similar material data to obtain a target image, and perform similarity determination based on the target image to divide the initial similar material data into target similar material data and non-similar material data.

[0082] Specifically, after the initial similar material data is identified through the similarity described above, depending on the actual business scenario, there may be a situation where the models are similar but the actual objects are quite different. Therefore, in order to further improve the judgment of similar data, the embodiment of the present application obtains the image corresponding to the initial similar material data, and judges the similar material data based on the similarity of the image, thereby dividing the initial similar material data into target similar material data and non-similar material data.

[0083] See also Figure 6 , Figure 6 A specific implementation of step S3 is shown, which is described in detail as follows:

[0084] S31: Acquire the image of the initial similar material data when the material is put into storage from the database to obtain the target image.

[0085] S32: Perform feature recognition based on the target image through a deep residual network to obtain a target feature vector.

[0086] Specifically, the image of the initial similar material data when the material is put into storage is obtained from the database to obtain the image, and then the deep residual network is used to perform feature recognition based on the target image to obtain the target feature vector. The target feature vector is a probability distribution, which indicates the possibility that the input image belongs to each category.

[0087] See also Figure 7 , Figure 7 A specific implementation of step S32 is shown, which is described in detail as follows:

[0088] S321: performing image preprocessing on the target image to obtain a preprocessed image.

[0089] S322: Input the preprocessed image into the deep residual network, and perform feature extraction on the preprocessed image through a convolution kernel to obtain image features.

[0090] S323: Perform down-sampling processing on the image features to obtain down-sampled features.

[0091] S324: Perform feature processing on the downsampled features through the residual block of the residual connection to obtain superimposed features.

[0092] S325: Perform global average pooling processing on the superimposed features to obtain the target feature vector.

[0093] Specifically, the target image is subjected to image preprocessing to obtain a preprocessed image. Image preprocessing includes image pixel unification, image normalization and data enhancement. In a specific embodiment, the pixels of the target image are unified to 224×224, and the pixel values ​​of the target image are scaled to between 0 and 1; data enhancement includes image flipping, cropping, and rotation, thereby increasing the robustness of the image. The preprocessed image is input into a deep residual network, and feature extraction is performed on the preprocessed image through a convolution kernel to obtain image features, and then the image features are downsampled to obtain downsampled features. Feature processing is performed on the downsampled features through a residual block of a residual connection to obtain a superimposed feature. Similarity calculation is performed based on the target feature vector to obtain a target similarity value.

[0094] Among them, the target image first passes through a series of convolutional layers of the deep residual network, which can extract the features of the image. Each convolutional layer is usually followed by a batch normalization layer and a ReLU activation function to reduce internal covariate shift and increase nonlinearity. The deep residual network builds the network through residual blocks. Each residual block contains several convolutional layers and has a skip connection that adds the input directly to the output of the block. The purpose of this is to allow the network to learn the residual (i.e., the difference) between the input and output instead of directly learning the output. If there is no difference between the input and output, then the residual block is actually an identity mapping, which helps the propagation of the gradient. The deep residual network makes the network very deep by stacking multiple residual blocks, avoiding the problem of gradient vanishing or exploding. At the end of the deep residual network, there is a global average pooling layer that compresses the spatial dimensions (height and width) of each feature map to 1, which can reduce the number of parameters of the fully connected layer. After global average pooling, the data is fed into one or more fully connected layers, which are responsible for mapping the features to the final category probabilities, and finally the deep residual network outputs the target feature vector.

[0095] S34: If the target similarity value exceeds a preset threshold, determining that the initial similar material data corresponding to the target similarity value is the target similar material data.

[0096] S35: If the target similarity value does not exceed the preset threshold, determining that the initial similar material data corresponding to the target similarity value is the non-similar material data.

[0097] Specifically, if the target similarity value exceeds the preset threshold, the initial similar material data corresponding to the target similarity value is determined to be the target similar material data; if the target similarity value does not exceed the preset threshold, the initial similar material data corresponding to the target similarity value is determined to be non-similar material data.

[0098] S4: merging the target similar material data and completing the non-similar material data to obtain cleaned material data.

[0099] Specifically, the target similar material data are merged, the dissimilar material data are completed, and marked as different items, so as to obtain the cleaned material data.

[0100] In an embodiment of the present application, a data set of materials to be processed is obtained, and the data set of materials to be processed is subjected to data standardization processing to obtain a standardized material data set; the data in the standardized material data set is converted into a numerical feature vector, and similarity judgment is performed based on the numerical feature vector to filter out initial similar material data from the standardized material data set; an image corresponding to the initial similar material data is obtained to obtain a target image, and similarity judgment is performed based on the target image to divide the initial similar material data into target similar material data and unsimilar material data; the target similar material data is subjected to data merging, and the unsimilar material data is subjected to data completion to obtain cleaned material data. The embodiment of the present invention facilitates the cleaning of material data by performing standardization processing on material data, and then performs similarity judgment on the standardized material data to preliminarily filter out similar material data, and then performs image recognition on the similar material data to determine the target similar material data, thereby achieving cleaning of material data, which is conducive to accurately identifying similar data from material data and improving the accuracy of material cleaning.

[0101] Please refer to Figure 8 , as a response to the above Figure 1 In order to realize the method shown in the figure, the present application provides an embodiment of a material data cleaning device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0102] like Figure 8 As shown, the material data cleaning device of this embodiment includes: a material data acquisition unit 51, a first similar data identification unit 52, a second similar data identification unit 53, and a material data cleaning unit 54, wherein:

[0103] The material data acquisition unit 51 is used to acquire a material data set to be processed, and perform data standardization processing on the material data set to be processed to obtain a standardized material data set;

[0104] A first similar data identification unit 52, configured to convert the data in the standardized material data set into a numerical feature vector, and perform similarity determination based on the numerical feature vector to filter out initial similar material data from the standardized material data set;

[0105] The second similar data identification unit 53 is used to obtain an image corresponding to the initial similar material data, obtain a target image, and perform similarity determination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data;

[0106] The material data cleaning unit 54 is used to merge the target similar material data and complete the non-similar material data to obtain cleaned material data.

[0107] Furthermore, the first similar data identification unit 52 includes:

[0108] A data classification unit, used for classifying the standardized material data set to obtain material data of different categories;

[0109] A data segmentation unit, used for segmenting the material data of different categories respectively, so as to convert the material data of the same category into a vocabulary sequence;

[0110] An initial similarity value calculation unit, used for converting the vocabulary sequence into the numerical feature vector, and calculating the similarity between every two numerical feature vectors to obtain an initial similarity value;

[0111] The first similarity determination unit is used to determine that the material data corresponding to the initial similarity value is similar material data if the initial similarity value exceeds a preset threshold, and to generate the initial similar material data when all the initial similarity values ​​are determined.

[0112] Furthermore, the initial similarity value calculation unit includes:

[0113] A vocabulary information extraction unit, used to extract vocabulary information from each of the vocabulary sequences, and construct a vocabulary table according to the vocabulary information, wherein each of the vocabulary information in the vocabulary table is mapped to a unique index;

[0114] A numerical feature vector conversion unit, configured to convert the vocabulary information into the numerical feature vector based on the vocabulary table by using a TF-IDF algorithm;

[0115] The similarity calculation unit is used to calculate the similarity between every two of the numerical feature vectors to obtain the initial similarity value.

[0116] Furthermore, the numerical feature vector conversion unit includes:

[0117] A document information acquisition unit, used to acquire document information of each word information in the word table;

[0118] A frequency calculation unit, used for calculating the word frequency and inverse document frequency of each of the vocabulary information according to the document information;

[0119] A vector conversion unit is used to convert the vocabulary information into the numerical feature vector according to the word frequency and the inverse document frequency.

[0120] Furthermore, the second similar data identification unit 53 includes:

[0121] A target image generating unit, used for acquiring an image of the initial similar material data when the material is put into storage from a database to obtain the target image;

[0122] A feature recognition unit, used to perform feature recognition based on the target image through a deep residual network to obtain a target feature vector;

[0123] A target similarity value generating unit, used for performing similarity calculation based on the target feature vector to obtain a target similarity value;

[0124] A first threshold processing unit, configured to determine that the initial similar material data corresponding to the target similarity value is the target similar material data if the target similarity value exceeds a preset threshold;

[0125] The second threshold processing unit is used to determine that the initial similar material data corresponding to the target similarity value is the non-similar material data if the target similarity value does not exceed a preset threshold.

[0126] Furthermore, the feature recognition unit includes:

[0127] An image preprocessing unit, used for performing image preprocessing on the target image to obtain a preprocessed image;

[0128] A feature extraction unit, used for inputting the preprocessed image into the deep residual network, performing feature extraction on the preprocessed image through a convolution kernel, and obtaining image features;

[0129] A downsampling unit, used for performing downsampling processing on the image features to obtain downsampling features;

[0130] A superposition feature processing unit, used for performing feature processing on the down-sampled features through a residual block of a residual connection to obtain a superposition feature;

[0131] A pooling processing unit is used to perform global average pooling processing on the superimposed features to obtain the target feature vector.

[0132] Furthermore, the material data acquisition unit 51 includes:

[0133] a format conversion unit, used for acquiring the material data set to be processed, and converting the data in the material data set to be processed into a preset format to obtain a converted material data set;

[0134] The standardization processing unit is used to standardize the data in the converted material data set according to preset naming rules and preset data rules, and to encode each data in the converted material data set to obtain the standardized material data set.

[0135] In an embodiment of the present application, a data set of materials to be processed is obtained, and the data set of materials to be processed is subjected to data standardization processing to obtain a standardized material data set; the data in the standardized material data set is converted into a numerical feature vector, and similarity judgment is performed based on the numerical feature vector to filter out initial similar material data from the standardized material data set; an image corresponding to the initial similar material data is obtained to obtain a target image, and similarity judgment is performed based on the target image to divide the initial similar material data into target similar material data and unsimilar material data; the target similar material data is subjected to data merging, and the unsimilar material data is subjected to data completion to obtain cleaned material data. The embodiment of the present invention facilitates the cleaning of material data by performing standardization processing on material data, and then performs similarity judgment on the standardized material data to preliminarily filter out similar material data, and then performs image recognition on the similar material data to determine the target similar material data, thereby achieving cleaning of material data, which is conducive to accurately identifying similar data from material data and improving the accuracy of material cleaning.

[0136] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0137] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 which are interconnected through a system bus. It should be noted that: Fig. 9Only a computer device 6 having three components, memory 61, processor 62, and network interface 63, is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0138] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0139] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the material data cleaning method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0140] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code of the material data cleaning method described above, to implement various embodiments of the material data cleaning method.

[0141] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0142] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a computer program. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of a material data cleaning method as described above.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0144] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is similarly within the scope of protection of the present application.

Claims

1. A material data cleaning method, characterized in that: include: Acquire a data set of materials to be processed, and perform data standardization processing on the data set of materials to be processed to obtain a standardized material data set; Converting the data in the standardized material data set into a numerical feature vector, and performing similarity determination based on the numerical feature vector to filter out initial similar material data from the standardized material data set; Acquire an image corresponding to the initial similar material data to obtain a target image, and perform similarity determination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data; The target similar material data are merged, and the non-similar material data are completed to obtain cleaned material data.

2. The material data cleaning method according to claim 1, characterized in that: The step of converting the data in the standardized material data set into a numerical feature vector and performing similarity determination based on the numerical feature vector to filter out initial similar material data from the standardized material data set includes: Classifying the standardized material data set to obtain material data of different categories; Performing word segmentation on the material data of different categories respectively, so as to convert the material data of the same category into a vocabulary sequence; Converting the vocabulary sequence into the numerical feature vector, and calculating the similarity between every two numerical feature vectors to obtain an initial similarity value; If the initial similarity value exceeds a preset threshold, the material data corresponding to the initial similarity value is determined to be similar material data. When all the initial similarity values ​​are determined, the initial similar material data is generated.

3. The material data cleaning method according to claim 2, characterized in that: The converting the vocabulary sequence into the numerical feature vector and calculating the similarity between every two numerical feature vectors to obtain an initial similarity value includes: Extracting vocabulary information from each of the vocabulary sequences respectively, and constructing a vocabulary table according to the vocabulary information, wherein each of the vocabulary information in the vocabulary table is mapped to a unique index; Using the TF-IDF algorithm to convert the vocabulary information into the numerical feature vector based on the vocabulary table; The similarity between every two of the numerical feature vectors is calculated to obtain the initial similarity value.

4. The material data cleaning method according to claim 3, characterized in that: The adopting the TF-IDF algorithm to convert the vocabulary information into the numerical feature vector based on the vocabulary table includes: Obtain document information of each word information in the word table; Calculate the word frequency and inverse document frequency of each of the vocabulary information according to the document information; The vocabulary information is converted into the numerical feature vector according to the word frequency and the inverse document frequency.

5. The material data cleaning method according to claim 1, characterized in that: The acquiring of the image corresponding to the initial similar material data to obtain the target image, and performing similarity determination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data, includes: Acquire the image of the initial similar material data when the material is put into storage from the database to obtain the target image; Performing feature recognition based on the target image through a deep residual network to obtain a target feature vector; Perform similarity calculation based on the target feature vector to obtain a target similarity value; If the target similarity value exceeds a preset threshold, determining that the initial similar material data corresponding to the target similarity value is the target similar material data; If the target similarity value does not exceed the preset threshold, the initial similar material data corresponding to the target similarity value is determined to be the non-similar material data.

6. The material data cleaning method according to claim 5, characterized in that: The step of performing feature recognition based on the target image through a deep residual network to obtain a target feature vector includes: Performing image preprocessing on the target image to obtain a preprocessed image; Inputting the preprocessed image into the deep residual network, performing feature extraction on the preprocessed image through a convolution kernel to obtain image features; Performing downsampling processing on the image features to obtain downsampled features; Performing feature processing on the downsampled features through a residual block of a residual connection to obtain a superimposed feature; Perform global average pooling processing on the superimposed features to obtain the target feature vector.

7. The material data cleaning method according to any one of claims 1 to 6, characterized in that: The step of obtaining a data set of materials to be processed and performing data standardization on the data set of materials to be processed to obtain a standardized data set of materials includes: Acquire the material data set to be processed, and convert the data in the material data set to be processed into a preset format to obtain a converted material data set; The data in the converted material data set is standardized according to a preset naming rule and a preset data rule, and each data in the converted material data set is data encoded to obtain the standardized material data set.

8. A material data cleaning device, characterized in that: include: A material data acquisition unit, used to acquire a material data set to be processed, and perform data standardization processing on the material data set to be processed to obtain a standardized material data set; a first similar data identification unit, configured to convert the data in the standardized material data set into a numerical feature vector, and perform similarity determination based on the numerical feature vector, so as to screen out initial similar material data from the standardized material data set; a second similar data identification unit, configured to obtain an image corresponding to the initial similar material data, obtain a target image, and perform similarity determination based on the target image to divide the initial similar material data into target similar material data and unsimilar material data; The material data cleaning unit is used to merge the target similar material data and complete the dissimilar material data to obtain cleaned material data.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the material data cleaning method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the material data cleaning method according to any one of claims 1 to 7 is implemented.

Images