Data processing method and device, equipment and storage medium
By using an object detection network model to automatically identify and classify defective images generated during the screen manufacturing process, the problem of low screening efficiency and easy error in existing technologies is solved, and efficient and accurate image data processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BOE TECHNOLOGY GROUP CO LTD
- Filing Date
- 2020-10-27
- Publication Date
- 2026-07-10
AI Technical Summary
In existing technologies, the screening of defective image data generated during screen manufacturing is inefficient and prone to errors, making it difficult to automate.
The system uses a target detection network model to identify and classify defects in product images. It automatically downloads and classifies product images by combining data query commands, and uses feature extractors, region candidate networks, and target detectors to achieve automated defect identification and classification.
It improved data query efficiency, enabled automated defect identification and classification, enhanced the efficiency and accuracy of screening and classification, and reduced human error.
Smart Images
Figure CN114511726B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method, apparatus, device, and storage medium. Background Technology
[0002] The screen manufacturing process is complex, and due to equipment adjustments, personnel operations, and environmental interference, various defects can arise at each stage. These defects are categorized by AOI (Automated Optical Inspection) equipment and stored in a database. The sheer volume of these images, categorized by process segment, site, and product, results in a complex and hierarchical directory structure, making it extremely difficult to select images that meet specific criteria. Currently, factory engineers manually filter and collect the data, which is inefficient and prone to errors, such as missing data or inaccurate data categorization. Summary of the Invention
[0003] This application addresses the shortcomings of existing methods by proposing a data processing method, apparatus, device, and storage medium to solve the technical problems of low efficiency and error-proneness in manual data filtering in the prior art.
[0004] In a first aspect, embodiments of this application provide a data processing method, including:
[0005] In response to a received data query command, the corresponding product image is downloaded from the stored product image data;
[0006] The target detection network model is used to identify and classify product defects in each product image, and the defect category and defect feature vector of the product image are obtained.
[0007] Based on the defect feature vectors of each product image, determine the defect sub-category of each product image within each defect category.
[0008] Secondly, embodiments of this application provide a data processing apparatus, including:
[0009] The image download module is used to download the corresponding product image from the stored product image data in response to the received data query command;
[0010] The first defect classification module is used to identify and classify product defects in each product image through a target detection network model, and obtain the defect category and defect feature vector of the product image.
[0011] The second defect classification module is used to determine the defect sub-category of each product image in each defect category based on the defect feature vector of each product image.
[0012] Thirdly, embodiments of this application provide a data processing apparatus, including:
[0013] Memory;
[0014] The processor is electrically connected to the memory;
[0015] The memory stores a computer program, which is executed by the processor to implement the data processing method provided in the first aspect of the embodiments of this application.
[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data processing method provided in the first aspect of embodiments of this application.
[0017] The technical solution provided in this application has at least the following beneficial effects:
[0018] The technical solution provided in this application can automatically download the corresponding product images according to the data query command, without the need for manual downloading one by one, thus improving the efficiency of data query and making data query more convenient and faster; it can automatically identify product defects in the product images and automatically classify the downloaded product images according to the identified product defects, without the need for manual screening and classification, thus improving the efficiency and accuracy of image screening and classification.
[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description
[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0021] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;
[0022] Figure 2 This is a schematic diagram of the image download interface in an embodiment of this application;
[0023] Figure 3 This is a schematic diagram of the login interface in an embodiment of this application;
[0024] Figure 4 This is a schematic diagram of the user management interface in an embodiment of this application;
[0025] Figure 5 This is a schematic diagram of the structural framework of an object detection network model in an embodiment of this application;
[0026] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application;
[0027] Figure 7 This is a schematic diagram of the image display interface in an embodiment of this application;
[0028] Figure 8 This is a schematic diagram of the image upload interface in an embodiment of this application;
[0029] Figure 9 This is a schematic diagram of the structural framework of a data processing device provided in an embodiment of this application;
[0030] Figure 10 This is a schematic diagram of the structural framework of a data processing device provided in an embodiment of this application. Detailed Implementation
[0031] This application is described in detail below. Examples of embodiments of this application are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. Furthermore, detailed descriptions of known technologies that are unnecessary for the features of this application are omitted. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.
[0032] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0033] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.
[0034] First, let's introduce and explain several terms used in this application:
[0035] Image retrieval refers to the process of finding images related to user input.
[0036] Feature (feature vector): A feature is a vector that represents the information of an image after it has been processed by a model.
[0037] Automated Optical Inspection (AOI) is a device that uses optical principles to detect defects in images.
[0038] The ADC (Automatic Defect Classification) system integrates functions such as automatic defect classification, manual review, model scheduling and training, and statistical reporting. It uses deep learning and other image processing methods to automatically classify defect images and identify the location of defects.
[0039] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.
[0040] This application provides a data processing method, such as... Figure 1 As shown, the method includes:
[0041] S101, in response to the received data query command, download the corresponding product image from the stored product image data.
[0042] The product images in this application embodiment can be obtained by AOI equipment taking pictures of products on the production line or by AOI equipment taking pictures of products on the production line and then performing preliminary classification. The obtained product images can be stored in a DFS (Distributed File System) or FTP (File Transfer Protocol) server.
[0043] Optionally, before downloading the corresponding product image from the stored product image data in response to the received data query command, an image download interface may be displayed first. The data query command may be a command triggered by the user's operation on the image download interface.
[0044] like Figure 2 As shown, the image download interface may include at least one of the following: download configuration module, save path setting option, download status display area, progress bar, start download option, and stop option.
[0045] Users can configure download conditions through the "Download Configuration Module." Configurable conditions include at least one of the following: process segment, data source, site, defect type and quantity, and operator employee number. The configured download conditions can be included in the data query command. Downloading product images based on the download conditions specified in the data query command ensures that the downloaded product images strictly conform to specific conditions, guarantees accuracy, and reduces unnecessary downloads.
[0046] Users can set the save path for downloaded product images through the "Save Path Settings" option. The download process can be started by clicking the "Start Download" option. The success or failure of each product image download is displayed in the "Download Status Display Area," and the overall download progress is shown in the "Progress Bar." The download can be terminated by clicking the "Stop" option. During the download process, information for each product image can be written to a temporary database for later uploading.
[0047] The data sources in this embodiment are two: the DFS system and the FTP server. Based on the different data sources, the product images are downloaded differently in this embodiment.
[0048] When the data source in the data query command is the DFS system, since the DFS system's directory structure is stored by site and product, the DFS directories that meet the conditions for the site and product can be traversed first to generate a preparation database file. The path and related information of each product image are recorded in the preparation database. In this way, when the product image of the same site and product is extracted again, it will be read directly from the preparation database.
[0049] When the data source in the data query command is an FTP server, the FTP server is actually the server used by the ADC system to store data. The images in the FTP server have been written into the database by the ADC, so the database can be read directly, and then the required product images can be downloaded from the FTP server. The downloaded product images can be stored in folders by category, and the download-related information (data source, site, product, defect category, etc.) will be saved in a temporary database.
[0050] The image downloading method described above is more efficient than existing manual data collection methods, and can facilitate image retrieval quickly and easily.
[0051] Optionally, before downloading the corresponding product image from the stored product image data in response to a received data query instruction (e.g., before displaying the image download interface), the process further includes: displaying a login interface; and verifying the user's identity in response to a received login request. If verification is successful, the corresponding product image is downloaded from the stored product image data in response to the received data query instruction; if verification fails, the current process is stopped.
[0052] like Figure 3 As shown in the embodiments of this application, the login interface may include: a username input box, a password input box, login options, and user management options. The username input box and password input box are used to input the username and password, respectively. After inputting the username and password, the login option is triggered to perform identity verification. After inputting the username and password, the user management option is triggered to perform identity verification for the administrator account.
[0053] User information can be stored in any of the following ways:
[0054] Method 1: Integrate with the ADC system. All user management operations are performed through the ADC system. The local system transmits the received username and password to the ADC system, and the ADC system verifies the information and returns a result indicating whether the verification was successful.
[0055] Method two involves encrypting the received user information and storing it locally to prevent direct modification. This can be done by setting up an administrator account for access. Figure 3 The user management option allows you to access the local system's user management interface and manage all user information. The user management interface is as follows: Figure 4 As shown, the administrator can modify the user's password and permissions (such as process section access permissions) in this user management interface, and can create or delete a user.
[0056] S102, the product defects in each product image are identified and classified by the object detection network model to obtain the defect category and defect feature vector of the product image.
[0057] Optionally, such as Figure 5 As shown, the object detection network model includes: a feature extractor, a region candidate network, and an object detector.
[0058] Optionally, a target detection network model is used to identify and classify product defects in each product image. The defect categories and defect feature vectors of the product images include:
[0059] Feature extraction is performed on the product image using a feature extractor to obtain a feature map. A region candidate network is used to determine whether each feature region in the feature map contains product defects, thus obtaining feature regions containing product defects. A target detector is used to extract features from the feature regions containing product defects to obtain defect feature vectors for the feature regions. The category of product defects contained in each feature region is determined, and the defect category of the product image is determined based on the category of product defects contained in each feature region.
[0060] Optionally, the feature extractor includes convolutional layers, pooling layers, and multiple convolutional blocks, as shown in the reference. Figure 5 For example, the convolutional layer is located before the pooling layer, and multiple convolutional blocks are concatenated after the pooling layer; the feature extractor can divide the input product image into multiple feature regions, extract features from each feature region, obtain the feature vector of that feature region, and multiple feature vectors form a complete feature map.
[0061] In each concatenated convolutional block, such as Figure 5 The diagram can be divided into four parts. The first part includes three convolutional blocks, each containing three convolutional layers connected in series, with kernel sizes of 1*1*64, 3*3*64, and 1*1*256, respectively. The second part includes four convolutional blocks, each containing three convolutional layers connected in series, with kernel sizes of 1*1*128, 3*3*128, and 1*1*512, respectively. The third part includes 23 convolutional blocks, each containing three convolutional layers connected in series, with kernel sizes of 1*1*256, 3*3*256, and 1*1*1024, respectively. The fourth part includes three convolutional blocks, each containing three convolutional layers connected in series, with kernel sizes of 1*1*512, 3*3*512, and 1*1*2048, respectively.
[0062] Optionally, the regional candidate network includes: a convolutional layer, and a first convolutional branch and a second convolutional branch cascaded in parallel after the convolutional layer.
[0063] The convolutional layer in the region candidate network is used to: extract features from the feature map output by the feature extractor to obtain the feature vector of each feature region; the first convolutional branch is used to: determine whether the feature region contains product defects based on the feature vector of each feature region and output it; the second convolutional branch is used to: adjust the detection range of the feature region to cover the product defects.
[0064] Reference Figure 5The first convolutional branch may include a foreground / background classification convolutional layer and an output layer (softmax). The foreground / background classification convolutional layer can classify the feature vectors of each feature region in the feature map as foreground / background, and determine whether each feature region is foreground (i.e., product defect) or background. The output layer can output the classification result of the foreground / background classification convolutional layer (i.e., the feature region classified as foreground). The second convolutional branch may include a box regression convolutional layer. The box regression convolutional layer can adjust the detection range of the feature region to better cover the product defect. After the two convolutional branches are processed, the first N detection boxes classified as foreground can be output as the N feature regions containing the product defect.
[0065] Optionally, the target detector includes: a pooling layer, a fully connected layer, and a first fully connected branch and a second fully connected branch cascaded in parallel after the fully connected layer;
[0066] The pooling layer and fully connected layer in the target detector are used to: perform pooling and full-connection processing on the feature vectors of feature regions containing product defects to obtain defect feature vectors and output them; the first fully connected branch is used to: determine the category of product defects based on the defect feature vectors of each feature region, determine the defect category of the product image based on the category of product defects contained in each feature region and output it; the second fully connected branch is used to: adjust the detection range of the feature region to cover product defects.
[0067] Reference Figure 5 The pooling layer in the target detector can be an ROI (Region of Interest) pooling layer. Through the ROI pooling layer and the fully connected layer, the feature regions output by the region candidate network can be extracted again to output the defect feature vector.
[0068] Reference Figure 5 The first fully connected branch may include a classification fully connected layer and an output layer (softmax). The classification fully connected layer can classify the defect feature vectors extracted by the pooling layer and the fully connected layer, determine the specific category of the product defect corresponding to each defect feature vector and the confidence score of the category. The output layer can output the classification result of the classification fully connected layer, such as the category of the product defect with the highest confidence score, as the defect category of the product image. The second convolutional branch may include a bounding box regression fully connected layer, which can adjust the detection range of the feature region to better cover the product defects. After the two fully connected branches are processed, the category of the product defect in the product image can be output as the defect category of the product image.
[0069] Optionally, the target detection network model in this embodiment can also locate product defects in the product image during processing and output the location of the product defects.
[0070] The target detection network model used in this embodiment can be trained in advance using a large number of product images containing product defect types as sample data. The sample data can be product images downloaded from DFS and reviewed by senior engineers to ensure the accuracy of the defect categories of the product images in the sample data, thereby ensuring the accuracy of the target detection network model's classification.
[0071] S103, based on the defect feature vector of each product image, determine the defect sub-category of each product image in each defect category.
[0072] In an optional implementation, determining the defect sub-category of each product image in each defect category based on the defect feature vector of each product image includes: determining the similarity between product images in the defect category based on the defect feature vector of each product image in each defect category; and determining the defect sub-category of each product image in the defect category based on the similarity between product images in each defect category.
[0073] This implementation method is applicable to situations where the number of downloaded product images is small, and the applicable range of the number of images can be determined according to actual needs.
[0074] Optionally, similarity can be represented by Euclidean distance. For the defect feature vectors of each product image in the same defect category (which can be the defect feature vector corresponding to the category of the product defect with the highest confidence score in each product image, assuming that the vector is an n-dimensional vector), calculate the Euclidean distance value between every two defect feature vectors (to reflect the magnitude of the difference between the two defect feature vectors), and obtain multiple Euclidean distance values; determine whether each Euclidean distance value is less than a preset distance threshold. For each product image in the defect category, the product image corresponding to the Euclidean distance value that is less than the distance threshold (as a similarity condition) is regarded as a product image in the same defect subcategory as the product image.
[0075] The Euclidean distance is calculated as follows: Where x i Let y represent the i-th element of the first defect feature vector x to be calculated. i This represents the i-th element of the second defect feature vector to be calculated.
[0076] In another optional implementation, based on the defect feature vector of each product image, a defect subcategory for each product image within each defect category is determined, including:
[0077] In the multidimensional space to which the defect feature vector of the product image belongs in each defect category, multiple defect feature vectors are selected as the centroids of multiple defect subcategories. The defect subcategories of the product image are determined based on the distance between the defect feature vectors and the centroids of each defect subcategory.
[0078] The following classification operation is performed periodically until the iteration condition is met: the centroid of each defect subclass is redefined based on the spatial location of the defect feature vector contained in each defect subclass; the defect subclass of the product image is determined based on the distance between the defect feature vector and the redefined centroid of each defect subclass.
[0079] Optionally, each defect feature vector can be used as a point in a multidimensional space. When initially selecting the centroid, it can be selected in the following way:
[0080] A point is randomly selected from the points in each defect feature vector (the first point). When selecting the second point, the point farther away from the first point has a higher probability of being selected; when selecting the third point, the point farther away from the sum of the distances to the first and second points has a higher probability of being selected; and so on, until K points are obtained. The value of K can be preset according to actual needs, and each centroid represents a defect subcategory.
[0081] Optionally, when determining the defect sub-category of a product image based on the distance between the defect feature vector and the centroid of each defect sub-category, for the remaining points outside the centroid in the same multidimensional space, the distance between each point and each centroid is calculated. The point is closest to which centroid, and the defect feature vector of that point can be assigned to the defect sub-category represented by that centroid, thereby realizing the classification of each defect feature vector and thus the classification of product defects in each product image.
[0082] Optionally, when redetermining the centroid of a defect subclass based on the spatial location of the defect feature vectors contained in each defect subclass, the average of the spatial coordinates of the current points in each defect subclass is calculated, and the point represented by the average can be used as the new centroid of the defect subclass.
[0083] Optionally, the iteration conditions for the periodic classification operation can be preset according to actual needs. For example, it can be set as follows: for each defect subcategory, when the position change of the current centroid relative to the previously determined centroid is less than a certain threshold or the number of periodic classification operations reaches the maximum number of iterations T, the iteration condition is considered to be met, the periodic classification operation is stopped, and the defect subcategory determined based on the last determined centroid is the final defect subcategory.
[0084] This implementation method is applicable to situations where the number of downloaded product images (assuming it is N) is large. The time complexity of this implementation method is approximately N*K*T. When further subdividing a certain defect category, there are usually not many defect subcategories of the same defect category. Therefore, the value of K is usually a single digit, and the maximum number of iterations T is usually chosen to be tens to hundreds. Therefore, when the number of product images N is large, the classification speed of this implementation method is relatively fast.
[0085] Optionally, the data processing method provided in this application embodiment, based on the above steps S101 to S103, further includes:
[0086] The product images of each defect subcategory are input into the target detection network model for training.
[0087] Steps S101 to S103 enable automatic and refined classification of product images, quickly obtaining refined product images. These images are then used as samples to input into the target detection network model, allowing for retraining and further refining the model's accuracy. By using each obtained product image as a sample for retraining, the target detection network model can be continuously updated, thereby continuously improving its accuracy.
[0088] Optionally, such as Figure 6 As shown, the data processing method provided in this application embodiment, based on the above steps S101 to S103, further includes the following steps S104 to S106:
[0089] S104, display product images for each defect subcategory, then execute S105 and S106.
[0090] S105, in response to the received image deletion command, delete the corresponding product image.
[0091] S106, in response to the received image modification instruction, modify the defect subcategory of the corresponding product image.
[0092] The classified product images obtained in this application embodiment can be obtained through, for example... Figure 7 The interface shown is displayed line by line. Figure 7 In the image, defect patterns A1-A3 are three defect subcategories within defect category A, and defect patterns B1-B2 are two defect subcategories within defect category B. Product images of the same defect subcategory are displayed in the same row.
[0093] Users can delete unnecessary images from the displayed images. For example, for images with an excessive number of certain defect types, some redundant information that reduces the number of defect types can be deleted, while balancing the distribution of the number of defect types. If a user finds that the current defect subclass of an image is inaccurate, that is, the image is not displayed in the correct position, the user can move the image to the correct image display position to correct the defect subclass of the image.
[0094] For the same defect category, if a user believes that the images within it involve multiple defect forms and it is necessary to add defect subcategories, a new subcategory folder can be set up, and some images from that defect category can be moved into the new subcategory folder to improve the classification at that level.
[0095] For the same defect subcategory, if the user believes that the images within it involve multiple defect submorphologies and it is necessary to add a more refined defect subcategory, then a new subcategory folder is set up, and some images from that defect subcategory are moved into the new subcategory folder to achieve more refined classification.
[0096] After a user deletes or modifies an item, the information in the temporary database can be automatically modified to maintain consistency with the displayed result.
[0097] Optionally, the data processing method provided in this application embodiment, after the above steps S105 or S106, further includes: inputting the deleted or modified product image into the target detection network model, training the target detection network model, so as to further improve the accuracy of the target detection network model.
[0098] Optionally, the data processing method provided in this application embodiment further includes, after any of the above steps S103, S105 and S106: uploading the categorized product images.
[0099] The image upload interface is as follows Figure 8 As shown, through Figure 8 As shown in the interface, after completing the local dataset path settings and server configuration (server login information configuration), clicking "Start Upload" will initiate the upload of similar product images. The success or failure of each product image upload will be displayed in the "Upload Status Display Area," and the overall upload progress will be displayed in the "Progress Bar." Users can click "Stop" at any time to terminate the upload. During upload, users can choose to upload only product images or upload both product images and information from the temporary database.
[0100] Based on the same inventive concept, this application provides a data processing apparatus, such as... Figure 9As shown, the data processing device 900 includes: an image download module 901, a first defect classification module 902, and a second defect classification module 903.
[0101] The image download module 901 is used to download the corresponding product image from the stored product image data in response to a received data query command.
[0102] The first defect classification module 902 is used to identify and classify product defects in each product image through a target detection network model, and obtain the defect category and defect feature vector of the product image.
[0103] The second defect classification module 903 is used to determine the defect sub-category of each product image in each defect category based on the defect feature vector of each product image.
[0104] Optionally, the first defect classification module 902 is specifically used for: extracting features from the product image using a feature extractor to obtain a feature map; determining whether each feature region of the feature map contains a product defect using a region candidate network to obtain a feature region containing a product defect; extracting features from the feature region containing a product defect using a target detector to obtain a defect feature vector of the feature region, and determining the category of the product defect contained in each feature region, and determining the defect category of the product image based on the category of the product defect contained in each feature region.
[0105] In an optional implementation, the second defect classification module 903 is specifically used to: determine the similarity between product images in each defect category based on the defect feature vector of each product image in each defect category; and determine the defect sub-category of each product image in each defect category based on the similarity between each product image in each defect category.
[0106] In another alternative implementation, the second defect classification module 903 is specifically used for:
[0107] In the multidimensional space to which the defect feature vector of the product image belongs in each defect category, multiple defect feature vectors are selected as the centroids of multiple defect subcategories. The defect subcategories of the product image are determined based on the distance between the defect feature vectors and the centroids of each defect subcategory.
[0108] The following classification operation is performed periodically until the iteration condition is met: the centroid of each defect subclass is redefined based on the spatial location of the defect feature vector contained in each defect subclass; the defect subclass of the product image is determined based on the distance between the defect feature vector and the redefined centroid of each defect subclass.
[0109] Optionally, the data processing apparatus 900 provided in this application further includes a training module.
[0110] The training module is used to input product images of each defect subcategory into the target detection network model and train the target detection network model.
[0111] Optionally, the data processing apparatus 900 provided in this application further includes: an image display module, an image deletion module, and an image modification module.
[0112] The image display module is used to display product images for each sub-defect category.
[0113] The image deletion module is used to delete the corresponding product image in response to a received image deletion command.
[0114] The image modification module is used to modify the corresponding defect subcategory of the product image in response to the received image modification command.
[0115] The data processing device 900 in this embodiment can execute any of the data processing methods provided in the embodiments of this application. The implementation principles are similar. For the contents not shown in detail in this embodiment, please refer to the method embodiments described above. They will not be repeated here.
[0116] Based on the same inventive concept, embodiments of this application provide a data processing device, which includes a memory and a processor, wherein the memory and the processor are electrically connected.
[0117] The memory stores a computer program, which is executed by the processor to implement any of the data processing methods provided in the embodiments of this application.
[0118] Those skilled in the art will understand that the electronic devices provided in the embodiments of this application can be specifically designed and manufactured for the desired purpose, or may include known devices in general-purpose computers. These devices have computer programs stored therein that are selectively activated or reconfigured. Such computer programs can be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and respectively coupled to a bus.
[0119] In one optional embodiment, this application provides a data processing device, such as... Figure 10 As shown, the data processing device 1000 includes a memory 1001 and a processor 1002, which are electrically connected, such as via a bus 1003.
[0120] Optionally, the memory 1001 is used to store application code that executes the scheme of this application, and the execution is controlled by the processor 1002. The processor 1002 is used to execute the application code stored in the memory 1001 to implement any of the data processing methods provided in the embodiments of this application.
[0121] The memory 1001 may be ROM (Read-Only Memory) or other types of static storage devices capable of storing static information and instructions; it may be RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions; it may also be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0122] Processor 1002 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1002 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0123] Bus 1003 may include a pathway for transmitting information between the aforementioned components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0124] Optionally, the data processing device 1000 may also include a transceiver 1004. The transceiver 1004 can be used for receiving and transmitting signals. The transceiver 1004 allows the data processing device 1000 to communicate wirelessly or wiredly with other devices to exchange data. It should be noted that in practical applications, the transceiver 1004 is not limited to one.
[0125] Optionally, the data processing device 1000 may further include an input unit 1005. The input unit 1005 can be used to receive input numbers, characters, images, and / or sound information, or to generate key signal inputs related to user settings and function control of the electronic device 1000. The input unit 1005 may include, but is not limited to, one or more of the following: a touchscreen, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, a joystick, a camera, a microphone, etc.
[0126] Optionally, the data processing device 1000 may further include an output unit 1006. The output unit 1006 can be used to output or display information processed by the processor 1002. The output unit 1006 may include, but is not limited to, one or more of a display device, a speaker, a vibration device, etc.
[0127] Although Figure 10 A data processing apparatus 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0128] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the data processing methods provided in embodiments of this application.
[0129] The computer-readable medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM, RAM, EPROM (Erasable Programmable Read-Only Memory), EEPROM, flash memory, magnetic cards, or optical cards. In other words, a readable medium includes any medium by which a device (e.g., a computer) stores or transmits information in a readable form.
[0130] This application provides various optional implementations of a computer-readable storage medium suitable for any of the above data processing methods, which will not be described in detail here.
[0131] These images are numerous, and after being categorized and stored according to process section, site, product, etc., the directory structure is complex and has many levels, making it very difficult to select images that meet specific criteria. Currently, the factory engineers manually filter and collect the data, which is inefficient and prone to errors, such as missing some data or selecting inaccurate data categories.
[0132] By applying the technical solutions of the embodiments of this application, at least the following beneficial effects can be achieved:
[0133] 1) It can automatically download the corresponding product images according to the data query command, without the need for manual downloading one by one, which improves the efficiency of data query and makes data query more convenient and faster; it can automatically identify product defects in product images and automatically classify the downloaded product images according to the identified product defects, without the need for manual screening and classification, which can improve the efficiency and accuracy of image screening and classification.
[0134] 2) By using a target detection network model trained with a large amount of sample data to identify product defects, the accuracy of defect identification can be improved, and a preliminary classification of product images based on product defects can be achieved. Based on the defect feature vectors output by the target detection network model, the similarity between product images can be determined or the multidimensional space to which the defect feature vectors belong can be classified, which can realize secondary classification of product images based on the target detection network model and improve the accuracy of classification.
[0135] 3) The classified product images obtained through the technical solutions provided in the embodiments of this application can be input into the target detection network model to retrain the target detection network model and update the target detection network model to make its defect recognition and classification functions more accurate.
[0136] 4) This application embodiment can display the determined classified product images to the user, delete the corresponding product images based on the received image deletion command, and realize image deduplication when storing duplicate images in the product images; it can also modify the category of the corresponding product based on the received image modification command, so as to realize the function of image category correction and further refined classification.
[0137] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.
[0138] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more.
[0139] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0140] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data processing method, characterized in that, include: In response to a received data query command, the corresponding product image is downloaded from the stored product image data; The product defects in each product image are identified and classified by the target detection network model to obtain the defect category and defect feature vector of the product image; Based on the defect feature vectors of each product image, determine the defect sub-category of each product image in each defect category; The target detection network model includes: a feature extractor, a region candidate network, and a target detector connected in sequence; The region candidate network includes: a convolutional layer, and a first convolutional branch and a second convolutional branch cascaded in parallel after the convolutional layer; The convolutional layer is used to: extract features from the feature map output by the feature extractor to obtain feature vectors for each feature region; The first convolutional branch is used to: determine whether a feature region contains the product defect based on the feature vector of each feature region and output the result; The second convolutional branch is used to: adjust the detection range of the feature region to cover the product defect; The target detector includes: a pooling layer, a fully connected layer, and a first fully connected branch and a second fully connected branch cascaded in parallel after the fully connected layer; The pooling layer and the fully connected layer are used to: perform pooling and full-connection processing on the feature vector of the feature region containing the product defect, to obtain the defect feature vector and output it; The first fully connected branch is used to: determine the category of the product defect based on the defect feature vector of each feature region, determine the defect category of the product image based on the category of the product defect contained in each feature region, and output the result. The second fully connected branch is used to: adjust the detection range of the feature region to cover the product defect.
2. The data processing method according to claim 1, characterized in that, The product defects in each product image are identified and classified using a target detection network model to obtain the defect category and defect feature vector of the product image, including: The feature extractor is used to extract features from the product image to obtain a feature map. The region candidate network is used to determine whether each feature region of the feature map contains the product defect, thereby obtaining the feature region containing the product defect; The target detector extracts features from the feature regions containing the product defects to obtain the defect feature vectors of the feature regions, and determines the category of the product defects contained in each feature region. Based on the category of the product defects contained in each feature region, the defect category of the product image is determined.
3. The data processing method according to claim 1, characterized in that, The step of determining the defect sub-category of each product image within each defect category based on the defect feature vector of each product image includes: Based on the defect feature vector of each product image in each defect category, determine the similarity between the product images in that defect category; Based on the similarity between product images in each defect category, the defect subcategories of each product image in that defect category are determined.
4. The data processing method according to claim 1, characterized in that, The step of determining the defect sub-category of each product image within each defect category based on the defect feature vector of each product image includes: In the multidimensional space to which the defect feature vector of the product image belongs in each defect category, multiple defect feature vectors are selected as centroids of multiple defect subcategories, and the defect subcategories of the product image are determined based on the distance between the defect feature vectors and the centroids of each defect subcategory. The following classification operation is performed periodically until the iteration condition is met: the centroid of the defect sub-category is re-determined based on the spatial position of the defect feature vector contained in each defect sub-category; the defect sub-category of the product image is determined based on the distance between the defect feature vector and the re-determined centroid of each defect sub-category.
5. The data processing method according to claim 1, characterized in that, After determining the defect sub-category of each product image within each defect category, the process further includes: The product images of each defect subcategory are input into the target detection network model to train the target detection network model.
6. The data processing method according to claim 1, characterized in that, Also includes: Display the product images for each defect subcategory; In response to a received image deletion command, the corresponding product image is deleted; In response to a received image modification instruction, the corresponding defect subcategory of the product image is modified.
7. A data processing apparatus for performing the data processing method as described in any one of claims 1-6, characterized in that, include: The image download module is used to download the corresponding product image from the stored product image data in response to the received data query command; The first defect classification module is used to identify and classify product defects in each product image through a target detection network model, and obtain the defect category and defect feature vector of the product image. The second defect classification module is used to determine the defect sub-category of each product image in each defect category based on the defect feature vector of each product image.
8. A data processing device, characterized in that, include: Memory; The processor is electrically connected to the memory; The memory stores a computer program, which is executed by the processor to implement the data processing method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-6.