Commodity quality inspection method and system, quality inspection equipment, storage medium and program product

Abstract

The invention provides a commodity quality inspection method and system, quality inspection equipment, a storage medium and a program product, and relates to the technical field of image processing. According to the method, the commodity feature information is extracted from the commodity image, quality inspection is carried out on the commodity in combination with the commodity information, the commodity feature information can reflect more detailed information of the commodity, the commodity information can provide more background information for commodity quality inspection as assistance, more comprehensive verification information can be provided in combination with the commodity information, misjudgment is reduced, and the commodity quality inspection efficiency is improved. And the quality inspection accuracy and the quality inspection efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology. Specifically, it relates to a method, a system, a quality inspection device, a storage medium, and a program product for quality inspection of commodities. Background Art

[0002] At present, physical commodities are prone to wear, scratches, collisions, etc. during the processes of production, storage, transportation, and use. The resulting damage and defects of the commodities will affect the sales of the commodities. Therefore, it is very important to conduct quality inspection on the commodities for correctly evaluating the value of the commodities.

[0003] For the quality inspection of commodities, currently most still rely on manual inspection methods. Taking shoes as an example, operators need to take out the shoes from the shoe box according to the operation process and check whether there are defects in the shoe box, shoes, accessories, etc. one by one. This method is inefficient, costly, and easily affected by human factors, resulting in inaccurate inspection results. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, a system, a quality inspection device, a storage medium, and a program product for quality inspection of commodities, so as to improve the problems of low efficiency and inaccurate inspection results caused by the existing manual quality inspection method.

[0005] In a first aspect, the embodiments of the present application provide a method for quality inspection of commodities, and the method includes:

[0006] Obtain commodity feature information and commodity information, where the commodity feature information is obtained by extracting features from a commodity image, and the commodity information includes commodity attribute information, or the commodity information includes commodity attribute information and commodity circulation information;

[0007] Conduct quality inspection on the commodity according to the commodity feature information and the commodity information to obtain a commodity quality inspection result.

[0008] In the above implementation process, by extracting commodity feature information from the commodity image and combining the commodity information to conduct quality inspection on the commodity, the commodity feature information can reflect more detailed features of the commodity, and the commodity information can provide more background information as an auxiliary for commodity quality inspection. Combining the commodity information can provide more comprehensive verification information, reduce misjudgment, and improve the accuracy and efficiency of quality inspection.

[0009] Optionally, obtaining commodity feature information includes:

[0010] Extract commodity feature information from the commodity image by using an image feature extraction model;

[0011] Among them, the image feature extraction model includes a mask decoder, an image encoder, a pixel decoder, and a feature extractor. Extracting commodity feature information from a commodity image using the image feature extraction model includes:

[0012] Using the image encoder to perform image feature extraction on the obtained commodity image to obtain image features;

[0013] Using the mask decoder to obtain a defect mask feature according to the image feature and a query vector, where the number of the query vectors is related to the number of candidate defects;

[0014] Using the pixel decoder and the feature extractor to obtain commodity feature information according to the image feature and the defect mask feature.

[0015] In the above implementation process, using encoders, decoders, etc. in deep learning for feature extraction can effectively extract and process feature information in the image, thereby improving the accuracy and robustness of feature extraction.

[0016] Optionally, the feature extractor includes a multi-layer perceptron, an average pooling layer, and a multiplication operator. Using the pixel decoder and the feature extractor to obtain commodity feature information according to the image feature and the defect mask feature includes:

[0017] Using the pixel decoder to process the image feature to obtain a pixel feature;

[0018] Using the multi-layer perceptron to process the defect mask feature to obtain a defect mask position feature;

[0019] Using the multiplication operator to perform a multiplication operation on the defect mask position feature and the pixel feature to obtain a defect position prediction feature;

[0020] Using the average pooling layer to perform an average operation on the defect mask feature to obtain an image query feature;

[0021] Using the multi-layer perceptron to process the defect mask feature respectively to obtain an image category prediction feature and a defect category prediction feature;

[0022] Among them, the commodity feature information includes the defect position prediction feature, the image query feature, the image category prediction feature, and the defect category prediction feature.

[0023] In the above implementation process, features are extracted from multiple dimensions, including pixel-level, local, and global information. The fusion of this multi-dimensional information enables the model to more comprehensively understand the image content and improve the accuracy and robustness of feature extraction.

[0024] Optionally, the loss function during the training of the image feature extraction model is:

[0025] L = L postion + L class + wL image ;

[0026] Wherein, L postion represents the loss function corresponding to the defect position prediction feature, L postion = w1L focal + w2L dice , H represents the height of the commodity image, W represents the width of the commodity image, G i,j represents the probability that the point at position (i, j) in the ground truth mask image input during training is the foreground, P i,j represents the value of the point at position (i, j) in the defect position prediction feature obtained during training. When G i,j takes the value of 1, a t = a, P t = P i,j , when the current G i,j takes the value of 0, a t = 1 - a, P t = 1 - P i,j , a is a parameter for balancing foreground and background pixel categories, and γ is a parameter for balancing easy and difficult samples; G, P are matching pairs obtained by matching M ground truth mask images and N defect position prediction features through the Hungarian algorithm. G represents the number of pixel points in the foreground part of the ground truth mask image, P represents the number of pixel points in the foreground part of the defect position prediction feature, G ∩ P represents the number of overlapping pixel points in the foreground part of the ground truth mask image and the defect position prediction feature, and w1, w2 represent weights;

[0027] L class represents the cross-entropy loss function corresponding to the defect category prediction feature, and L image represents the CE loss function corresponding to the image category prediction feature, and w represents the loss weight corresponding to the image category prediction feature.

[0028] In the above implementation process, by optimizing the loss function corresponding to the defect position prediction feature, the model can more accurately predict the mask of the defect area, thereby improving the segmentation accuracy. Moreover, by combining the pixel-level difference and the global matching degree, the model can simultaneously focus on local details and the overall structure to generate a more accurate mask.

[0029] Optionally, the quality inspection of the commodity according to the commodity feature information and the commodity information to obtain the commodity quality inspection result includes:

[0030] Perform feature fusion and normalization processing on the image query feature and the image category prediction feature to obtain the global feature of the commodity image;

[0031] Perform feature fusion and normalization processing on the defect position prediction feature, the defect category prediction feature and the commodity information feature corresponding to the commodity information to obtain the commodity defect feature;

[0032] Perform quality inspection on the commodity according to the global feature of the commodity image and the commodity defect feature to obtain the commodity quality inspection result.

[0033] In the above implementation process, through multi-dimensional feature fusion, defects and normal areas can be identified more accurately, the possibility of misjudgment can be reduced, and the accuracy of quality inspection can be improved.

[0034] Optionally, the performing quality inspection on the commodity according to the global feature of the commodity image and the commodity defect feature to obtain the commodity quality inspection result includes:

[0035] Using a multi-modal model, based on the global feature of the commodity image and the commodity defect feature, obtain a defect score, a defect category, and a defect position, and the commodity quality inspection result includes the defect score, the defect category, and the defect position.

[0036] In the above implementation process, the global feature of the commodity image and the commodity defect feature respectively characterize the quality state of the commodity from different perspectives. The global feature provides overall information, while the defect feature provides local details. By combining these two features, the model can evaluate the quality of the commodity more comprehensively, thereby improving the accuracy of quality inspection.

[0037] Optionally, the commodity image is a foreground image obtained by taking pictures of the commodity from multiple angles. In this way, all-round quality inspection of the commodity can be performed, omissions can be reduced, and the accuracy of quality inspection can be improved.

[0038] In a second aspect, an embodiment of the present application provides a commodity quality inspection system, and the system includes:

[0039] A plurality of image collectors, which are respectively arranged at different positions in the commodity quality inspection area, and are used to collect images of the commodity in the commodity quality inspection area at different positions;

[0040] Quality inspection equipment, which is used to extract features from the commodity images collected by the plurality of image collectors to obtain commodity feature information, and perform quality inspection on the commodity according to the commodity feature information and the obtained commodity information to obtain the commodity quality inspection result, wherein the commodity information includes commodity attribute information, or the commodity information includes commodity attribute information and commodity circulation information.

[0041] In a third aspect, an embodiment of the present application provides a quality inspection device, including a processor and a memory. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect above are run.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the first aspect above are run.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer program instructions. When the computer program instructions are read and run by a processor, the steps in the method provided in the first aspect above are executed.

[0044] Other features and advantages of the present application will be described in the subsequent specification. And, partly, they will become obvious from the specification, or can be understood by implementing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a structural block diagram of a commodity quality inspection system provided by an embodiment of the present application;

[0047] Figure 2 It is a flowchart of a commodity quality inspection method provided by an embodiment of the present application;

[0048] Figure 3 It is a structural schematic diagram of an image feature extraction model provided by an embodiment of the present application;

[0049] Figure 4 It is a structural schematic diagram of a quality inspection model provided by an embodiment of the present application;

[0050] Figure 5 It is a structural block diagram of a commodity quality inspection device provided by an embodiment of the present application;

[0051] Figure 6 It is a structural schematic diagram of a quality inspection device for executing a commodity quality inspection method provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0053] It should be noted that the terms "system" and "network" in the embodiments of the present invention can be used interchangeably. "Multiple" means two or more. In view of this, in the embodiments of the present invention, "multiple" can also be understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " generally represents an "or" relationship between the associated objects before and after, unless otherwise specified.

[0054] It should also be noted that all actions of obtaining signals, information, or data in this application are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and with the authorization given by the owner of the corresponding device.

[0055] The embodiment of the present application provides a commodity quality inspection method. This method extracts commodity feature information from a commodity image and combines it with commodity information to perform quality inspection on the commodity. The commodity feature information can reflect more detailed features of the commodity, while the commodity information can provide more background information as an auxiliary for commodity quality inspection. Therefore, by combining commodity information, more comprehensive verification information can be provided, reducing misjudgment and improving the accuracy and efficiency of quality inspection.

[0056] First, the commodity quality inspection system 10 applied to the quality inspection method of the present application will be introduced below, as Figure 1 shown. This system includes a plurality of image collectors 11 and a quality inspection device 12. The plurality of image collectors 11 can be respectively arranged at different positions in the commodity placement area and are used to collect images of commodities in the commodity quality inspection area at different positions. The quality inspection device 12 is used to extract features from the commodity images collected by the plurality of image collectors 11 to obtain commodity feature information, and perform quality inspection on the commodity according to the commodity feature information and the obtained commodity information to obtain a commodity quality inspection result. Here, the commodity information includes commodity attribute information and commodity circulation information. The quality inspection device 12 is also used to execute the commodity quality inspection method of this solution.

[0057] Among them, the commodity quality inspection area refers to the area for placing commodities to be inspected. For example, the commodity quality inspection area can be a commodity cabinet. When quality inspection is required, the commodity can be placed in the commodity cabinet. Image collectors 11 can be installed at different positions in the commodity cabinet. The image collectors 11 are cameras. For example, cameras can be installed on the upper, lower, left, right, front, and rear surfaces in the commodity cabinet to capture images of the commodity from different angles, so as to obtain multi-angle images of the commodity. At this time, multiple commodity images can be obtained. For each commodity image, the commodity quality inspection method of this solution can be used for quality inspection, so as to detect the defect conditions of different angles of the commodity and achieve full-range quality inspection.

[0058] The quality inspection device 12 can refer to a host computer, such as a terminal, a computer, etc. A sensor can also be set in the commodity quality inspection area. The sensor is used to detect whether there is a commodity placed in the commodity quality inspection area. When the sensor detects that there is a commodity placed, it can send a signal to the quality inspection device 12. After receiving the signal, the quality inspection device 12 can control multiple image collectors 11 to perform image acquisition. The multiple image collectors 11 send the acquired commodity images to the quality inspection device 12, and the quality inspection device 12 can perform subsequent quality inspection processes after receiving the commodity images. In this case, the quality inspection device 12 is at a remote end.

[0059] The quality inspection device 12 can also be a processor installed on the commodity cabinet. At this time, the quality inspection device 12 and the image collector 11 can be understood as an integrated device, that is, the commodity cabinet.

[0060] The following details the implementation process of the commodity quality inspection method executed by the quality inspection device 12.

[0061] Please refer to Figure 2 , Figure 2 which is a flowchart of a commodity quality inspection method provided by an embodiment of this application. The method includes the following steps:

[0062] Step S110: Obtain commodity feature information and commodity information.

[0063] The commodities involved in this solution can include but are not limited to: shoes, clothes, bags, watches, accessories, and other commodities.

[0064] Among them, the commodity feature information is obtained by extracting features from the commodity image. The commodity image can be obtained by multi-directional image acquisition of the commodity by multiple image collectors as in the above embodiment. For example, the commodity can be placed in the commodity quality inspection area, and cameras are installed at multiple angles in the commodity quality inspection area. Through these cameras, multi-directional image acquisition of the commodity can be performed. For example, 6 cameras are used to capture images of the front, rear, left, right, upper, and lower angles of the shoes respectively, and each captured image is used as a commodity image for feature extraction. Here, relevant feature extraction algorithms can be used to extract commodity feature information from the commodity image.

[0065] In some embodiments, the product image is a foreground image obtained by taking pictures of the product from multiple angles. For example, after multiple image collectors capture images of the product at different positions and send them to the quality inspection device, for each product image, the quality inspection device can use an image segmentation algorithm (such as FCN, Segnet, maskformer, etc.) to segment the product area in the product image to extract the image of the product area, and this image can be used as the image for subsequent product quality inspection to avoid the influence of too much background area in the originally collected image on the product quality inspection result. In this way, the product can be comprehensively quality inspected, reducing omissions and improving the accuracy of quality inspection.

[0066] To improve the accuracy of quality inspection, it is also necessary to obtain product information. The product information includes product attribute information, or the product information includes product attribute information and product circulation information. The product attribute information may include but is not limited to the basic information of the product, such as spu (Standard Product Unit), sku (Stock Keeping Unit), size, article number, color, material, style, price, type, etc. The product circulation information may include but is not limited to the circulation record, sales channel, sales record, production date, batch, historical quality inspection information, etc.

[0067] Among them, the attribute information of the product can be obtained by comparing the product image. For example, compare the product image collected from the front of the product with the product images stored in the product database, and match the most similar product image in the product database. The product database also records the product images corresponding to various products and the attribute information of the product. Therefore, after matching the product image, the attribute information of the product can be obtained from the product database.

[0068] In some embodiments, the product attribute information can also be directly obtained from the product database. For example, according to the basic information of the product to be quality inspected uploaded by the user, more product attribute information can be retrieved from the product database based on the basic information.

[0069] The product database can also store the product circulation information of each product. After obtaining the above product attribute information, the product circulation information of the product can be obtained from the product database. Or after determining the product attribute information, the circulation information of the product can be found from other databases storing product circulation information according to the product attribute information, or the product circulation information can also be retrieved from the product database according to the basic information of the product to be quality inspected uploaded by the user.

[0070] Step S120: Quality inspect the product according to the product feature information and the product information to obtain the product quality inspection result.

[0071] After obtaining the product feature information and product information, the model can be used to combine this information to conduct quality inspection on the product. For example, if the product is shoes, information such as the defect category, defect score, and defect location of each defect on the shoes can be detected based on this information. Each defect can correspond to a defect category, a defect score, and a defect location, and then this information can be output as the product quality inspection result for quality inspectors to refer to.

[0072] The defect score can be understood as representing the severity of the defect. The higher the defect score, the more severe the defect; the lower the defect score, the lower the severity of the defect.

[0073] In some embodiments, the product quality inspection result may include a quality inspection score, which can be comprehensively obtained based on information such as the defect category and defect score. For example, the defect scores of each defect category can be weighted or added to obtain the quality inspection score, and then the quality inspection score can be output as the product quality inspection result to the quality inspector. The lower the quality inspection score, the more defects the product has; the higher the quality inspection score, the fewer defects the product has.

[0074] In some embodiments, the product quality inspection result may include whether the inspection is qualified. For example, it can be determined whether the product is qualified based on information such as the defect category, defect score, and defect location of the shoes (specifically, the corresponding judgment rules can be set according to actual needs), and whether it is qualified is output as the product quality inspection result to the quality inspector.

[0075] In some embodiments, the product can be quality inspected according to predefined rules and thresholds, and these rules can be formulated based on the product information and product feature information. For example, there are obvious patterns in a certain area of the product. Directly based on the product feature information, it may be determined as a defect, but in combination with the product information, if this area is alligator skin and has patterns by itself, it will not be determined as a defect. Another example is that for some product defects, it is not clear whether they are caused by wear or intentional aging of the product itself. By combining the product circulation information and finding that it has been circulated many times, it can be inferred that the defect is caused by wear. Another example is that the manifestation of the same defect is different in different colors (such as white products or black products) and different materials, that is, if the product attribute information is different, the quality inspection results for the same defect may also be different. In this way, the product can be quality inspected according to the product feature information and product information according to the predefined rules, and the product quality inspection result can be output.

[0076] In the above implementation process, by extracting the product feature information from the product image and combining the product information to conduct quality inspection on the product, the product feature information can reflect more detailed features of the product, while the product information can provide more background information as an auxiliary for the product quality inspection. By combining the product information, more comprehensive verification information can be provided, reducing misjudgment and improving the accuracy and efficiency of quality inspection.

[0077] Based on the above embodiments, in the manner of obtaining the product feature information, an image feature extraction model can also be used to extract the product feature information, where the image feature extraction model includes a mask decoder, an image encoder, a pixel decoder, and a feature extractor.

[0078] The structure of the image feature extraction model can be as Figure 3 shown. The product image can be input into the image encoder, and the image encoder is used to extract features from the product image to obtain image features. The image features can be input into the pixel decoder and the mask decoder. The pixel decoder can be used to further perform pixel-level feature extraction on the image features to obtain pixel features. The mask decoder can be used to extract features from the input query vector and image features to obtain defect mask features. The feature extractor (which can include a Multilayer Perceptron (MLP), an average pooling layer, and a multiplication operator) is then used to further extract features from the pixel features and defect mask features to obtain the product feature information.

[0079] The image encoder can be used to perform image feature extraction on the obtained product image to obtain image features. The mask decoder can be used to obtain defect mask features based on the image features and the query vector, where the number of query vectors is related to the number of candidate defects. The pixel decoder and the feature extractor are used to obtain the product feature information based on the image features and the defect mask features.

[0080] Among them, the image encoder can be used to extract the feature description of the input product image, that is, perform downsampling encoding on the product image to obtain image features. It can be implemented based on open-source network architectures such as resnet, vit, swin-transformer, and convnext. Taking convnext as an example of the image encoder. The 3*H*W product image can be encoded into a C1*H / s*W / s feature vector. To ensure the extraction performance of the image encoder for image detail features, H = W = 1024, C1 = 1024 can be set, where C1 is the number of channels of the image features, and s = 32, where s is the downsampling factor of the image. Of course, the specific values of each parameter can be flexibly set according to actual needs.

[0081] The mask decoder refers to taking the query vector and image features as inputs, and after processing, outputting defect mask features. The defect mask features can include a mask generated for each detected defect. In some embodiments, the mask decoder can be a decoder based on the Transformer structure. In a specific implementation, the corresponding number of layers of the Transformer can be selected according to the dimension of the input features to process the input features. For example, 6 layers of the Transformer can be selected for processing according to the experimental results.

[0082] Generally, the upper limit of the number of candidate defects appearing in a commodity image generally does not exceed 50. Therefore, when setting the number of query vectors, it can be greater than the number of candidate defects, that is, N is greater than or equal to 50. If the number of query vectors is N, it is related to the number of candidate defects. For example, the query vector is a two-dimensional vector of N*C, where N is the number of candidate defects, and the specific value can be determined according to the application scenario. For example, N = 64 is used as the number of query vectors. Here, C represents the length of the query vector. To balance accuracy and computational efficiency, C can be set to 256. Initially, the query vector can be initialized in various ways, such as random initialization or all initialized to 0, etc.

[0083] The input image features are the key vector and value vector of the Transformer structure in the mask decoder.

[0084] The query vector and the image features pass through the mask decoder of the 6-layer Transformer, and a defect mask feature necklace of N*S*C2 can be output. Here, N = 64 is the number of input query vectors, S = 6 is the number of layers of the mask decoder, and C2 is the number of channels of the feature vector. C2 can be set to 256. In this solution, the intermediate results of the 6-layer mask decoder can be selected to be all output as the defect mask features, which can improve the feature expression ability of the mask features.

[0085] Among them, the defect mask feature can represent the feature representation of the defect area in the commodity image. The defect mask feature can be a feature map with the same size as the commodity image. Among them, the value of each pixel represents whether the pixel belongs to a certain defect. Usually, the value of the mask feature is binary 0 or 1, where 1 means the pixel belongs to a certain defect, and 0 means the pixel does not belong to a certain defect. Or in some other ways, the value of the mask feature can also be continuous, such as a probability value between 0 and 1, indicating the probability that the pixel belongs to a certain defect. Therefore, by obtaining the defect mask feature, it can be used to identify and locate the defect area in the commodity image. Through the defect mask feature, it can be known which pixels belong to the defect area and which pixels belong to the normal area.

[0086] The pixel decoder can map the input image features to pixel features with the same length and width as the original image by means of upsampling. In some embodiments, the pixel decoder can be implemented by using a Feature Pyramid Network (FPN).

[0087] In the above implementation process, using encoders, decoders, etc. in deep learning for feature extraction can effectively extract and process the feature information in the image, thereby improving the accuracy and robustness of feature extraction.

[0088] Based on the above embodiments, the feature extractor may include a multi-layer perceptron, an average pooling layer, and a multiplier. In the method of obtaining the commodity feature information, the pixel decoder may be used to process the image features to obtain pixel features, the multi-layer perceptron may be used to process the defect mask features to obtain defect mask position features, and then the multiplier may perform a multiplication operation on the defect mask position features and the pixel features to obtain defect position prediction features. The average pooling layer may perform an average operation on the defect mask features to obtain image query features, and the multi-layer perceptron may be used to process the defect mask features respectively to obtain image category prediction features and defect category prediction features. Among them, the commodity feature information includes the above-mentioned defect position prediction features, image query features, image category prediction features, and defect category prediction features.

[0089] Among them, the multi-layer perceptron is a feed-forward neural network that can be composed of multiple fully connected layers and is usually used for feature transformation and classification tasks. In this solution, the multi-layer perceptron can transform the input features into higher-level feature representations.

[0090] When processing the input defect mask features, the defect mask features may represent pixel-level information of the defect area in the commodity image. These information may be local and fragmented. The multi-layer structure of the multi-layer perceptron can gradually extract the global information of the defect mask features and generate a feature vector that can represent the entire defect area, that is, the defect mask position feature. The defect mask position feature can represent the global information of the defect area and is a low-dimensional vector with a dimension of N*C2, which can capture the overall features of the defect area.

[0091] The pixel features are local features extracted from the image features and are usually a high-dimensional feature map representing the feature information of each pixel, which can capture the local details in the image.

[0092] The defect mask position features provide the overall information of the defect area, such as the shape, size, position, etc. of the defect, and the pixel features provide the detailed information of each pixel, such as texture, color, etc. The multiplier can perform a multiplication operation on the defect mask position features and the pixel features, which can combine the global information and the local information to generate a comprehensive feature representation, that is, the defect position prediction features, which can more accurately predict the position and shape of the defect area.

[0093] The multiplication operation can be regarded as an attention mechanism. The defect mask position feature can be regarded as a weight vector, representing the importance of the defect area. By multiplying the defect mask position feature with the pixel feature, the pixel feature can be weighted, highlighting the features of the defect area and suppressing the features of the background area. The finally generated defect position prediction feature can represent the probability that each pixel belongs to the defect area. Through the multiplication operation, the global information of the defect mask position feature can be combined with the local information of the pixel feature to generate a probability map.

[0094] Among them, the dimension of the input defect mask position feature is N*C2, and the dimension of the pixel feature is C2*H*W. A three-dimensional matrix of N*H*W is obtained through matrix multiplication. N represents the number of predicted defects, and H*W is a two-dimensional mask image of the same size as the original commodity image, where the pixel value ranges from 0 to 1, and the value of each pixel represents the probability that the current pixel is a foreground defect; this defect position prediction feature determines whether the pixel is a foreground defect or a background point through the threshold t. In this solution, t can be set to 0.5. If the pixel value > 0.5, it is the pixel corresponding to the foreground defect, otherwise it is a background point.

[0095] Performing an average operation on the defect mask feature can integrate the local information in the defect mask feature into global information. This global information can represent the features of the entire image, rather than just the features of local regions. The feature vector after the average operation can be used as the query feature of the image, which can characterize the overall features of the defect object in the image, including information such as shape, position, and category, and can be used for subsequent image retrieval, classification, or segmentation tasks. The dimension of the image query matrix is N*C2.

[0096] The image category prediction feature is obtained after the defect mask feature is processed by a multi-layer perceptron, and the output dimension is N*2. 2 indicates whether the current image has defects or not. One position represents having defects, and the other position represents being defect-free.

[0097] The defect category prediction feature is also obtained after the defect mask feature is processed by a multi-layer perceptron, and the output dimension is N*(K + 1). K represents the number of candidate defect categories. For example, if the value is 17, it can be set according to the actual situation. +1 represents the non-defect category of the background. This feature can obtain the defect category of each target after softmax, including the background. The defect category prediction feature can characterize the category information of local defects in the image, such as whether there are defects in a certain area and the category of the defects. It is a probability vector and can be used to analyze the local defects in the image in detail.

[0098] The defect categories here can be set according to the commodity. For example, for shoes, the set defect categories can include damage, dirt, scratches, oxidation and yellowing, uneven lengths, etc. The specific categories can be set according to the actual situation.

[0099] In the solution of this application, the image query feature and the image category prediction feature can be used as global defect features, and the defect position prediction feature and the defect category prediction feature can be used as basic defect features. Combining the global defect features and the basic defect features in this way can effectively detect the defects of the commodity and improve the accuracy of detection.

[0100] In the above implementation process, features are extracted from multiple dimensions, including pixel-level, local, and global information. The fusion of this multi-dimensional information enables the model to understand the image content more comprehensively and improves the accuracy and robustness of feature extraction.

[0101] On the basis of the above embodiments, after obtaining the image query feature, the defect position prediction feature, the image category prediction feature, and the defect category prediction feature, these features can be used to perform quality inspection on the commodity. Specifically, the image query feature and the image category prediction feature can be subjected to feature fusion and normalization processing to obtain the global feature of the commodity image. The defect position prediction feature, the defect category prediction feature, and the commodity information feature corresponding to the commodity information are subjected to feature fusion and normalization processing to obtain the commodity defect feature. Then, the commodity is subjected to quality inspection according to the global feature of the commodity image and the commodity defect feature to obtain the commodity quality inspection result.

[0102] Here, quality inspection can be performed by processing the above features through a quality inspection model. The quality inspection model is as Figure 4 shown. The image query feature and the image category prediction feature can be input into the image feature processing module for feature fusion and normalization processing. The image feature processing module can include a fully connected layer and a normalization layer, and the normalization layer is the LayerNorm layer.

[0103] The fully connected layer can be used to convert the input features into output features for feature mixing and transformation. The fully connected layer linearly combines the input image query feature and the image category prediction feature through a weight matrix, and then introduces non-linearity through an activation function, thereby converting the input features into a higher-level feature representation.

[0104] The normalization layer is used to perform normalization processing on the features. This normalization processing can reduce the scale difference between different features and prevent certain features from dominating during the training process.

[0105] The dimension of the global feature of the commodity image output after passing through the image feature processing module is (M + 1) * C5, where M is the number of image query features, and C5 can take the value of 256, which is the length of the image feature encoding.

[0106] The defect location prediction features and the defect category prediction features can be input into the text encoder for encoding to obtain a coding vector. The product information can also be input into the text encoder for encoding to obtain a coding vector. The two coding vectors can be input into the text feature processing module for feature fusion and normalization processing.

[0107] The text feature processing module includes a fully connected layer and a normalization layer. The two encoding vectors can be fused and normalized through the fully connected layer and the normalization layer, so that the features can be mapped to the same dimension. The dimension of the product defect feature output by the text feature processing module is (K+1)*C5, and C5 can take a value of 256, which is the length of the image feature code.

[0108] After obtaining the global features of the product image and the product defect features, these two features can be combined to perform quality inspection on the product. The global features of the product image can locate the defective area in the defective image, and the product defect features can locate the specific defect position, defect category and other information. Therefore, combining these features can accurately detect whether there are defects in the product, the defect category and other information.

[0109] In the above implementation process, multi-dimensional feature fusion can more accurately identify defects and normal areas, reduce the possibility of misjudgment, and improve quality inspection accuracy.

[0110] Based on the above embodiment, a multimodal model can be used to obtain defect scores, defect categories and defect locations according to the global features of the product image and the product defect features. The product quality inspection results include defect scores, defect categories and defect locations.

[0111] like Figure 4 As shown, the multimodal model may include a multimodal feature interaction module and a multilayer perceptron, wherein the global features of the product image and the product defect features may be input into the multimodal feature interaction module, which may be an encoder-decoder structure and may be composed of multi-head attention and Cross-Attention. The encoder may map the input features to a latent semantic space, extract image features and defect features through multi-head attention, and the decoder maps the latent semantic space to an output sequence, using the product defect features as a query, and the global features of the product image as keys and values, and fills them through cross-attention, thereby realizing information interaction between different modalities and capturing the correlation of features across modalities. The output of this module is connected to three multilayer perceptrons, respectively, and the defect score, defect category and defect location are output respectively. The defect score is the score corresponding to each defect category. The higher the score, the greater the severity of the defect. Conversely, the lower the score, the smaller the severity of the defect. The defect location refers to the specific location of the defect in the image.

[0112] The global features of the product can be used to represent the defect status of the entire image, and the product defect feature information can be used to characterize the detailed information of the defects in the image. After fusing these two features multimodally and combining the global and local information, through a multi-layer perceptron respectively, information such as defect scores, defect categories, and defect positions can be obtained. That is, the quality inspection results can include information such as defect scores, defect categories, and defect positions. Of course, the quality inspection results can also be a result of whether the quality inspection is qualified judged comprehensively based on information such as defect scores, defect categories, and defect positions. The specific form of the quality inspection results can be set according to the actual situation.

[0113] In the above implementation process, the global features of the product image and the product defect features respectively characterize the quality status of the product from different perspectives. The global features provide overall information, while the defect features provide local details. By combining these two features, the model can more comprehensively evaluate the quality of the product, thereby improving the accuracy of quality inspection.

[0114] Based on the above embodiments, the above image feature extraction model and quality inspection model can be combined into a product quality inspection model. The output of the image feature extraction model is connected to the input of the quality inspection model. The product quality inspection model can be pre-trained. During training, the image feature extraction model and the quality inspection model can be trained separately, that is, each uses a loss function for training, or they can be trained as an overall product quality inspection model.

[0115] Among them, during training, for an image, assume that the input ground truth mask image is gt_mask (abbreviated as G), and the class label of each ground truth mask image is gt_label. After binarizing the predicted defect position features, the IOU (Intersection Over Union) value of the ground truth mask and the predicted mask (i.e., the predicted defect position features) is used as an evaluation index. IOU can be used to measure the similarity between the predicted mask and the true mask. M ground truth masks and N predicted masks can be matched through the Hungarian algorithm to obtain M ground truth (G)-predicted (P) matching pairs. IOU is the ratio of the position intersection (A∩B) of the predicted mask A and the ground truth mask B to the position union (A∪B) of the predicted mask A and the ground truth mask B.

[0116] During the training process, for the image feature extraction model, its loss function during training can be:

[0117] L = L postion + L class + wL image ;

[0118] Among them, L postion represents the loss function corresponding to the predicted defect position features, and L postion = w1L focal+w2L dice , H represents the height of the product image, W represents the width of the product image, and G i,j represents the value at the point (i, j) in the ground truth mask image input during training, and P i,j represents the probability that the point (i, j) in the defect location prediction feature obtained during training is the foreground. When G i,j takes the value of 1 (indicating the foreground), a t = a, P t = P i,j , when the current G i,j takes the value of 0 (indicating the background), a t = 1 - a, P t = 1 - P i,j , a is a parameter for balancing foreground and background pixel classes, which can be determined empirically, such as taking the value of 0.25; γ is a parameter for balancing easy and difficult samples, which can be determined empirically, such as taking the value of 2; G and P are the matching pairs obtained by matching M ground truth mask images and N defect location prediction features through the Hungarian algorithm. G represents the number of pixel points in the foreground part of the ground truth mask image, P represents the number of pixel points in the foreground part of the defect location prediction feature, G∩P represents the number of overlapping pixel points in the foreground parts of the ground truth mask image and the defect location prediction feature, and w1 and w2 represent weights, which can be determined empirically, such as taking the values of 18 and 1. Among them, L dice in represents the IOU.

[0119] The loss function corresponding to the above defect location prediction feature can be used to measure the spatial consistency between the predicted mask and the true mask. During the model training process, the segmentation accuracy of the model for the defect area can be improved by optimizing this loss function.

[0120] In this loss function, pixel-level differences (L focal ) and mask matching degree (L image ) are considered. L focal can be used to measure the difference between the predicted mask and the true mask at the position (i, j). a is a balancing weight, which can be used to handle the class imbalance problem between foreground and background pixels. For foreground pixels, G i,j = 1, and for background pixels G i,j = 0. By introducing the balancing weight, the model can better handle the class imbalance problem between foreground and background pixels, thereby improving the segmentation accuracy of the defect area. This L focal By calculating the binary cross-entropy loss at each pixel position, the model can learn how to more accurately predict the class (foreground or background) of each pixel.

[0121] Ldice A global metric (such as IOU) is introduced, which enables the model to not only focus on the accuracy of local pixels but also on the matching degree of the overall mask. This helps the model generate a more complete mask rather than just accurately predicting locally.

[0122] In addition, L class represents the cross-entropy loss function corresponding to the defect category prediction feature and is defined as where g i is the ground truth, y i is the predicted class probability, and K represents the number of defect categories of candidates.

[0123] L image represents the CE loss function corresponding to the image category prediction feature and is defined as L image = -0.5[logP0 + log(1 - P1)], where P0 represents the probability that the first image in the image category prediction feature has defects, P1 represents the probability that the second image in the image category prediction feature has no defects, and w represents the loss weight corresponding to the image category prediction feature, which can be set according to experience, such as taking the value of 10.

[0124] It can be understood that for the quality inspection model, corresponding loss functions can also be used for training. During the training process, when the loss functions of the image feature extraction model and the quality inspection model reach the convergence condition respectively, it indicates that the training is completed. The image feature model and the quality inspection model after training can be used for subsequent product quality inspection.

[0125] In the above implementation process, by optimizing the loss function corresponding to the defect position prediction feature, the model can more accurately predict the mask of the defect area, thereby improving the segmentation accuracy. Moreover, by combining the pixel-level difference and the global matching degree, the model can focus on both local details and the overall structure simultaneously to generate a more accurate mask.

[0126] Please refer to Figure 5 , Figure 5 which is the structural block diagram of a product quality inspection device 200 provided by an embodiment of the present application. The device 200 can be a module, a program segment, or code on a quality inspection device. It should be understood that the device 200 corresponds to the above Figure 2 method embodiment and can execute Figure 2 each step involved in the method embodiment. The specific functions of the device 200 can be seen in the above description. To avoid repetition, the detailed description is appropriately omitted here.

[0127] Optionally, the device 200 includes:

[0128] An information acquisition module 210, configured to acquire product feature information and product information, where the product feature information is obtained by performing feature extraction on a product image, and the product information includes product attribute information, or the product information includes product attribute information and product circulation information;

[0129] A quality inspection module 220, configured to perform quality inspection on a product according to the product feature information and the product information to obtain a product quality inspection result.

[0130] Optionally, the information acquisition module 210 is configured to extract product feature information from a product image by using an image feature extraction model;

[0131] Wherein, the image feature extraction model includes a mask decoder, an image encoder, a pixel decoder, and a feature extractor. The information acquisition module 210 is specifically configured to perform image feature extraction on the obtained product image by using the image encoder to obtain image features; use the mask decoder to obtain a defect mask feature according to the image features and a query vector, where the number of the query vectors is related to the number of candidate defects; use the pixel decoder and the feature extractor to obtain product feature information according to the image features and the defect mask feature.

[0132] Optionally, the feature extractor includes a multi-layer perceptron, an average pooling layer, and a multiplication operator. The information acquisition module 210 is configured to process the image features by using the pixel decoder to obtain pixel features; process the defect mask features by using the multi-layer perceptron to obtain defect mask position features; perform a multiplication operation on the defect mask position features and the pixel features by using the multiplication operator to obtain defect position prediction features; perform an average operation on the defect mask features by using the average pooling layer to obtain image query features; process the defect mask features respectively by using the multi-layer perceptron to obtain image category prediction features and defect category prediction features;

[0133] Wherein, the product feature information includes the defect position prediction features, the image query features, the image category prediction features, and the defect category prediction features.

[0134] Optionally, the loss function during training of the image feature extraction model is:

[0135] L = L postion + L class + wL image ;

[0136] Wherein, L postion represents the loss function corresponding to the defect position prediction features, and L postion = w1Lfocal +w2L dice , H represents the height of the product image, W represents the width of the product image, and G i,j represents the value of the point at position (i, j) in the ground truth mask image input during training, and P i,j represents the probability that the point at position (i, j) in the defect location prediction feature obtained during training is the foreground. When G i,j takes the value of 1, a t = a, P t = P i,j , when the current G i,j takes the value of 0, a t = 1 - a, P t = 1 - P i,j , where a is a parameter for balancing the foreground and background pixel classes, and γ is a parameter for balancing easy and difficult samples; G and P are matching pairs obtained by matching M ground truth mask images and N defect location prediction features through the Hungarian algorithm. G represents the number of pixel points in the foreground part of the ground truth mask image, P represents the number of pixel points in the foreground part of the defect location prediction feature, G ∩ P represents the number of overlapping pixel points in the foreground part of the ground truth mask image and the defect location prediction feature, and w1, w2 represent weights;

[0137] L class represents the cross-entropy loss function corresponding to the defect class prediction feature, and L image represents the CE loss function corresponding to the image class prediction feature, and w represents the loss weight corresponding to the image class prediction feature.

[0138] Optionally, the quality inspection module 220 is configured to perform feature fusion and normalization processing on the image query feature and the image class prediction feature to obtain the global product image feature; perform feature fusion and normalization processing on the defect location prediction feature, the defect class prediction feature, and the product information feature corresponding to the product information to obtain the product defect feature; perform quality inspection on the product according to the global product image feature and the product defect feature to obtain the product quality inspection result.

[0139] Optionally, the quality inspection module 220 is configured to use a multimodal model to obtain a defect score, a defect class, and a defect location according to the global product image feature and the product defect feature, and the product quality inspection result includes the defect score, the defect class, and the defect location.

[0140] Optionally, the product image is a foreground image obtained by photographing the product from multiple angles.

[0141] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0142] Please refer to Figure 6 , Figure 6 FIG. is a schematic structural diagram of a quality inspection device for implementing a commodity quality inspection method provided by an embodiment of the present application. The quality inspection device may include: at least one processor 310, such as a CPU, at least one communication interface 320, at least one memory 330, and at least one communication bus 340. Among them, the communication bus 340 is used to implement connection communication between these components. Among them, the communication interface 320 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 330 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. Optionally, the memory 330 may also be at least one storage device located far from the foregoing processor. The memory 330 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 310, the quality inspection device executes the above Figure 2 shown method process.

[0143] It can be understood that Figure 6 the structure shown is only schematic, and the quality inspection device may further include more or fewer components than those Figure 6 shown, or have a different configuration from that Figure 6 shown. Figure 6 Each component shown in can be implemented by hardware, software, or a combination thereof.

[0144] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it executes the method process executed by the quality inspection device in the method embodiment as Figure 2 shown.

[0145] This embodiment discloses a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the foregoing method embodiments. For example, it includes:

[0146] Obtain commodity feature information and commodity information, where the commodity feature information is obtained by extracting features from a commodity image, and the commodity information includes commodity attribute information, or the commodity information includes commodity attribute information and commodity circulation information;

[0147] Quality inspection of the commodity is performed based on the commodity feature information and the commodity information to obtain the commodity quality inspection result.

[0148] In summary, the embodiments of the present application provide a commodity quality inspection method, system, quality inspection device, storage medium, and program product. By extracting commodity feature information from a commodity image and combining it with the commodity information to perform quality inspection on the commodity, the commodity feature information can reflect more detailed information of the commodity, while the commodity information can provide more background information as an auxiliary for commodity quality inspection. Combining the commodity information can provide more comprehensive verification information, reduce misjudgment, and improve the accuracy and efficiency of quality inspection.

[0149] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical, or other form.

[0150] In addition, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] Furthermore, in each embodiment of the present application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0152] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0153] The above description is only for the embodiments of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A commodity quality inspection method, characterized in that: The method comprises: Acquire commodity feature information and commodity information, wherein the commodity feature information is obtained by extracting features from commodity images, and the commodity information includes commodity attribute information, or the commodity information includes commodity attribute information and commodity circulation information; The product is quality inspected according to the product feature information and the product information to obtain a product quality inspection result.

2. The method according to claim 1, characterized in that Get product feature information, including: Extract product feature information from product images using an image feature extraction model; The image feature extraction model includes a mask decoder, an image encoder, a pixel decoder and a feature extractor, and the method of extracting commodity feature information from the commodity image using the image feature extraction model includes: Using the image encoder to extract image features from the obtained commodity image to obtain image features; Obtaining defect mask features using the mask decoder according to the image features and query vectors, wherein the number of query vectors is related to the number of candidate defects; The pixel decoder and the feature extractor are used to obtain commodity feature information according to the image features and the defect mask features.

3. The method according to claim 2, characterized in that The feature extractor includes a multilayer perceptron, an average pooler, and a multiplier. The pixel decoder and the feature extractor are used to obtain product feature information according to the image features and the defect mask features, including: Processing the image features using the pixel decoder to obtain pixel features; Processing the defect mask feature using the multi-layer perceptron to obtain a defect mask position feature; Using the multiplier to multiply the defect mask position feature with the pixel feature to obtain a defect position prediction feature; Using the average pooler to perform an average operation on the defect mask features to obtain image query features; Using the multi-layer perceptron to process the defect mask features respectively, to obtain image category prediction features and defect category prediction features; The commodity feature information includes the defect location prediction feature, the image query feature, the image category prediction feature, and the defect category prediction feature.

4. The method according to claim 3, characterized in that The loss function when training the image feature extraction model is: L=L postion +L class +wL image ; Among them, L postion represents the loss function corresponding to the defect location prediction feature, L postion =w1L focal +w2L dice , H represents the height of the product image, W represents the width of the product image, and G i,j represents the value of the point (i, j) in the true value mask image input during training, P i,j It indicates the probability that the position (i, j) in the defect position prediction feature obtained during training is the foreground. When G i,j When the value is 1, a t =a,P t =P i,j , current G i,j When the value is 0, a t =1-a,P t =1-P i,j , a is the parameter for balancing foreground and background pixel categories, and γ is the parameter for balancing difficult and easy samples; G, P are matching pairs obtained by matching M true value mask images with N defect position prediction features through the Hungarian algorithm, G represents the number of pixels in the foreground part of the true value mask image, P represents the number of pixels in the foreground part of the defect position prediction feature, G∩P represents the number of overlapping pixels in the foreground part of the true value mask image and the defect position prediction feature, and w1, w2 represent weights; L class represents the cross entropy loss function corresponding to the defect category prediction feature, L image represents the CE loss function corresponding to the image category prediction feature, and w represents the loss weight corresponding to the image category prediction feature.

5. The method according to claim 3, characterized in that: The step of performing quality inspection on the commodity according to the commodity feature information and the commodity information to obtain a commodity quality inspection result includes: Performing feature fusion and normalization processing on the image query feature and the image category prediction feature to obtain a global feature of the product image; Performing feature fusion and normalization processing on the defect location prediction feature, the defect category prediction feature, and the product information feature corresponding to the product information to obtain a product defect feature; The product is quality inspected according to the global features of the product image and the product defect features to obtain a product quality inspection result.

6. The method according to claim 5, characterized in that The performing quality inspection on the commodity according to the global features of the commodity image and the features of the commodity defects to obtain the commodity quality inspection result includes: A multimodal model is used to obtain a defect score, a defect category, and a defect location according to the global features of the product image and the product defect features. The product quality inspection result includes the defect score, the defect category, and the defect location.

7. The method according to any one of claims 1 to 6, characterized in that: The commodity image is a foreground image obtained by photographing the commodity from multiple angles.

8. A commodity quality inspection system, characterized in that: The system comprises: A plurality of image collectors are arranged at different positions of the commodity quality inspection area, and are used to collect images of commodities in the commodity quality inspection area at different positions; The quality inspection device is used to extract features from the commodity images collected by the multiple image collectors to obtain commodity feature information, and to perform quality inspection on the commodities based on the commodity feature information and the obtained commodity information to obtain commodity quality inspection results, wherein the commodity information includes commodity attribute information, or the commodity information includes commodity attribute information and commodity circulation information.

9. A quality inspection device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is performed.

11. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images