A product detection method, an electronic device, and a storage medium

Abstract

The present invention discloses a product detection method, an electronic device and a storage medium, which relate to the field of industrial vision technology. First, an image to be detected and a template image are obtained. Then, the spatial transformation information between the image to be detected and the template image is determined. By comparing the global features between the image to be detected and the template image and performing image transformation on the image to be detected according to the spatial transformation information, a target image corresponding to the image to be detected is obtained. Thus, the spatial correlation attributes of the two images have a high degree of consistency. Finally, according to the pixel difference information in the target image and the template image, it is determined whether the image to be detected is abnormal. In the case where the image to be detected is abnormal, an abnormal region image corresponding to the image to be detected is determined and displayed. It performs precise correction on the image to be detected and has the characteristics of high detection accuracy and wide adaptation range.

Description

Technical Field

[0001] The present invention relates to the field of industrial vision technology, and in particular, to a product detection method, an electronic device, and a storage medium. Background Art

[0002] In the process of industrial production, it is often necessary to perform abnormal detection on the surface of the produced products to determine whether there are defects in the product appearance. However, the existing methods for product surface detection mainly use scale-invariant feature transform to describe the local features of the image, perform positioning correction on the image according to the local features of the image, and then use the method of comparing with a standard template to determine product defects. This detection method has high requirements for the stability of the image to be detected and a narrow application range. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a product detection method, an electronic device, and a storage medium that overcome the above problems or at least partially solve the above problems.

[0004] According to a first aspect of the present invention, there is provided a product detection method, the method comprising:

[0005] Obtaining an image to be detected and a template image, where the template image is an image of a product when the surface of the product to be detected is normal;

[0006] Determining the spatial transformation information between the image to be detected and the template image;

[0007] Performing image transformation on the image to be detected according to the spatial transformation information to obtain a target image corresponding to the image to be detected;

[0008] Determining the pixel difference information between the target image and the template image, and determining whether there is an abnormality in the image to be detected according to the pixel difference information;

[0009] In the case where the image to be detected is abnormal, determining the abnormal region image corresponding to the image to be detected and displaying it.

[0010] According to a second aspect of the present invention, there is provided a product detection device, the device comprising:

[0011] An image acquisition module, configured to acquire an image to be detected and a template image, where the template image is an image of a product when the surface of the product to be detected is normal.

[0012] A transformation information determination module, configured to determine the spatial transformation information between the image to be detected and the template image.

[0013] A target image determination module, configured to perform image transformation on the image to be detected according to the spatial transformation information to obtain a target image corresponding to the image to be detected.

[0014] An abnormality determination module, configured to determine pixel difference information between the target image and the template image, and determine whether there is an abnormality in the image to be detected according to the pixel difference information.

[0015] An abnormal area determination module, configured to, when the image to be detected is abnormal, determine an abnormal area image corresponding to the image to be detected and display it.

[0016] According to a third aspect of the present invention, there is provided an electronic device, including:

[0017] One or more processors;

[0018] A memory;

[0019] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute any one of the above product detection methods.

[0020] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program for use in conjunction with an electronic device, the computer program being executable by a processor to complete any one of the above product detection methods.

[0021] In the solution of the present invention, first, an image to be detected and a template image are obtained, and the template image is an image of a product when the surface of the product to be detected is normal. Then, the spatial transformation information between the image to be detected and the template image is determined. By comparing the global features between the image to be detected and the template image and performing image transformation on the image to be detected according to the spatial transformation information, a target image corresponding to the image to be detected is obtained. Thus, the spatial correlation attributes of the target image and the template image are highly consistent. Finally, according to the pixel difference information between the target image and the template image, it is determined whether there is an abnormality in the image to be detected. When the image to be detected is abnormal, an abnormal area image corresponding to the image to be detected is determined and displayed. It can accurately correct the image to be detected and has the characteristics of high detection accuracy and wide adaptation range.

[0022] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the following specifically describes the embodiments of the present invention. Description of the Drawings

[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiments. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Also, throughout the drawings, the same reference numerals are used to represent the same components.

[0024] In the drawings:

[0025] Figure 1 is a flowchart of the steps of a product detection method provided by an embodiment of the present invention;

[0026] Figure 2 is a flowchart of the steps of another product detection method provided by an embodiment of the present invention;

[0027] Figure 3 is a schematic structural diagram of an example of a detection model provided by an embodiment of the present invention;

[0028] Figure 4 is a schematic structural diagram of an example of a feature extraction network provided by an embodiment of the present invention;

[0029] Figure 5 is a schematic structural diagram of an example of a similarity network provided by an embodiment of the present invention;

[0030] Figure 6 is a schematic structural diagram of an example of a filtering network provided by an embodiment of the present invention;

[0031] Figure 7 is a schematic flowchart of the training steps of a detection model provided by an embodiment of the present invention;

[0032] Figure 8 is a block diagram of a product detection device provided by an embodiment of the present invention. Detailed Embodiments

[0033] The exemplary embodiments of the present invention will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0034] In some surface defect detection scenarios of products, a camera device can be installed to capture the surface of a printed product. Thus, a to-be-detected image and a template image are obtained by the camera device, and it is determined whether there is an abnormality on the surface of the corresponding product based on each method embodiment of the following product detection method. For example, the following method embodiments can be applied to the detection of product abnormalities with high-precision requirements, such as the surface defect detection of printed products. Printing defects are mainly product appearance defects. During the printing process, including ink printing, laser printing, etc., defects such as incomplete printing characters, ink dots, missing printing, dirty dots, and pinholes appear on the appearance of the printed product.

[0035] Referring to Figure 1 , a step flowchart of a product detection method provided by an embodiment of the present invention is shown. The method may include:

[0036] S101. Obtain a to-be-detected image and a template image.

[0037] In an embodiment of the present invention, the template image is an image of a product when the surface of the to-be-detected product is normal. Before mass production of the product, the template image can be captured by a camera device during the production of samples and pre-stored. During the production process of the product, the surface image of the product is captured in real time as the to-be-detected image. And the to-be-detected image and the template image are used as a pair of images to perform the next processing.

[0038] S102. Determine the spatial transformation information between the to-be-detected image and the template image.

[0039] In one embodiment, determining the spatial transformation information between the to-be-detected image and the template image includes: inputting the to-be-detected image and the template image into a detection model for image recognition, and determining the spatial transformation information between the to-be-detected image and the template image. Wherein, the spatial transformation information is used to characterize the spatial changes of the to-be-detected image compared to the template image. For example, from the spatial transformation information, it can be determined whether there are changes in spatial-related attributes such as translation, scaling, rotation, and illumination changes of the to-be-detected image compared to the template image.

[0040] S103. According to the spatial transformation information, perform image transformation on the to-be-detected image to obtain the target image corresponding to the to-be-detected image.

[0041] In an embodiment of the present invention, according to the spatial transformation information, image transformation can be performed on the to-be-detected image, so as to correct the to-be-detected image through a deep learning detection model and transform it into a target image with the same spatial-related attributes as the template image.

[0042] S104. Determine the pixel difference information between the target image and the template image, and determine whether there is an abnormality in the image to be detected based on the pixel difference information.

[0043] S105. In the case where the image to be detected is abnormal, determine the abnormal region image corresponding to the image to be detected and display it.

[0044] In the embodiments of the present invention, the pixel difference information refers to the combination of the RGB differences of the same pixel point in two images. Since the spatial correlation attributes (such as illumination change, shooting angle, scaling ratio, etc.) of the transformed target image and the template image are highly unified, and since the sizes of the target image and the template image are the same, it is possible to determine whether there is an abnormality in the image to be detected through the pixel difference information. After determining the target image, calculate the pixel value differences of the corresponding pixel points in the target image and the template image respectively, and use the pixel value differences of all corresponding pixel points in the two images as the pixel difference information. And determine whether there is an abnormality in the image to be detected by whether the pixel difference information meets the preset difference condition.

[0045] In one example, the preset difference condition may be that the pixel value difference is greater than or equal to the difference threshold. If the RGB difference of the same pixel point in two images is equal to or greater than the difference threshold, it means that the pixel difference information meets the preset difference condition, and it is determined that there is an abnormality at the corresponding position of the image to be detected. That is, as long as any pixel value difference in the pixel difference information meets the preset difference condition, it is determined that there is an abnormality in the image to be detected. If the RGB difference of the same pixel point in two images is less than the difference threshold, it means that the pixel difference information does not meet the preset difference condition, and it is determined that the corresponding position of the image to be detected is normal. Only when all pixel value differences in the pixel difference information do not meet the preset difference condition, it is determined that the image to be detected is normal.

[0046] Since the target image is determined by image transformation of the image to be detected, therefore, if the image to be detected is abnormal, the abnormal pixel points in the image to be detected are inconsistent with the abnormal pixel points in the target image. It is necessary to perform an inverse image transformation on the abnormal region in the target image, so as to obtain the abnormal region image corresponding to the image to be detected and display it. The above detection method can accurately correct the image to be detected, and has the characteristics of high detection accuracy and wide adaptation range.

[0047] Refer to Figure 2 and Figure 3 , which shows the step flowchart of another product detection method provided by the embodiments of the present invention. The detection model includes a feature extraction network, a similarity network, and a filtering network. The method may include:

[0048] S201. Obtain the image to be detected and the template image, where the template image is the product image when the surface of the product to be detected is normal.

[0049] In the embodiment of the present invention, the template image is the product image when the surface of the product to be detected is normal. Before mass production of the product, the template image can be captured by a camera device during the process of making samples and pre-stored. During the product production process, the surface image of the product is captured in real time as the image to be detected. And the image to be detected and the template image are used as a pair of images to perform the next processing.

[0050] S202. Input the image to be detected and the template image into the feature extraction network respectively for feature extraction, to obtain the detection feature information corresponding to the image to be detected and the template feature information corresponding to the template image.

[0051] In the embodiment of the present invention, the image to be detected and the template image are input into the feature extraction network respectively for feature extraction. In order to extract all the image features of the image to be detected and the template image and facilitate subsequent determination of similarity information, in one example, the image to be detected and the template image can be input into the same feature extraction network in chronological order. In another example, two feature extraction networks with the same structure can be set up, so that after obtaining the pair of images, the two images in the pair of images can be input into the two feature extraction networks in parallel. After the feature extraction network extracts the features of the image to be detected, the corresponding detection feature information is obtained, and after the feature extraction network extracts the features of the template image, the corresponding template feature information is obtained.

[0052] In one example, referring to Figure 4As shown, the feature extraction network may include a first convolutional module, a second convolutional module, a third convolutional module, and a fourth convolutional module arranged in sequence. For example, the first convolutional module may perform the first feature extraction on the image using a 7*7 convolutional kernel. The extracted image features are input into the second convolutional module for the second feature extraction. Among them, the second convolutional module, the third convolutional module, and the fourth convolutional module may each include a number of block units, and those skilled in the art can select the number of the above three convolutional modules according to the actual situation to perform different numbers of convolutional operations. For example, each block unit consists of a residual structure. Thus, first, a convolutional subunit with a 1x1 convolutional kernel is used to reduce the dimension of the channels, then a convolutional subunit with a 3x3 convolutional kernel is used for feature extraction, and finally, a convolutional subunit with a 1x1 convolutional kernel is used for feature recovery. And the output of the fourth convolutional module is used as the output of the feature extraction network, that is, the detection feature information corresponding to the image to be detected or the template feature information corresponding to the template object. Among them, the feature information output by the feature extraction network is N*C*H*W, where N is the number of images in one run, C is the number of channels in one image (i.e., the number of image features), and H*W refers to the size of each image feature. Therefore, the detection feature information and the template feature information can also be regarded as a combination of several C feature matrices.

[0053] S203. Input the detection feature information and the template feature information into the similarity network for similarity calculation to determine the similarity information between the detection feature information and the template feature information.

[0054] In the embodiment of the present invention, referring to Figure 5 As shown, the similarity network may perform the following information calculation steps: Multiply the several feature matrices corresponding to the detection feature information and the several feature matrices corresponding to the template feature information. Among them, during the matrix multiplication, the several feature matrices corresponding to the above two feature information can be preferentially subjected to matrix conversion and transformation, so that the several feature matrices corresponding to the detection feature information are converted into a new feature matrix, and the several feature matrices corresponding to the template feature information are converted into a new feature matrix, and the two new feature matrices are subjected to matrix transformation to meet the conditions for matrix multiplication.

[0055] After obtaining the relevant feature matrix after matrix multiplication, the relevant feature matrix is processed by the ReLU activation function and the L2 norm in sequence. The similarity information between the detected feature information and the template feature information is determined. The similarity information is used to determine the matching scores of each local feature between the image to be detected and the template image. Among them, the ReLU activation function can increase the non-linear ability of the neural network model, overcome the problem of gradient disappearance, and make the model training faster. The L2 norm is used to normalize the eigenvalues.

[0056] S204. Input the similarity information into the filtering network for matching processing to obtain spatial transformation information.

[0057] S205. According to the spatial transformation information, perform image transformation on the image to be detected to obtain the target image corresponding to the image to be detected.

[0058] In the embodiment of the present invention, in order to improve the accuracy of determining the invariant feature information, the similarity information can also be processed by the filtering network for matching. Among them, as shown in Figure 6 , the filtering network can be composed of multiple multi-dimensional convolution modules, and a ReLU activation function is set after each convolution module. Input the similarity information into the filtering network, and the filtering network can filter out the noise in the similarity information, and calculate the matching scores between each local feature of the image to be detected and the template image according to the similarity information after the filtering operation. Thus, the local features whose matching scores meet the preset similarity conditions are used as invariant feature information. For example, the preset similarity condition can be the local feature with the highest matching score, etc. And according to the invariant feature information, spatial transformation information is generated and output. Among them, the invariant feature information refers to the consistent RGB features of the corresponding pixel points between the image to be detected and the template image.

[0059] In one example, the detection model may further include an anomaly detection locator (PDetector shown in Figure 6 ). The spatial transformation information can be input into the anomaly detection locator, and the anomaly detection locator performs image transformation on the image to be detected according to the spatial transformation information, so that the image to be detected can be depth-corrected according to the template image, and the image to be detected is transformed into a target image with the same spatial correlation attributes as the template image.

[0060] Since the feature extraction network performs intensive and in-depth extraction of global features on the image to be detected and the template image, in the case where the image to be detected has serious offsets, rotations, scalings, illumination changes, etc. compared with the template image, it will not affect the determination of invariant features. Thus, the applicable range of the product detection method can be expanded, and at the same time, the abnormal detection result of the method can be promoted, and micro-printing defects such as printing defects, ink dots, missing prints, dirty dots, and pinholes can be effectively detected.

[0061] S206. Determine the pixel difference information between the target image and the template image, and determine whether the image to be detected is abnormal according to the pixel difference information.

[0062] S207. In the case where the image to be detected is abnormal, determine the target region image corresponding to the target image.

[0063] S208. Perform an inverse image transformation on the target region image to determine the abnormal region image corresponding to the image to be detected, and display it.

[0064] In the embodiment of the present invention, the pixel difference information refers to the combination of the RGB differences of the same pixel point in two images. Since the spatial correlation attributes (such as illumination changes, shooting angles, scaling ratios, etc.) of the transformed target image and the template image are highly unified, and since the sizes of the target image and the template image are the same, it is possible to determine whether the image to be detected is abnormal through the pixel difference information. After the target image is determined, calculate the pixel value differences of the corresponding pixel points among all pixel points in the target image and the template image respectively, and use the pixel value differences of all corresponding pixel points in the two images as the pixel difference information. And determine whether the image to be detected is abnormal by whether the pixel difference information meets the preset difference condition.

[0065] In one example, the preset difference condition may be that the pixel value difference is greater than or equal to the difference threshold. If the RGB difference of the same pixel point in two images is equal to or greater than the difference threshold, it means that the pixel difference information meets the preset difference condition, and it is determined that there is an abnormality at the corresponding position of the image to be detected. That is, as long as any pixel value difference in the pixel difference information meets the preset difference condition, it is determined that the image to be detected is abnormal. If the RGB difference of the same pixel point in two images is less than the difference threshold, it means that the pixel difference information does not meet the preset difference condition, and it is determined that the corresponding position of the image to be detected is normal. Only when all pixel value differences in the pixel difference information do not meet the preset difference condition, it is determined that the image to be detected is normal.

[0066] Since the target image is determined by image transformation of the image to be detected, if the image to be detected is abnormal, the abnormal pixel points in the image to be detected are inconsistent with the abnormal pixel points in the target image. According to the position of the abnormal area in the target image, a target area image highlighting the abnormal area can be generated. For example, the target area image can be a binary image, the RGB value of its corresponding normal area can be 0, and the RGB value of its corresponding abnormal area can be 255. Thus, the abnormal area can be clearly distinguished. Moreover, the abnormal detection locator can perform inverse image transformation on the target area image based on the spatial transformation information, so as to obtain the abnormal area image corresponding to the image to be detected and display it. Therefore, the abnormal area in the image to be detected can be quickly located through the abnormal area image. The above detection method can accurately correct the image to be detected, and has the characteristics of high detection accuracy and wide adaptation range.

[0067] In an optional embodiment of the invention, the method further includes a training step of the detection model, and the detection model includes a feature extraction network, a similarity network, and a filtering network. Refer to Figure 7 As shown, the training step may include:

[0068] S701. Obtain an image pair with a preset image label, where the image pair includes a first sample image and a second sample image, and the image label is used to determine whether the first sample image and the second sample image are surface images of the same category of products.

[0069] In the embodiment of the present invention, a sample set is established in the form of an image pair. Each image pair includes a first sample image and a second sample image. In the process of establishing the sample set, it is not necessary to label each sample image, and only the image label of the image pair needs to be established. That is, only when establishing a group of image pairs, it is only necessary to mark whether the two sample images in the image pair are in a matching mode or a non-matching mode through the image label, which reduces the complexity of data collection and annotation. Among them, the matching mode means that the first sample image and the second sample image are surface images of the same category of products, and the second sample image can be an image of the surface of the corresponding category of products when it is normal. For printed products, their product categories can be divided according to the printed patterns.

[0070] S702. Input the first sample image and the second sample image into the feature extraction network respectively for feature extraction, and obtain first feature information corresponding to the first sample image and second feature information corresponding to the second sample image.

[0071] S703. Input the first feature information and the second feature information into the similarity network for similarity calculation, and determine the similarity information between the first feature information and the second feature information.

[0072] S704. Input the similarity information and the image label into the filtering network for matching processing to obtain spatial transformation information, where the matching processing includes at least the following steps: determine a matching score according to the similarity information, and determine the spatial transformation information according to the matching score.

[0073] S705. Adjust the model parameters of the detection model according to the image label and the matching score. When the matching score meets the preset parameter adjustment condition, stop adjusting the model parameters and obtain the trained detection model.

[0074] In the embodiment of the present invention, the first sample image and the second sample image are input into the feature extraction network for feature extraction, and the first feature information corresponding to the first sample image and the second feature information corresponding to the second sample image obtained are respectively input into the similarity network to determine the similarity information between the two feature information. The similarity information refers to the similar feature information of each local feature between the image to be detected and the template image.

[0075] After determining the similarity information, the similarity information and the image label can be input into the filtering network, and the filtering network performs matching processing on the similarity information. For example, a matching score between the image to be detected and the template image is determined through the similarity information. The matching score is used to represent the similarity degree of each local feature between the image to be detected and the template image. For example, the higher the matching score, the higher the similarity degree between the image to be detected and the template image, and the higher the image consistency. For another example, the lower the matching score, the lower the similarity degree between the detected image and the template image, and the lower the image consistency. Therefore, during the model training process, the model parameters can be adjusted according to the image label and the matching score.

[0076] In one example, the preset parameter adjustment conditions include: the matching scores corresponding to the similarity information between the surface images of products of the same category satisfy a first preset condition. Among them, the first preset condition can be that when the image label is in the matching mode, the corresponding matching score is determined, and the model parameters (such as convolution kernels, etc.) in the feature extraction network, similarity network, and filtering network are adjusted, so that the matching scores corresponding to two images in the matching mode are maximized. And, the matching scores corresponding to the similarity information between the surface images of products of different categories satisfy a second preset condition. The second preset condition can be the minimization of the matching score. That is, when the image label is in the non-matching mode, the corresponding matching score is determined, and the model parameters in the feature extraction network, similarity network, and filtering network are adjusted, so that the matching score in the matching mode is minimized. Among them, the maximization of the matching score will result in an ideal effect of good recognition and matching in the image pairs in the matching mode. Similarly, the minimization of the matching score will gradually weaken the matching of the image pairs in the non-matching mode. And, the parameters can be adjusted by the gradient descent method through the loss function.

[0077] Therefore, in the anomaly detection process, the image to be detected and the template image are of the same product category, which is in the matching mode. After being processed by the feature extraction network, similarity network, and filtering network in sequence, the corresponding spatial transformation information is obtained. Among them, the spatial transformation information is determined based on the maximized matching score. The target image obtained by performing image transformation on the image to be detected through the spatial transformation information has a high degree of consistency with the spatial related attributes of the template image.

[0078] In summary, a product detection method provided by an embodiment of the present invention first obtains an image to be detected and a template image, where the template image is an image of the product when the surface of the product to be detected is normal. Then, the spatial transformation information between the image to be detected and the template image is determined. By comparing the global features between the image to be detected and the template image and performing image transformation on the image to be detected according to the spatial transformation information, the target image corresponding to the image to be detected is obtained. Thus, the spatial related attributes of the target image and the template image have a high degree of consistency. Finally, based on the pixel difference information in the target image and the template image, it is determined whether the image to be detected is abnormal. In the case where the image to be detected is abnormal, the abnormal region image corresponding to the image to be detected is determined and displayed. It can accurately correct the image to be detected and has the characteristics of high detection accuracy and wide adaptation range.

[0079] It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of the present application are not limited by the described action sequences, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present application.

[0080] Referring to Figure 8 , a product detection device provided by an embodiment of the present invention is shown. The device may include:

[0081] An image acquisition module 801, configured to acquire an image to be detected and a template image, where the template image is an image of the product when the surface of the product to be detected is normal.

[0082] A transformation information determination module 802, configured to determine the spatial transformation information between the image to be detected and the template image.

[0083] A target image determination module 803, configured to perform image transformation on the image to be detected according to the spatial transformation information to obtain a target image corresponding to the image to be detected.

[0084] An abnormality determination module 804, configured to determine the pixel difference information between the target image and the template image, and determine whether there is an abnormality in the image to be detected according to the pixel difference information.

[0085] An abnormal area determination module 805, configured to determine an abnormal area image corresponding to the image to be detected and display it when the image to be detected is abnormal.

[0086] An optional embodiment of the invention, the transformation information determination module 802 may include:

[0087] A feature extraction sub-module, configured to input the image to be detected and the template image into the feature extraction network respectively for feature extraction, to obtain the detection feature information corresponding to the image to be detected and the template feature information corresponding to the template image.

[0088] A feature recognition sub-module, configured to input the detection feature information and the template feature information into the image recognition sub-model for feature recognition, and determine the spatial transformation information between the detection feature information and the template feature information.

[0089] An optional embodiment of the invention, the image recognition sub-module includes a similarity network and a filtering network, and the feature recognition sub-module may further include:

[0090] A similarity calculation unit, configured to input the detected feature information and the template feature information into the similarity network for similarity calculation, and determine the similarity information between the detected feature information and the template feature information.

[0091] A transformation information determination unit, configured to input the similarity information into the filtering network for matching processing to obtain spatial transformation information.

[0092] An optional invention embodiment, the anomaly determination module 804 may include:

[0093] A difference calculation sub-module, configured to calculate the pixel value differences of corresponding pixel points in the target image and the template image respectively.

[0094] A difference information determination sub-module, configured to use a plurality of pixel value differences as pixel difference information.

[0095] An optional invention embodiment, the anomaly determination module 804 may further include:

[0096] An anomaly determination sub-module, configured to determine that the image to be detected has an anomaly if any pixel value difference in the pixel difference information satisfies a preset difference condition.

[0097] A normal determination sub-module, configured to determine that the image to be detected is normal if a plurality of pixel value differences in the pixel difference information do not satisfy a preset difference condition.

[0098] An optional invention embodiment, the anomaly region determination module 805 may further include:

[0099] A target region determination sub-module, configured to determine a target region image corresponding to the target image.

[0100] An anomaly region determination sub-module, configured to perform an inverse image transformation on the target region image according to the spatial transformation information to obtain an anomaly region image corresponding to the image to be detected.

[0101] An optional invention embodiment, the device further includes a training module for training a detection model, the detection model includes a feature extraction network, a similarity network, and a filtering network, and the training module may include:

[0102] An image pair acquisition sub-module, configured to acquire an image pair with a preset image label, the image pair includes a first sample image and a second sample image, and the image label is used to determine whether the first sample image and the second sample image are surface images of products of the same category.

[0103] The sample feature extraction sub-module is used to input the first sample image and the second sample image into the feature extraction network respectively for feature extraction, so as to obtain the first feature information corresponding to the first sample image and the second feature information corresponding to the second sample image.

[0104] The similarity calculation sub-module is used to input the first feature information and the second feature information into the similarity network for similarity calculation, and determine the similarity information between the first feature information and the second feature information.

[0105] The information update sub-module is used to input the similarity information and the image label into the filtering network for matching processing to obtain the spatial transformation information.

[0106] The parameter adjustment sub-module is used to adjust the model parameters of the detection model according to the image label and the spatial transformation information. When the matching score meets the preset parameter adjustment condition, stop adjusting the model parameters and obtain the trained detection model.

[0107] An optional embodiment of the invention, the preset parameter adjustment condition includes: the matching score corresponding to the similarity information between the surface images of products of the same category meets the first preset condition, and the matching score corresponding to the similarity information between the surface images of products of different categories meets the second preset condition.

[0108] In summary, a product detection device provided by an embodiment of the present invention first obtains a to-be-detected image and a template image, and the template image is a product image when the surface of the to-be-detected product is normal. Then, the spatial transformation information between the to-be-detected image and the template image is determined. It compares the global features between the to-be-detected image and the template image, and performs image transformation on the to-be-detected image according to the spatial transformation information to obtain the target image corresponding to the to-be-detected image. Thus, the spatial correlation attributes of the target image and the template image are highly consistent. Finally, according to the pixel difference information in the target image and the template image, it is determined whether the to-be-detected image is abnormal. In the case where the to-be-detected image is abnormal, the abnormal region image corresponding to the to-be-detected image is determined and displayed. It can accurately correct the to-be-detected image, and has the characteristics of high detection accuracy and wide adaptation range.

[0109] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other.

[0110] It is easy for those skilled in the art to think that any combination application of the above-mentioned embodiments is feasible. Therefore, any combination of the above-mentioned embodiments is an implementation solution of the present invention. However, due to space limitations, this specification will not elaborate on them one by one here.

[0111] In the specification provided herein, numerous specific details are set forth. However, it will be understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0112] Similarly, it should be understood that in order to streamline the present invention and assist in understanding one or more of the various inventive aspects, in the foregoing description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected by the claims, the inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0113] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and disposed in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0114] An electronic device, comprising:

[0115] One or more processors;

[0116] A memory;

[0117] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the methods described in the foregoing embodiments.

[0118] A computer-readable storage medium storing a computer program for use in conjunction with an electronic device, the computer program being executable by a processor to perform the methods described in the foregoing embodiments.

[0119] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention can take the form of all-hardware embodiments, all-software embodiments, or embodiments combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0121] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0123] Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

[0124] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or terminal device comprising said element.

[0125] The above has introduced in detail a product detection method and a product detection device provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A product detection method, characterized in that, The method includes: Obtaining a to-be-detected image and a template image, where the template image is an image of a product when the surface of the to-be-detected product is normal; Determining the spatial transformation information between the to-be-detected image and the template image; Performing image transformation on the to-be-detected image according to the spatial transformation information to obtain a target image corresponding to the to-be-detected image; Determining the pixel difference information between the target image and the template image, and determining whether the to-be-detected image is abnormal according to the pixel difference information; In the case where the to-be-detected image is abnormal, determining the abnormal region image corresponding to the to-be-detected image and displaying it; The determining the spatial transformation information between the to-be-detected image and the template image includes: inputting the to-be-detected image and the template image into a detection model for image recognition to determine the spatial transformation information between the to-be-detected image and the template image; The method further includes a training step of the detection model. The detection model includes a feature extraction network and an image recognition sub-model. The image recognition sub-model includes a similarity network and a filtering network. The training step includes: Obtaining an image pair with a preset image label, where the image pair includes a first sample image and a second sample image, and the image label is used to determine whether the first sample image and the second sample image are surface images of products of the same category; Respectively inputting the first sample image and the second sample image into the feature extraction network for feature extraction to obtain first feature information corresponding to the first sample image and second feature information corresponding to the second sample image; Inputting the first feature information and the second feature information into the similarity network for similarity calculation to determine the similarity information between the first feature information and the second feature information; Inputting the similarity information and the image label into the filtering network for matching processing to obtain spatial transformation information, where the matching processing at least includes the following steps: determining a matching score according to the similarity information, and determining the spatial transformation information according to the matching score; Adjusting the model parameters of the detection model according to the image label and the matching score, and stopping adjusting the model parameters and obtaining a trained detection model when the matching score meets the preset parameter adjustment condition.

2. The product detection method according to claim 1, wherein The determining the spatial transformation information between the to-be-detected image and the template image includes: Respectively inputting the to-be-detected image and the template image into the feature extraction network for feature extraction to obtain detection feature information corresponding to the to-be-detected image and template feature information corresponding to the template image; Inputting the detection feature information and the template feature information into the image recognition sub-model for feature recognition to determine the spatial transformation information between the detection feature information and the template feature information.

3. The product detection method according to claim 2, characterized in that: The image recognition sub-model includes a similarity network and a filtering network. The inputting the detection feature information and the template feature information into the image recognition sub-model for feature recognition to determine the spatial transformation information between the detection feature information and the template feature information includes: Input the detected feature information and the template feature information into the similarity network for similarity calculation to determine the similarity information between the detected feature information and the template feature information; Input the similarity information into the filtering network for matching processing to obtain spatial transformation information.

4. The product detection method according to any one of claims 1-3, characterized in that, The determination of the pixel difference information in the target image and the template image includes: Calculate the pixel value differences of the corresponding pixel points among all the pixel points in the target image and the template image respectively to obtain pixel difference information.

5. The product detection method according to claim 4, characterized in that The determination of whether the image to be detected is abnormal based on the pixel difference information includes: If any pixel value difference in the pixel difference information meets the preset difference condition, determine that the image to be detected is abnormal; If several pixel value differences in the pixel difference information do not meet the preset difference condition, determine that the image to be detected is normal.

6. The product detection method according to claim 5, characterized in that The determination of the abnormal region image corresponding to the image to be detected includes: Determine the target region image corresponding to the target image; According to the spatial transformation information, perform an inverse image transformation on the target region image to obtain the abnormal region image corresponding to the image to be detected.

7. The product detection method according to claim 1, wherein The preset parameter adjustment condition includes: the matching scores corresponding to the similarity information between the surface images of products of the same category meet the first preset condition, and the matching scores corresponding to the similarity information between the surface images of products of different categories meet the second preset condition.

8. A product detection device, characterized in that, The device includes: An image acquisition module, configured to acquire an image to be detected and a template image, where the template image is an image of a product when the surface of the product to be detected is normal; A transformation information determination module, configured to determine the spatial transformation information between the image to be detected and the template image; the determination of the spatial transformation information between the image to be detected and the template image includes: inputting the image to be detected and the template image into a detection model for image recognition to determine the spatial transformation information between the image to be detected and the template image; It further includes a training step of the detection model. The detection model includes a feature extraction network and an image recognition sub-model. The image recognition sub-model includes a similarity network and a filtering network. The training step includes: Obtain an image pair with a preset image label. The image pair includes a first sample image and a second sample image. The image label is used to determine whether the first sample image and the second sample image are surface images of products of the same category; Input the first sample image and the second sample image into the feature extraction network for feature extraction respectively to obtain first feature information corresponding to the first sample image and second feature information corresponding to the second sample image; Input the first feature information and the second feature information into the similarity network for similarity calculation to determine the similarity information between the first feature information and the second feature information; Input the similarity information and the image label into the filtering network for matching processing to obtain spatial transformation information, where the matching processing at least includes the following steps: determine a matching score according to the similarity information, and determine spatial transformation information according to the matching score; Adjust the model parameters of the detection model according to the image tags and matching scores. When the matching scores meet the preset parameter adjustment conditions, stop adjusting the model parameters and obtain the trained detection model; A target image determination module, configured to perform image transformation on the image to be detected according to the spatial transformation information to obtain a target image corresponding to the image to be detected; An abnormality determination module, configured to determine pixel difference information between the target image and the template image, and determine whether there is an abnormality in the image to be detected according to the pixel difference information; An abnormal area determination module, configured to determine an abnormal area image corresponding to the image to be detected and display it when the image to be detected is abnormal.

9. An electronic device, comprising: One or more processors; A memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the product detection method according to any one of claims 1-7.

10. A computer-readable storage medium, storing a computer program for use in combination with an electronic device, the computer program being executable by a processor to complete the product detection method according to any one of claims 1-7.

Images