Appearance inspection method and device, electronic equipment and storage medium

Abstract

The invention provides an appearance inspection method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing, the method comprises the steps that a target image is acquired, and the target image comprises a target object to be subjected to appearance inspection; based on the color of each pixel point in the target image, clustering the pixel points in the target image to obtain a clustering mask corresponding to the target object; performing image segmentation on the target image to obtain a reference object mask and an inspection object mask, the reference object mask being a mask corresponding to a main body of the target object, and the inspection object mask being a mask corresponding to a component to be subjected to appearance inspection in the target object; and determining an appearance inspection result of the component to be inspected in the target object based on the clustering mask, the reference object mask and the inspection object mask. According to the scheme, the background color reflected by the surface of the product is filtered, the influence of the reflected background color on the surface color of the product is reduced, and then the accuracy of the inspection result of the product appearance assembly is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an appearance inspection method, device, electronic device and storage medium. Background Art

[0002] Appearance inspection is an important part of product launch and plays a significant role in product production. For example, as an important part of vehicle quality inspection, the color inspection of vehicle appearance components is an important step to ensure the consistency of vehicle appearance color.

[0003] In the related art, the image to be inspected is divided into a color inspection object and a reference object, and the similarity between the color inspection object and the reference object in the image is calculated by a color histogram, so as to determine whether the colors of the color inspection object and the reference object are consistent according to a preset threshold.

[0004] In actual applications, affected by the product surface coating and the background environment in which the product is located, the product surface is prone to reflect the image of the background environment. When the background reflection area is large, the similarity of the same color will be low, resulting in inaccurate appearance inspection results. Summary of the invention

[0005] The purpose of the present application is to provide an appearance inspection method, device, electronic device and storage medium, aiming to solve the problem of inaccurate traditional product appearance inspection.

[0006] A first aspect of an embodiment of the present application provides a method for appearance inspection, the method comprising:

[0007] Acquire a target image, wherein the target image includes a target object to be visually inspected;

[0008] Clustering the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object;

[0009] Performing image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to a main body of the target object, and the inspection object mask is a mask corresponding to a component to be visually inspected in the target object;

[0010] Based on the clustering mask, the reference object mask and the inspection object mask, a visual inspection result of a component to be inspected in the target object is determined.

[0011] In some embodiments, determining the appearance inspection result of the component to be inspected in the target object based on the cluster mask, the reference object mask and the inspection object mask includes:

[0012] If there is no intersection mask region between the cluster mask and the inspection object mask, determining that the appearance inspection result is color inspection inconsistency;

[0013] If there is an intersection mask region between the cluster mask and the inspection object mask, updating the inspection object mask and the reference object mask based on the cluster mask;

[0014] The appearance inspection result is determined based on the updated inspection object mask and the updated reference object mask.

[0015] In some embodiments, updating the inspection object mask and the reference object mask based on the clustering mask comprises:

[0016] determining a first intersection mask between the cluster mask and the inspection object mask;

[0017] Determining the first intersection mask as a new inspection object mask;

[0018] determining a second intersection mask between the cluster mask and the reference object mask;

[0019] The second intersection mask is determined as a new reference object mask.

[0020] In some embodiments, determining the appearance inspection result based on the updated inspection object mask and the updated reference object mask includes:

[0021] determining a first image region of the inspection object from the target image based on the updated inspection object mask;

[0022] determining, from the target image, a second image region of the reference object based on the updated reference object mask;

[0023] constructing a first color histogram of the first image area, and constructing a second color histogram of the second image area;

[0024] The appearance inspection result is determined based on a similarity between the first color histogram and the second color histogram.

[0025] In some embodiments, constructing a first color histogram of the first image region, and constructing a second color histogram of the second image region, comprises:

[0026] Extracting image channels related to color features in the target image;

[0027] A first color histogram of the first image region is constructed based on the image channels associated with the color features, and a second color histogram of the second image region is constructed based on the image channels associated with the color features.

[0028] In some embodiments, clustering the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object includes:

[0029] Based on the color of each pixel in the target image, clustering the pixels in the target image to obtain a plurality of classification results, wherein each of the classification results includes a plurality of pixels of a color corresponding to the classification result;

[0030] From the multiple classification results, a mask area corresponding to the classification result with the most pixel points is determined to obtain the cluster mask.

[0031] In some embodiments, clustering the pixels in the target image based on the color of each pixel in the target image to obtain multiple classification results includes:

[0032] Enhancing the saturation of the target image to obtain a high-saturation image;

[0033] Extracting color features of the high-saturation image;

[0034] Determining the number of cluster centers of the target image;

[0035] Based on the number of cluster centers, uniformly sampling the color features of the high-saturation image to obtain a plurality of cluster centers;

[0036] The high-saturation image is iterated by clustering based on the multiple cluster centers to obtain the multiple classification results.

[0037] A second aspect of an embodiment of the present application provides a visual inspection device, the device comprising:

[0038] An acquisition unit, used for acquiring a target image, wherein the target image includes a target object to be visually inspected;

[0039] A clustering unit, configured to cluster the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object;

[0040] An image processing unit, configured to perform image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to a main body of the target object, and the inspection object mask is a mask corresponding to a component of the target object to be visually inspected;

[0041] A determining unit is used to determine a visual inspection result of a component to be inspected in the target object based on the clustering mask, the reference object mask and the inspection object mask.

[0042] In some embodiments, the determination unit is used to determine that the appearance inspection result is color inspection inconsistency if there is no intersection mask area between the clustering mask and the inspection object mask; if there is an intersection mask area between the clustering mask and the inspection object mask, update the inspection object mask and the reference object mask based on the clustering mask; and determine the appearance inspection result based on the updated inspection object mask and the updated reference object mask.

[0043] In some embodiments, the determination unit is used to determine a first intersection mask between the clustering mask and the inspection object mask; determine the first intersection mask as a new inspection object mask; determine a second intersection mask between the clustering mask and the reference object mask; and determine the second intersection mask as a new reference object mask.

[0044] In some embodiments, the determination unit is used to determine a first image area of ​​the inspection object from the target image based on the updated inspection object mask; determine a second image area of ​​the reference object from the target image based on the updated reference object mask; construct a first color histogram of the first image area, and construct a second color histogram of the second image area; and determine the appearance inspection result based on the similarity between the first color histogram and the second color histogram.

[0045] In some embodiments, the determination unit is used to extract image channels related to color features in the target image; construct a first color histogram of the first image area based on the image channels related to color features, and construct a second color histogram of the second image area based on the image channels related to color features.

[0046] In some embodiments, the clustering unit is used to cluster the pixels in the target image based on the color of each pixel in the target image to obtain multiple classification results, wherein each classification result includes multiple pixels of the color corresponding to the classification result; from the multiple classification results, determine the mask area corresponding to the target classification result with the most pixels to obtain the clustering mask.

[0047] In some embodiments, the clustering unit is used to enhance the saturation of the target image to obtain a high-saturation image; extract color features of the high-saturation image; determine the number of cluster centers of the target image; based on the number of cluster centers, uniformly sample the color features of the high-saturation image to obtain multiple cluster centers; and iteratively cluster the high-saturation image based on the multiple cluster centers to obtain the multiple classification results.

[0048] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the appearance inspection method described above when executing the computer program.

[0049] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the appearance inspection method described above is implemented.

[0050] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0051] In an embodiment of the present application, after acquiring a target image containing a target object, the pixels in the target image are clustered based on the color of each pixel in the target image to obtain a clustering mask for representing the image area where the target object is located, and the target image is segmented to obtain a reference object mask corresponding to the main body of the target object in the target image and an inspection object mask corresponding to the component to be inspected for appearance, thereby determining the appearance inspection result of the component to be inspected in the target object by comparing the clustering mask, the object mask and the inspection object mask. In this way, the color of the main body of the product is extracted by color clustering, thereby filtering the background color reflected on the product surface, reducing the influence of the reflected background color on the product surface color, and thus improving the accuracy of the inspection result of the product appearance component. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic flow chart of a visual inspection method provided by an exemplary embodiment is shown;

[0053] Figure 2 A schematic flow chart of a visual inspection method provided by an exemplary embodiment is shown;

[0054] Figure 3 A schematic diagram showing a system involved in a visual inspection method provided by an exemplary embodiment;

[0055] Figure 4 A schematic flow chart of a visual inspection method provided by an exemplary embodiment is shown;

[0056] Figure 5 A schematic flow chart of a visual inspection method provided by an exemplary embodiment is shown;

[0057] Figure 6 A schematic flow chart of a visual inspection method provided by an exemplary embodiment is shown;

[0058] Figure 7 A schematic flow chart of a visual inspection method provided by an exemplary embodiment is shown;

[0059] Figure 8 A schematic structural diagram of a visual inspection device provided by an exemplary embodiment is shown;

[0060] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0063] Appearance inspection is an important quality inspection link before a product is put on the market and plays a significant role in product production. For example, as an important part of vehicle quality inspection, the color inspection of vehicle appearance components is an important step to ensure the consistency of vehicle appearance color.

[0064] In some embodiments, the product is inspected for appearance on the quality inspection line. During the inspection process, a target image containing the product to be inspected is captured by a camera, and image processing is performed on the target image to determine whether the appearance color of the product is qualified. For example, the target image is divided into a color inspection object and a reference object, and then the color histograms of the color inspection object and the reference object are determined respectively. Based on the similarity between the color histograms of the color inspection object and the reference object, it is determined whether the product appearance inspection is qualified, wherein when the similarity between the color histograms of the color inspection object and the reference object is greater than a preset threshold, it is determined that the product appearance inspection is qualified, and when the similarity between the color histograms of the color inspection object and the reference object is not greater than the preset threshold, it is determined that the product appearance inspection is unqualified.

[0065] See also Figure 1 , which shows a flow chart of color inspection of the appearance of a vehicle provided by an exemplary embodiment. Figure 1 , perform image segmentation on the input image to obtain a segmentation result, which includes a vehicle body area, a door handle area and a rearview mirror area. After color space conversion, construct a color histogram corresponding to the image segmentation result, compare the color histogram of the door handle with the color histogram of the vehicle body, and obtain the color histogram similarity between the door handle and the vehicle body. And, compare the color histogram of the rearview mirror with the color histogram of the vehicle body, and obtain the color histogram similarity between the rearview mirror and the vehicle body, and then obtain the color inspection results of the door handle and the rearview mirror.

[0066] For another example, the color of different areas in the image can be determined by training a color classification network, and then the colors of the products can be compared to see if they are the same. In some embodiments, the foreground color is automatically located from the image through a gradient descent algorithm, and the appearance semantics at the color level are learned to achieve color classification of different objects. Then, the color consistency is determined based on the classification results. Figure 2 , which shows a flowchart of a method for determining whether the appearance colors of products are the same based on a color classification network provided by an exemplary embodiment. Figure 2 , perform image segmentation on the input image to obtain the segmentation results, identify the colors in the image corresponding to each area in each result through the color classification network, and compare whether the recognition results are consistent.

[0067] In practical applications, the surface coating of the product is often set to a coating with high gloss, such as the paint coating of a vehicle. Its surface is very delicate, so it is very easy to reflect the background environment around the product. Especially in the quality inspection process, the quality inspection workshop is well lit, and the reflection of the background environment will be further aggravated. This leads to the fact that when the reflection area of ​​the background environment is large, due to the different background colors reflected in different areas, even the same color cannot be recognized as the same result or the color histogram similarity is low. Or, the surfaces of products of different colors reflect the same color of the environment background, resulting in the same recognition result or the color histogram similarity is high.

[0068] In addition, since the image acquisition device can automatically adjust parameters such as exposure and white balance, when acquiring the target image of the product, it may cause color deviation between the acquired image and the actual product, ultimately leading to inaccurate recognition results.

[0069] In summary, due to the influence of factors such as the product surface coating and the parameters of the image acquisition device, when inspecting the appearance of the product, the problem of inaccurate appearance inspection results often occurs. The present application provides an appearance inspection method, device, electronic device and storage medium. After acquiring a target image containing a target object, the pixels in the target image are clustered based on the color of each pixel in the target image to obtain a clustering mask for representing the image area where the target object is located, and the target image is segmented to obtain a reference object mask corresponding to the main body of the target object in the target image and an inspection object mask corresponding to the component to be inspected for appearance, thereby determining the appearance inspection result of the component to be inspected in the target object by comparing the clustering mask, the object mask and the inspection object mask. In this way, the color of the product body is extracted by color clustering, thereby filtering the background color reflected on the product surface, reducing the influence of the reflected background color on the product surface color, and thereby improving the accuracy of the inspection results of the product appearance components.

[0070] See also Figure 3 , which shows a schematic diagram of a system involved in a visual inspection method provided by an exemplary embodiment. Figure 3 The system includes an image acquisition device 10 and an electronic device 20. The image acquisition device 10 and the electronic device 20 can perform data transmission.

[0071] The image acquisition device 10 is used to acquire a target image containing a target object and send the target image to the electronic device 20. The image acquisition device 10 can periodically acquire the target image according to a preset image acquisition cycle. The image acquisition device 10 can also acquire the target image after receiving an image acquisition instruction. The target acquisition instruction can be triggered by the electronic device 20 or by the user through the image acquisition device 10. In the embodiment of the present application, there is no specific limitation on this.

[0072] The electronic device 20 is used to receive the target image sent by the image acquisition device 10, and perform appearance inspection on the target object based on the target image. In some embodiments, the electronic device 20 is also used to send an image acquisition instruction to the image acquisition device 10. Accordingly, the electronic device 20 can also interact with the product inspector, and when receiving the acquisition operation triggered by the product inspector, it generates an image acquisition instruction and sends the image acquisition instruction to the image acquisition device 10.

[0073] It should be noted that the image acquisition device 10 and the electronic device 20 can be integrated devices. For example, the electronic device 20 is a mobile terminal such as a mobile phone or a tablet with a camera module, and accordingly, the image acquisition device 10 is a camera module on the mobile terminal. The image acquisition device 10 and the electronic device 20 can also be independent devices. For example, the image acquisition device 10 is a camera, and the electronic device 20 is a computer, a mobile terminal, or other device. In the embodiments of the present application, the image acquisition device 10 and the electronic device 20 are not specifically limited.

[0074] The appearance inspection method provided by the present application is described below in conjunction with specific embodiments. Figure 4 , which shows a flow chart of a method for visual inspection provided by an exemplary embodiment. As an example but not a limitation, the method is applied to electronic equipment.

[0075] S401, an electronic device acquires a target image, where the target image includes a target object to be visually inspected.

[0076] The target image is an image including the target object to be visually inspected. In some embodiments, the image acquisition device performs image acquisition upon receiving an image acquisition instruction sent by the electronic device, and sends the acquired image to the electronic device. Accordingly, the electronic device determines the received image as the target image. In some embodiments, the image acquisition device periodically or continuously sends the acquired image to the electronic device. Accordingly, the electronic device receives multiple images sent by the image acquisition device, and filters out an image including a complete target object from the multiple images to obtain the target image.

[0077] It should be noted that the process of the electronic device acquiring the target image for appearance inspection can be triggered by the user, that is, when the electronic device receives the appearance inspection instruction triggered by the user, it executes the step of acquiring the target image. The electronic device can also periodically acquire the target image for appearance inspection. Among them, the cycle of the appearance inspection can be determined according to the speed of the quality inspection line in the quality inspection workshop, and this is not specifically limited in the embodiments of the present application.

[0078] S402: The electronic device clusters the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object.

[0079] The cluster mask is a mask image composed of pixels corresponding to the main color block in the target image. Since the target object occupies the highest proportion of the screen in the target image, the mask image composed of pixels corresponding to the main color block is the mask image of the main body of the target object.

[0080] The electronic device clusters pixels of the same color in the target image into one category to obtain multiple clustering results, selects the clustering result with the most pixels from the clustering results, and composes the pixels in the clustering result with the most pixels into a clustering mask corresponding to the target object.

[0081] S403, the electronic device performs image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to the main body of the target object, and the inspection object mask is a mask corresponding to the component to be visually inspected in the target object.

[0082] The electronic device performs image segmentation on the target object through an image segmentation algorithm, and obtains a reference object mask and a check object mask from the complete target image through image segmentation.

[0083] It should be noted that for a target object, the number of components to be inspected may be one or more, so the number of inspection object masks in the image segmentation result may be one or more. For example, when the target object is a vehicle, the components to be inspected may include components such as rearview mirrors and door handles, so when the target object is a vehicle, the number of components to be inspected may be two or more.

[0084] The electronic device may use any method to segment the target image, for example, using a network such as U-Net, Mask R-CNN, YOLOv8-seg, etc. In the embodiment of the present application, the image segmentation method is not specifically limited.

[0085] In some embodiments, the electronic device may use a preset image segmentation network to perform image segmentation on the target image. In order to ensure the accuracy of the processing result of the image segmentation network, the input target image needs to be unified. Accordingly, before performing image segmentation on the target image, the electronic device may also scale or enlarge the target image and fill it to a preset size, and then normalize the adjusted target image, and input the normalized target image into the image segmentation network, thereby obtaining the reference object mask and the inspection object mask.

[0086] The preset size can be determined according to the processing capability of the electronic device, and in the embodiment of the present application, the preset size is not specifically limited. For example, the preset size is 640*640 or 1080*1080, etc.

[0087] S404: The electronic device determines a visual inspection result of a component to be inspected in the target object based on the cluster mask, the reference object mask and the inspection object mask.

[0088] Since the clustering mask is a mask corresponding to the main body of the target object composed of pixels of the same color in the target image, and the inspection object mask is a mask of the inspection object to be inspected based on image segmentation, when the inspection object of the target object is the same color as the main body of the target object, the clustering mask includes the image area corresponding to the inspection object. Therefore, if the inspection object of the target object is the same color as the main body of the target object, the clustering mask and the inspection object mask have an intersecting image area.

[0089] Therefore, the electronic device can determine whether the color of the component to be inspected of the target object is consistent with the main body of the target object based on whether there is an intersection between the clustering mask and the inspection object mask. If there is no intersection between the clustering mask and the inspection object mask, it means that the component corresponding to the inspection object mask is inconsistent with the color of the main body of the target object. If there is an intersection between the clustering mask and the inspection object mask, the appearance inspection result is further determined in combination with the reference object mask and the inspection object mask.

[0090] In an embodiment of the present application, after acquiring a target image containing a target object, the pixels in the target image are clustered based on the color of each pixel in the target image to obtain a clustering mask for representing the image area where the target object is located, and the target image is segmented to obtain a reference object mask corresponding to the main body of the target object in the target image and an inspection object mask corresponding to the component to be inspected for appearance, thereby determining the appearance inspection result of the component to be inspected in the target object by comparing the clustering mask, the object mask and the inspection object mask. In this way, the color of the main body of the product is extracted by color clustering, thereby filtering the background color reflected on the product surface, reducing the influence of the reflected background color on the product surface color, and thus improving the accuracy of the inspection result of the product appearance component.

[0091] In practical applications, the electronic device clusters the pixels in the target image and then selects the target clustering results that meet the conditions. Further, the above step S402 can be implemented by steps S502-S504 in the following embodiments. Figure 5 , which shows a flow chart of a method for visual inspection provided by an exemplary embodiment. As an example but not a limitation, the method is applied to electronic equipment.

[0092] S501, an electronic device acquires a target image, where the target image includes a target object to be visually inspected.

[0093] The principle of this step is the same as that of step S401 and will not be repeated here.

[0094] S502, the electronic device clusters the pixels in the target image based on the color of each pixel in the target image to obtain multiple classification results, wherein each of the classification results includes multiple pixels of the color corresponding to the classification result.

[0095] In some embodiments, when the electronic device clusters the target image, in order to improve the saturation of the color of the pixels in the target image and thus improve the accuracy of the clustering result, it is often necessary to pre-process the target image. Accordingly, this step can be implemented by the following steps S5021-S5025, including:

[0096] S5021, the electronic device enhances the saturation of the target image to obtain a high-saturation image.

[0097] In order to improve the saturation of the target image, the electronic device obtains a saturation channel for representing the saturation of the target image, and adjusts parameters of the saturation channel to improve the saturation of the target image.

[0098] Among them, the electronic device can convert the target image into the LCH color space (L represents the lightness channel of the image, i.e. Lightness; C represents the color saturation channel of the image, i.e. Chroma; H represents the hue channel of the image, i.e. Hue), obtain multiple feature channels of the target image, determine the saturation channel of the target image from the multiple feature channels, and increase the image parameters in the saturation channel by a preset multiple. In this way, the saturation of the target image is improved by increasing the saturation parameters of the target image, so that the difference between the foreground of the product body and the reflected background color in the Mubai image is increased, so as to improve the accuracy of target clustering.

[0099] Among them, the preset multiple can be set as needed. In the embodiment of the present application, the preset multiple is set to a number greater than 1. For example, the preset multiple can be 3, 5 or 5.5, etc.

[0100] It should be noted that the electronic device may also use other methods to increase the saturation of the target object. In the embodiment of the present application, the method for the electronic device to increase the saturation of the target object is not specifically limited.

[0101] S5022: The electronic device extracts color features of the high-saturation image.

[0102] In some embodiments, the electronic device can directly extract the color features of the high-saturation image through a preset color feature extraction algorithm. In other embodiments, in order to better characterize the color features of the high-saturation image, the electronic device converts the high-saturation image into the LAB color space (L represents the luminosity channel of the image, i.e., Luminosity; A represents the color channel ranging from red to green; B represents the color channel ranging from blue to yellow), and the electronic device performs color clustering on the color features corresponding to the A channel and the B channel in the LAB color space, respectively.

[0103] S5023, the electronic device determines the number of cluster centers of the target image.

[0104] The number of cluster centers determines the number of classification results obtained when clustering is completed. In some embodiments, the number of cluster centers is the number of colors of the scene in the target image. In other embodiments, the number of cluster centers can be the number of user inputs received by the electronic device. The number of cluster centers can be set as needed. In the embodiment of the present application, the number of cluster centers is not specifically limited. For example, the number of cluster centers can be 3, which respectively represent the main color of the product, a color darker than the main color of the product, and a color brighter than the main color.

[0105] S5024: The electronic device uniformly samples the color features of the high-saturation image based on the number of cluster centers to obtain multiple cluster centers.

[0106] The electronic device determines the position of each cluster center in the target image respectively to obtain multiple cluster centers.

[0107] In this step, the electronic device divides the target image into image regions having the same number of cluster centers based on the number of cluster centers, and for each image region, collects the central pixel point in the image region as a cluster center.

[0108] S5025: The electronic device performs clustering iterations on the high-saturation image based on the multiple cluster centers to obtain the multiple classification results.

[0109] The electronic device can select any color clustering algorithm to perform clustering iterations on the highly saturated image. For example, the K-means algorithm is used for clustering. In some embodiments, the electronic device counts the clustering rounds, and when the clustering rounds reach a preset number, it is determined that the clustering is finished, and the multiple classification results are obtained. Among them, the preset number of times can be set as needed. In the embodiment of the present application, the preset number of times is not specifically limited. For example, the preset number of times can be 500, 1000 or 2000, etc. In other embodiments, the electronic device counts the change amplitude of the cluster center after each iteration. If the change amplitude of the cluster center after two adjacent iterations is less than the preset amplitude, it is determined that the clustering is finished, and the multiple classification results are obtained.

[0110] In this implementation, the saturation of pixels in the target image is increased to improve the difference between the main color and other colors in the target image, so that different colors can be better distinguished during clustering, thereby achieving the purpose of improving the accuracy of clustering results.

[0111] S503: The electronic device determines, from the multiple classification results, a mask area corresponding to a target classification result with the most pixel points, to obtain the cluster mask.

[0112] The electronic device determines a target classification result including the most pixels from the multiple classification results, and forms the cluster mask with the multiple pixels in the target classification result.

[0113] It should be noted that the process of the electronic device performing image segmentation on the target image and the process of the electronic device performing color clustering on the target image can be performed simultaneously. Figure 6 , which shows a flow chart of a visual inspection method provided by an exemplary embodiment. Figure 6 After inputting the target image, the electronic device can simultaneously perform image segmentation process and color clustering process on the target.

[0114] S504, the electronic device performs image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to the main body of the target object, and the inspection object mask is a mask corresponding to the component to be visually inspected in the target object.

[0115] The principle of this step is the same as that of step S403 and will not be repeated here.

[0116] S505: The electronic device determines a visual inspection result of a component to be inspected in the target object based on the cluster mask, the reference object mask and the inspection object mask.

[0117] The principle of this step is the same as that of step S404 and will not be repeated here.

[0118] In an embodiment of the present application, after acquiring a target image containing a target object, the pixels in the target image are clustered based on the color of each pixel in the target image to obtain a clustering mask for representing the image area where the target object is located, and the target image is segmented to obtain a reference object mask corresponding to the main body of the target object in the target image and an inspection object mask corresponding to the component to be inspected for appearance, thereby determining the appearance inspection result of the component to be inspected in the target object by comparing the clustering mask, the object mask and the inspection object mask. In this way, the color of the main body of the product is extracted by color clustering, thereby filtering the background color reflected on the product surface, reducing the influence of the reflected background color on the product surface color, and thus improving the accuracy of the inspection result of the product appearance component.

[0119] In some embodiments, the electronic device determines the appearance inspection result according to whether there is an intersection between the cluster mask, the reference object mask and the inspection object mask. Further, the above step S403 can be implemented by steps S702-S705 in the following embodiments. Figure 7 , which shows a flow chart of a method for visual inspection provided by an exemplary embodiment. As an example but not a limitation, the method is applied to electronic equipment.

[0120] S701, an electronic device acquires a target image, where the target image includes a target object to be visually inspected.

[0121] The principle of this step is the same as that of step S401 and will not be repeated here.

[0122] S702: The electronic device clusters the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object.

[0123] The principle of this step is the same as that of step S402 and will not be repeated here.

[0124] See also Figure 6 After the electronic device obtains the clustering mask, the reference object mask and the inspection object mask, it determines whether the clustering mask and the reference object mask have an intersection. If there is no intersection mask area between the clustering mask and the inspection object mask, the electronic device executes step S703.

[0125] If there is an intersection mask area between the cluster mask and the inspection object mask, the electronic device determines that the cluster mask includes an area corresponding to the component to be inspected, so the color of the component to be inspected may be the same as the main body color.

[0126] In some embodiments, if there is an intersection mask area between the cluster mask and the inspection object mask, the electronic device can output the appearance inspection result as color inspection consistency. In other embodiments, in order to improve the accuracy of the appearance inspection, the electronic device can continue to determine whether the appearance color of the product is consistent based on the cluster mask, the inspection object mask and the reference object mask. Accordingly, if there is an intersection mask area between the cluster mask and the inspection object mask, the electronic device executes step S704.

[0127] S703: If there is no intersection mask region between the cluster mask and the inspection object mask, the electronic device determines that the appearance inspection result is color inspection inconsistency.

[0128] Since the cluster mask is a mask corresponding to the main body of the target object composed of pixels of the same color in the target image, and the inspection object mask is a mask of the inspection object to be inspected based on image segmentation, when the inspection object of the target object has the same color as the main body of the target object, the cluster mask includes the image area corresponding to the inspection object. Therefore, if the inspection object of the target object has the same color as the main body of the target object, the cluster mask and the inspection object mask have an intersecting image area. Therefore, if there is no intersection between the cluster mask and the inspection object mask, it means that the component corresponding to the inspection object mask is inconsistent with the main body color of the target object, and the electronic device can output the appearance inspection result as color inspection inconsistency.

[0129] S704: If there is an intersection mask region between the cluster mask and the inspection object mask, the electronic device updates the inspection object mask and the reference object mask based on the cluster mask.

[0130] If there is an intersection mask area between the cluster mask and the inspection object mask, in order to make the inspection object mask and the reference object mask more accurate, the electronic device can update the inspection object mask and the reference object mask based on the intersection mask area in the cluster mask. This process can be implemented by the following steps S7041-S7044, including:

[0131] S7041: The electronic device determines a first intersection mask between the clustering mask and the inspection object mask.

[0132] The electronic device determines the pixel points at the same position in the cluster mask and the inspection object mask as an intersection mask to obtain the first intersection mask.

[0133] S7042: The electronic device determines the first intersection mask as a new inspection object mask.

[0134] S7043: The electronic device determines a second intersection mask between the cluster mask and the reference object mask.

[0135] The principle of this step is the same as that of step S7041, and will not be repeated here.

[0136] S7044: The electronic device determines the second intersection mask as a new reference object mask.

[0137] In this implementation, the clustering mask is compared with the inspection object mask, and the intersection of the two is determined as the new inspection object mask. Similarly, the clustering mask is compared with the reference object mask, and the intersection of the two is determined as the new reference object mask. This makes the image area corresponding to the inspection object mask and the reference object mask more accurate, thereby improving the accuracy of appearance inspection.

[0138] S705: The electronic device determines the appearance inspection result based on the updated inspection object mask and the updated reference object mask.

[0139] The electronic device compares the colors of the inspection object and the reference object based on the updated inspection object mask and the updated reference object mask to obtain the appearance inspection result. In some embodiments, the electronic device can determine the appearance inspection result by determining the phase velocity between the color histograms between the reference object mask and the inspection object mask. The process can be implemented by the following steps S7051-S7054, including:

[0140] S7051: The electronic device determines a first image region of the inspection object from the target image based on the updated inspection object mask.

[0141] The electronic device extracts a first image region corresponding to the updated inspection object mask from the target image. In some embodiments, based on the updated inspection object mask, a first vector is extracted from the target image to obtain the first image region. The first vector includes an image feature corresponding to the inspection object mask in the target image.

[0142] S7052: The electronic device determines a second image region of the reference object from the target image based on the updated reference object mask.

[0143] The principle of this step is the same as that of step S7051, and will not be repeated here.

[0144] S7053: The electronic device constructs a first color histogram for the first image area, and constructs a second color histogram for the second image area.

[0145] In some embodiments, the electronic device constructs the different color histograms based on the first image area and the second image area, respectively. The method for constructing the color histogram can be selected as needed, and in the embodiments of the present application, the method for constructing the color histogram is not specifically limited.

[0146] In some embodiments, in order to reflect the color difference between the first image area and the second image area, an image channel with color features can be selected to construct the color histogram. The process can be: the electronic device extracts the image channel related to the color feature in the target image; based on the image channel related to the color feature, constructs the first color histogram of the first image area; and, based on the image channel related to the color feature, constructs the second color histogram of the second image area.

[0147] The electronic device can convert the target image into the LAB color space to obtain the L feature channel, A feature channel and B feature channel of the target image, wherein the A feature channel and the B feature channel are channels related to the characteristic features in the target image, and thus the A feature channel and the B feature channel are selected to construct a first color histogram of the first image area and a second color histogram of the second image area.

[0148] Among them, the electronic device can select any color histogram construction method to construct the first color histogram and the second color histogram. In the embodiment of the present application, there is no specific limitation on this.

[0149] In this implementation, feature channels related to the color image in the target image are selected to construct a color histogram, so that the color features of the constructed color histogram are more obvious, and the calculation of features such as brightness is reduced, thereby improving the efficiency of constructing the color histogram and reducing resource usage.

[0150] S7054: The electronic device determines the appearance inspection result based on the similarity between the first color histogram and the second color histogram.

[0151] The electronic device may determine the similarity between the first color histogram and the second color histogram through a similarity function. Accordingly, the electronic device determines the similarity between the first color histogram and the second color histogram based on the similarity function, and when the similarity is greater than a preset similarity, determines that the color of the component to be inspected is consistent with the color of the product body, and when the similarity is not greater than the preset similarity, determines that the color of the component to be inspected is inconsistent with the color of the product body.

[0152] The similarity function can be selected as needed, and in the embodiment of the present application, the similarity function is not specifically limited. For example, the similarity function can be a function such as Euclidean distance, cosine similarity, or chi-square distance.

[0153] In an embodiment of the present application, after acquiring a target image containing a target object, the pixels in the target image are clustered based on the color of each pixel in the target image to obtain a clustering mask for representing the image area where the target object is located, and the target image is segmented to obtain a reference object mask corresponding to the main body of the target object in the target image and an inspection object mask corresponding to the component to be inspected for appearance, thereby determining the appearance inspection result of the component to be inspected in the target object by comparing the clustering mask, the object mask and the inspection object mask. In this way, the color of the main body of the product is extracted by color clustering, thereby filtering the background color reflected on the product surface, reducing the influence of the reflected background color on the product surface color, and thus improving the accuracy of the inspection result of the product appearance component.

[0154] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0155] See also Figure 8 , which shows a schematic diagram of the structure of a visual inspection device provided by the present application, and includes various units for performing various steps in the above embodiments, see Figure 8 , the appearance inspection device comprises:

[0156] An acquisition unit 801 is used to acquire a target image, where the target image includes a target object to be visually inspected;

[0157] A clustering unit 802 is used to cluster the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object;

[0158] The image processing unit 803 is used to perform image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to the main body of the target object, and the inspection object mask is a mask corresponding to the component to be visually inspected in the target object;

[0159] The determining unit 804 is configured to determine a visual inspection result of a component to be inspected in the target object based on the clustering mask, the reference object mask and the inspection object mask.

[0160] In some embodiments, the determination unit 804 is used to determine that the appearance inspection result is a color inspection inconsistency if there is no intersection mask area between the clustering mask and the inspection object mask; if there is an intersection mask area between the clustering mask and the inspection object mask, update the inspection object mask and the reference object mask based on the clustering mask; and determine the appearance inspection result based on the updated inspection object mask and the updated reference object mask.

[0161] In some embodiments, the determination unit 804 is used to determine a first intersection mask between the clustering mask and the inspection object mask; determine the first intersection mask as a new inspection object mask; determine a second intersection mask between the clustering mask and the reference object mask; and determine the second intersection mask as a new reference object mask.

[0162] In some embodiments, the determination unit 804 is used to determine a first image area of ​​the inspection object from the target image based on the updated inspection object mask; determine a second image area of ​​the reference object from the target image based on the updated reference object mask; construct a first color histogram of the first image area, and construct a second color histogram of the second image area; and determine the appearance inspection result based on the similarity between the first color histogram and the second color histogram.

[0163] In some embodiments, the determination unit 804 is used to extract image channels related to color features in the target image; construct a first color histogram of the first image area based on the image channels related to the color features, and construct a second color histogram of the second image area based on the image channels related to the color features.

[0164] In some embodiments, the clustering unit 802 is used to cluster the pixels in the target image based on the color of each pixel in the target image to obtain multiple classification results, wherein each classification result includes multiple pixels of the color corresponding to the classification result; from the multiple classification results, determine the mask area corresponding to the target classification result with the most pixels to obtain the clustering mask.

[0165] In some embodiments, the clustering unit 802 is used to enhance the saturation of the target image to obtain a high-saturation image; extract color features of the high-saturation image; determine the number of cluster centers of the target image; based on the number of cluster centers, uniformly sample the color features of the high-saturation image to obtain multiple cluster centers; and iteratively cluster the high-saturation image based on the multiple cluster centers to obtain the multiple classification results.

[0166] In an embodiment of the present application, after acquiring a target image containing a target object, the pixels in the target image are clustered based on the color of each pixel in the target image to obtain a clustering mask for representing the image area where the target object is located, and the target image is segmented to obtain a reference object mask corresponding to the main body of the target object in the target image and an inspection object mask corresponding to the component to be inspected for appearance, thereby determining the appearance inspection result of the component to be inspected in the target object by comparing the clustering mask, the object mask and the inspection object mask. In this way, the color of the main body of the product is extracted by color clustering, thereby filtering the background color reflected on the product surface, reducing the influence of the reflected background color on the product surface color, and thus improving the accuracy of the inspection result of the product appearance component.

[0167] Fig. 9 FIG. 1 is a schematic diagram of an electronic device provided by an exemplary embodiment of the present application. Fig. 9 As shown, the electronic device 9 of this embodiment includes: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90, such as a visual inspection program. When the processor 90 executes the computer program 92, the steps in the above-mentioned various visual inspection method embodiments are implemented, such as Figure 4 Alternatively, when the processor 90 executes the computer program 92, the functions of each unit in the above-mentioned device embodiments are realized, for example Figure 8 The functions of the acquisition unit 801, clustering unit 802, image processing unit 803 and determination unit 804 are shown.

[0168] Exemplarily, the computer program 92 may be divided into one or more units, which are stored in the memory 91 and executed by the processor 90 to complete the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 92 in the electronic device 9. For example, the computer program 92 may be divided into an acquisition unit 801, a clustering unit 802, an image processing unit 803, and a determination unit 804, and the specific functions of each module are as follows:

[0169] An acquisition unit 801 is used to acquire a target image, where the target image includes a target object to be visually inspected;

[0170] A clustering unit 802 is used to cluster the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object;

[0171] The image processing unit 803 is used to perform image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to the main body of the target object, and the inspection object mask is a mask corresponding to the component to be visually inspected in the target object;

[0172] The determining unit 804 is configured to determine a visual inspection result of a component to be inspected in the target object based on the clustering mask, the reference object mask and the inspection object mask.

[0173] The electronic device 9 may be any electronic device having a control function. The electronic device 9 may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 9 and does not constitute a limitation of the electronic device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 9 may also include input and output devices, network access devices, buses, etc.

[0174] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0175] The memory 91 may be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. The memory 91 may also be an external storage device of the electronic device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 9. Further, the memory 91 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 91 is used to store the computer program and other programs and data required by the terminal device. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0176] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0177] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0178] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0179] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0182] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0183] The embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0184] The embodiment of the present application also provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0185] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for appearance inspection, characterized in that: The method comprises: Acquire a target image, wherein the target image includes a target object to be visually inspected; Clustering the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object; Performing image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to a main body of the target object, and the inspection object mask is a mask corresponding to a component to be visually inspected in the target object; Based on the clustering mask, the reference object mask and the inspection object mask, a visual inspection result of a component to be inspected in the target object is determined.

2. The method according to claim 1, characterized in that The determining, based on the clustering mask, the reference object mask and the inspection object mask, a visual inspection result of a component to be inspected in the target object comprises: If there is no intersection mask region between the cluster mask and the inspection object mask, determining that the appearance inspection result is color inspection inconsistency; If there is an intersection mask region between the cluster mask and the inspection object mask, updating the inspection object mask and the reference object mask based on the cluster mask; The appearance inspection result is determined based on the updated inspection object mask and the updated reference object mask.

3. The method according to claim 2, characterized in that The updating of the inspection object mask and the reference object mask based on the clustering mask comprises: determining a first intersection mask between the cluster mask and the inspection object mask; Determining the first intersection mask as a new inspection object mask; determining a second intersection mask between the cluster mask and the reference object mask; The second intersection mask is determined as a new reference object mask.

4. The method according to claim 2, characterized in that The determining the appearance inspection result based on the updated inspection object mask and the updated reference object mask includes: determining a first image region of the inspection object from the target image based on the updated inspection object mask; determining, from the target image, a second image region of the reference object based on the updated reference object mask; constructing a first color histogram of the first image area, and constructing a second color histogram of the second image area; The appearance inspection result is determined based on a similarity between the first color histogram and the second color histogram.

5. The method according to claim 4, characterized in that The step of constructing a first color histogram of the first image region and constructing a second color histogram of the second image region comprises: Extracting image channels related to color features in the target image; A first color histogram of the first image region is constructed based on the image channels associated with the color features, and a second color histogram of the second image region is constructed based on the image channels associated with the color features.

6. The method according to claim 1, characterized in that The clustering of the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object includes: Based on the color of each pixel in the target image, clustering the pixels in the target image to obtain a plurality of classification results, wherein each of the classification results includes a plurality of pixels of a color corresponding to the classification result; From the multiple classification results, a mask area corresponding to the classification result with the most pixel points is determined to obtain the cluster mask.

7. The method according to claim 6, characterized in that The clustering of the pixels in the target image based on the color of each pixel in the target image to obtain multiple classification results includes: Enhancing the saturation of the target image to obtain a high-saturation image; Extracting color features of the high-saturation image; Determining the number of cluster centers of the target image; Based on the number of cluster centers, uniformly sampling the color features of the high-saturation image to obtain a plurality of cluster centers; The high-saturation image is iterated by clustering based on the multiple cluster centers to obtain the multiple classification results.

8. A visual inspection device, characterized in that: The device comprises: An acquisition unit, used for acquiring a target image, wherein the target image includes a target object to be visually inspected; A clustering unit, configured to cluster the pixels in the target image based on the color of each pixel in the target image to obtain a clustering mask corresponding to the target object; An image processing unit, configured to perform image segmentation on the target image to obtain a reference object mask and an inspection object mask, wherein the reference object mask is a mask corresponding to a main body of the target object, and the inspection object mask is a mask corresponding to a component to be visually inspected in the target object; A determining unit is used to determine a visual inspection result of a component to be inspected in the target object based on the clustering mask, the reference object mask and the inspection object mask.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the appearance inspection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the appearance inspection method according to any one of claims 1 to 7 is implemented.

Images