Image to-be-detected area extraction method, device and product defect detection system

Abstract

The present application discloses a method, device and product defect detection system for extracting an area to be detected in an image. The method of the present application includes: acquiring a product image, and generating a mask image for describing the area to be detected based on the product image; performing channel-level splicing on the mask image and the product image to obtain a spliced ​​image, wherein the number of channels of the spliced ​​image is the sum of the number of channels of the mask image and the product image; in the spliced ​​image, based on the correspondence between the pixel points of the mask image and the product image, extracting the image to be detected including the area to be detected from the product image. The technical solution of the present application can quickly extract the location of the area to be detected of the product, reduce the impact of changes in the background of the product shooting on the test results of the defect detection model, and at the same time make the proportion of the product part in the picture input to the defect detection model larger, so as to be more conducive to the defect detection of small targets.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for extracting a region to be detected in an image, and a product defect detection system. Background Art

[0002] In the traditional precision manufacturing industry, product defects are generally classified and detected manually. In recent years, with the continuous maturity of detection technology, target detection methods based on deep learning have gradually replaced manual detection and become an important part of intelligent manufacturing. The target detection method based on deep learning generally uses a dedicated camera on the production line to automatically capture product images on the assembly line, and transmits the images to the neural network model, which determines whether the product has defects.

[0003] In this process, the quality of the images transmitted to the neural network model is very important, but the image quality is easily affected by many factors. For example, if the suction nozzle that absorbs the product becomes loose during use or the product is not neatly placed in the tray, the product image taken will be tilted to varying degrees; if the lighting or production line environment changes during the shooting process, a shadow will appear on the edge of the product image; if other products fall into the lens, the background of the product image will be disturbed by other products.

[0004] All of the above situations can easily cause the neural network model to make misjudgments, thereby automatically sorting out normal products as defective products, resulting in waste of materials and resources. Summary of the invention

[0005] The present application provides a method and device for extracting an area to be detected in an image and a product defect detection system to solve or partially solve the above-mentioned problems.

[0006] On the one hand, the present application provides a method for extracting an image to be detected area, comprising:

[0007] Acquire a product image, and generate a mask image for describing a to-be-detected area according to the product image;

[0008] The mask image and the product image are spliced ​​at the channel level to obtain a spliced ​​image, where the number of channels of the spliced ​​image is the sum of the number of channels of the mask image and the product image;

[0009] In the spliced ​​image, the image to be detected including the area to be detected is extracted from the product image according to the corresponding relationship between the pixels of the mask image and the product image.

[0010] On the other hand, the present application provides a device for extracting an image to-be-detected area, comprising:

[0011] An image segmentation unit, used for acquiring a product image and generating a mask image for describing a region to be detected according to the product image;

[0012] An image stitching unit, used for performing channel-level stitching on the mask image and the product image to obtain a stitched image, where the number of channels of the stitched image is the sum of the number of channels of the mask image and the product image;

[0013] The image extraction unit is used to extract the image to be detected including the area to be detected from the product image according to the corresponding relationship between the pixels of the mask image and the product image in the spliced ​​image.

[0014] On the other hand, the present application provides a product defect detection system, including: a camera, a memory, a processor, and a neural network platform;

[0015] A camera that captures product images of products on the production line and sends the product images to a processor;

[0016] a memory storing computer executable instructions;

[0017] A processor, which executes the method for extracting the image to be detected according to the computer executable instructions, and outputs the obtained image to be detected to the neural network platform;

[0018] The neural network platform uses a neural network model to determine whether there are defects in the products on the production line.

[0019] On the other hand, the present application provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, a method for extracting a region to be detected in an image is implemented.

[0020] The beneficial effect of the present application is that before the image to-be-detected area extraction method of the present invention is used for product defect detection, the image transmitted to the defect detection model is pre-processed in advance to quickly extract the location of the product's to-be-detected area, which can effectively reduce the impact of changes in the product shooting background on the defect detection model test results. At the same time, the proportion of the product part in the image input to the defect detection model is increased, which is more conducive to the defect detection of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of a method for extracting an image to-be-detected area according to an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of a process for extracting a region to be detected according to an embodiment of the present application;

[0023] Figure 3 This is a schematic diagram of the Unet network structure of an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a region to be detected in a product image according to an embodiment of the present application;

[0025] Figure 5 This is a schematic diagram of calculating the tilt angle and tilt direction according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of calculating the tilt angle and tilt direction according to another embodiment of the present application;

[0027] Figure 7 is a schematic diagram of an image to be detected obtained by rotating a target image in one embodiment of the present application;

[0028] Figure 8 This is a functional block diagram of a device for extracting an image to-be-detected area according to an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0030] Fig.10 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0034] On precision manufacturing production lines, due to errors in mechanical positioning accuracy, unstable processes, loosening of the machinery during use, and changes in the environment in the factory, the product images taken on the automated production line often have the following phenomena:

[0035] 1. The products in the pictures are tilted at different angles;

[0036] 2. There are partial lighting shadows in the background of the product being photographed;

[0037] 3. There are other products dropped in the background of the product.

[0038] The product images in the above situation have a negative impact on further deep learning target detection, and are prone to misjudgment of the background part, thereby detecting normal products as defective products, resulting in waste of production materials and resources on the production line.

[0039] In view of the above phenomenon existing in the captured product images, the present application embodiment proposes a method for extracting the image to be detected area. Figure 1 FIG. 1 is a flow chart of a method for extracting an image to-be-detected area according to an embodiment of the present application. Figure 1 As shown, the method of this embodiment includes the following steps:

[0040] Step S110 , acquiring a product image, and generating a mask image for describing the area to be detected according to the product image.

[0041] In this step, the image segmentation algorithm can be used to process the product image, predict the location of the area to be detected, and generate the corresponding mask image.

[0042] The mask image in this step is also called a mask image. The mask image is a binary image. In this embodiment, the grayscale value of the pixel points in the area to be detected in a mask image is 1, and the grayscale value of the pixel points in other positions is 0.

[0043] Step S120, performing channel-level splicing on the mask image and the product image to obtain a spliced ​​image, wherein the number of channels of the spliced ​​image is the sum of the number of channels of the mask image and the product image.

[0044] In this step, before performing channel-level stitching, the image sizes of the mask image and the product image need to be adjusted to the same size. For example, the mask image can be resized so that the processed mask image has the same size as the product image to facilitate channel-level stitching.

[0045] Step S130 : extracting the image to be detected including the area to be detected from the product image according to the corresponding relationship between the pixels of the mask image and the product image in the stitched image.

[0046] Depend on Figure 1 As shown, the method for extracting the area to be detected in the image of the present embodiment can quickly extract the location of the area to be detected of the product by pre-processing the image transmitted to the defect detection model before defect detection. This can effectively reduce the impact of changes in the background of the product shot on the test results of the defect detection model, and at the same time make the proportion of the product part in the image input to the defect detection model larger, so as to be more conducive to defect detection of small targets.

[0047] Taking a product image as an example, the implementation steps of the method for extracting the area to be detected of the product image are specifically described below.

[0048] refer to Figure 2 The extraction method of the area to be detected includes three design processes: generating a mask image, extracting a target image, and outputting an image to be detected. Figures 3 to 6 The three design processes are described in detail.

[0049] Generate mask image

[0050] This embodiment extracts the contour features of the area to be detected based on the image segmentation algorithm. After experiments, based on a small sample data consisting of dozens of images, the shallow Unet network structure can quickly complete the tasks of fitting and extracting the image area to be detected. Therefore, this embodiment uses a shallow Unet network to generate a mask image.

[0051] In one embodiment, a shallow Unet network is pre-constructed and trained, and the shallow Unet network includes a convolution part and an upsampling part, wherein the feature map output by the upsampling part is a feature map obtained by channel-level splicing with the feature map output by the convolution part, that is, the feature map output by the upsampling part is a feature map obtained by channel-level splicing of the feature map directly output by the upsampling part and the feature map output by the convolution part.

[0052] When training the shallow Unet network, the labeling tool labelme is used to mark the area to be detected in each sample image in the training set, and a mask image of the area to be detected is obtained. The grayscale value of the pixels in the area to be detected in the mask image is 1, and the grayscale value of the pixels in other positions is 0. Each sample image in the training set corresponds to a mask image of the image. During training, the sample image and the corresponding mask image are trained simultaneously.

[0053] like Figure 3 The shallow Unet network structure shown in the figure, the convolution part of the shallow Unet network extracts features from product images through 3x3 convolution kernels. The number of the first two convolution kernels is 64. After each additional convolution operation, the number of convolution kernels will increase by 2 times. Every two convolution kernels will cooperate with a 2x2 maximum pooling operation. After each maximum pooling, the length and width of the feature map will be halved.

[0054] The number of convolution kernels in the upsampling part starts to decrease from 1024. After each convolution operation, the number of convolution kernels decreases by half. With upsampling, the length and width of the feature map will double after each upsampling.

[0055] In the upsampling part, the feature maps collected at each level are concatenated with the feature maps collected at the corresponding level of the convolution part at the channel level. Figure 3 , the four horizontal arrows show that the feature maps of the four levels are concatenated at the corresponding channel level to add the large-size image features corresponding to the convolution part to the upsampling part, avoid the loss of image features, and improve the integrity of the features in the area to be detected in the mask image.

[0056] The mask image generated by the upsampling part is compared with the mask image obtained by annotating the sample image, and the loss function is calculated, and the parameters of the Unet network are continuously corrected. Among them, the loss function adopts the Dice-coefficient loss function (derived from binary classification, essentially measuring the overlap of two samples). Compared with the loss function of the original Unet network, the Dice-coefficient loss function is more effective for the problem of class imbalance, and can quickly measure the similarity between the generated mask image and the annotated mask image.

[0057] After the shallow Unet network is trained, when it is necessary to extract the area to be detected, the product image is input into the trained shallow Unet network, and the contour features of the area to be detected in the product image are extracted using the shallow Unet network to obtain a mask image including the area to be detected output by the shallow Unet network.

[0058] Extract target image

[0059] In one embodiment, the target image is extracted based on the following method:

[0060] According to the correspondence between the pixels of the mask image and the product image, the pixel position of the area to be detected is determined in the product image; based on the pixel position, a geometric figure including the area to be detected is selected on the product image, and the image of the area where the geometric figure is located is cropped on the product image to obtain the target image.

[0061] refer to Figure 4 , the geometry is preferably Figure 4 The dotted rectangle frame, Figure 4 The two elongated rectangular frames filled with diagonal stripes are used as interference, and a rectangle including the area to be detected is selected in the product image, and the target image is cropped according to the size of the rectangle. In order to make the product part of the image to be detected input into the defect detection model account for a larger proportion, it is preferred to select the minimum circumscribed rectangle including the area to be detected, and crop the image of the area where the minimum circumscribed rectangle is located on the product image to obtain the target image, which is more conducive to the defect detection of small targets.

[0062] Output the image to be detected

[0063] In one embodiment, after obtaining the target image, the tilt degree of the target image is calculated; the target image is adjusted to be straight according to the tilt degree to obtain the image to be detected.

[0064] In one embodiment, the angle between a border line of the geometric figure and a preset reference line is calculated, and the angle is used as the inclination angle of the target image. The inclination angle is used to represent the degree of inclination of the target image relative to the horizontal reference line or the vertical reference line.

[0065] The preset reference line includes a horizontal reference line or a vertical reference line. Figure 5 For the target image tilted in the first direction, the angle θ1 between the border line A1B1 between the vertex A1 and the vertex B1 and the horizontal reference line L1 can be taken as the tilt angle of the target image; or the angle θ1 between the border line A1D1 between the vertex A1 and the vertex D1 and the vertical reference line L1' can be taken as the tilt angle of the target image. Figure 6 For the target image tilted in the second direction shown, the angle θ2 between the border line A2D2 between vertex A2 and vertex B2 and the horizontal reference line L2 can be taken as the tilt angle of the target image; or, the angle θ2 between the border line A2B2 between vertex A2 and vertex D2 and the vertical reference line L2' can be taken as the tilt angle of the target image.

[0066] After obtaining the inclination angle of the target image, it is possible to determine whether the target image is straight based on the inclination angle, that is, when the inclination angle is zero, the target image is in a straight state, and the target image can be directly used as the image to be detected; when the inclination angle is not zero, the target image is in an inclined state, and the inclination direction of the target image can be calculated based on the vertex coordinates of the geometric figure, and the target image can be rotated according to the inclination direction until the inclination angle is zero, and the target image when the inclination angle is zero is used as the image to be detected.

[0067] In one embodiment, when the target image is tilted in a first direction, the target image is rotated clockwise until the tilt angle is zero; when the target image is tilted in a second direction, the target image is rotated counterclockwise until the tilt angle is zero. Figure 5 and Figure 6 The first tilting direction can be understood as a counterclockwise direction, and the second tilting direction can be understood as a clockwise direction.

[0068] In general production lines, the tilt angle of product images varies from -60° to 60°, that is, the products are mostly tilted, rather than 360-degree flips, 180-degree inverted stands, or 90-degree rollovers. Therefore, the tilt direction of the target image can be calculated based on the relationship between the vertex coordinates of the geometric figures corresponding to the target image.

[0069] In one embodiment, a method for calculating the tilt direction of a target image is described by taking a geometric figure as the minimum circumscribed rectangle including the area to be detected as an example:

[0070] like Figure 5 or Figure 6 As shown, in the rectangular coordinate system constructed with the upper left corner pixel of the product image as the origin, the vertex with the largest ordinate value among the four vertices of the minimum circumscribed rectangle is marked as the first vertex, and the other three vertices of the minimum circumscribed rectangle are marked in a counterclockwise direction with the first vertex as the reference point as the second vertex, the third vertex and the fourth vertex; refer to Figure 5 In the minimum circumscribed rectangle A1B1C1D1, the vertical coordinate value of vertex A1 is the largest. Therefore, vertex A1 is marked as the first vertex 01 (x1, y1), vertex B1 is marked as the second vertex 02 (x2, y2), vertex C1 is marked as the third vertex 03 (x3, y3), and vertex D1 is marked as the fourth vertex 04 (x4, y4).

[0071] The tilt direction of the target image is calculated according to the magnitude relationship between the horizontal coordinates of the first vertex 01 (x1, y1) and the third vertex 03 (x3, y3), specifically: if the horizontal coordinate |x3| of the third vertex 03 (x3, y3) is greater than the horizontal coordinate |x1| of the first vertex 01 (x1, y1), the target image tilts toward the first direction; if the horizontal coordinate |x3| of the third vertex 03 (x3, y3) is less than the horizontal coordinate |x1| of the first vertex 01 (x1, y1), the target image tilts toward the second direction;

[0072] Alternatively, the tilt direction of the target image is calculated based on the slope of the straight line determined by the first vertex 01 (x1, y1) and the third vertex 03 (x3, y3), specifically: if the slope of the straight line is greater than zero, the target image tilts in the first direction; if the slope of the straight line is less than zero, the target image tilts in the second direction. Figure 5 As shown, the slope of the straight line A1C1 determined by the first vertex 01 (x1, y1) and the third vertex 03 (x3, y3) is greater than zero, so, Figure 5 The target image in is tilted towards the first direction. Figure 6 The slope of the straight line A2C2 determined by the first and third vertices is less than zero, so, Figure 6 The target image in is tilted toward the second direction.

[0073] When the target image is tilted toward the first direction, the angle between the border line determined by the second vertex and the first vertex and the horizontal reference line is calculated as the tilt angle; Figure 5 As shown, the tilt angle

[0074] When the target image is tilted toward the second direction, the angle between the border line determined by the fourth vertex and the first vertex and the horizontal reference line is calculated as the tilt angle; Figure 6 As shown, the tilt angle

[0075] like Figure 7 As shown, through the above method steps, the target image in a tilted state can be rotated to a straight state to obtain a straight image to be detected, and the straight image to be detected is sent to the defect detection model, which can effectively reduce the influence of external factors such as background on the image effect and effectively improve the accuracy of defect detection.

[0076] Corresponding to the above method, the present application also provides a device for extracting an image to-be-detected area, Figure 8 is a functional block diagram of an image to-be-detected region extraction device according to an embodiment of the present application. Figure 8 The device 800 of this embodiment includes:

[0077] An image segmentation unit 810 is used to obtain a product image and generate a mask image for describing a region to be detected according to the product image;

[0078] An image stitching unit 820 is used to perform channel-level stitching on the mask image and the product image to obtain a stitched image, where the number of channels of the stitched image is the sum of the number of channels of the mask image and the product image;

[0079] The image extraction unit 830 is used to extract the image to be detected including the area to be detected from the product image according to the corresponding relationship between the pixels of the mask image and the product image in the spliced ​​image.

[0080] In one embodiment, the image segmentation unit 810 includes a network construction and training module;

[0081] The network construction and training module is used to pre-construct and train a shallow Unet network, which includes a convolution part and an upsampling part, wherein the feature map output by the upsampling part is a feature map obtained by channel-level concatenation with the feature map output by the convolution part;

[0082] The image segmentation unit 810 is used to input the product image into the trained shallow Unet network, use the shallow Unet network to extract the contour features of the area to be detected in the product image, and obtain the mask image output by the shallow Unet network including the area to be detected.

[0083] In one embodiment, the image extraction unit 830 includes: a target image extraction module, a tilt calculation module and an image rotation module;

[0084] A target image extraction module is used to determine the pixel position of the area to be detected in the product image according to the corresponding relationship between the pixel points of the mask image and the product image; select a geometric figure including the area to be detected on the product image based on the pixel position, and crop the image of the area where the geometric figure is located on the product image to obtain a target image;

[0085] An inclination calculation module, used to calculate the inclination degree of the target image;

[0086] The image rotation module is used to adjust the target image to be straight according to the tilt degree to obtain the image to be detected.

[0087] In one embodiment, the inclination calculation module is used to calculate the angle between a border line of the geometric figure and a preset reference line; the angle is used as the inclination angle of the target image, and the inclination angle is used to represent the degree of inclination of the target image relative to the preset reference line.

[0088] In one embodiment, an image rotation module is used to use the target image as the image to be detected when the inclination angle is zero; when the inclination angle is not zero, calculate the inclination direction of the target image according to the vertex coordinates of the geometric figure, and rotate the target image according to the inclination direction until the inclination angle is zero, and use the target image when the inclination angle is zero as the image to be detected.

[0089] The image rotation module is further used to rotate the target image clockwise until the tilt angle is zero when the target image tilts in a first direction; and to rotate the target image counterclockwise until the tilt angle is zero when the target image tilts in a second direction.

[0090] In one embodiment, the geometric figure is the minimum circumscribed rectangle including the area to be detected, and the image extraction unit 830 further includes: a tilt direction calculation module;

[0091] The tilt direction calculation module is used to mark the vertex with the largest ordinate value among the four vertices of the minimum circumscribed rectangle as the first vertex in a rectangular coordinate system constructed with the upper left corner pixel point of the product image as the origin, and mark the other three vertices of the minimum circumscribed rectangle in a counterclockwise direction with the first vertex as the reference point as the second vertex, the third vertex and the fourth vertex; calculate the tilt direction of the target image according to the size relationship between the horizontal coordinates of the first vertex and the third vertex, specifically: if the horizontal coordinate of the third vertex is greater than the horizontal coordinate of the first vertex, the target image is tilted in the first direction; if the horizontal coordinate of the third vertex is less than the horizontal coordinate of the first vertex, the target image is tilted in the second direction; or calculate the tilt direction of the target image according to the slope of the straight line determined by the first vertex and the third vertex, specifically: if the slope of the straight line is greater than zero, the target image is tilted in the first direction; if the slope of the straight line is less than zero, the target image is tilted in the second direction.

[0092] In one embodiment, the preset reference line includes a horizontal reference line, and accordingly, the inclination calculation module is used to calculate the angle between the border line determined by the second vertex and the first vertex and the horizontal reference line as the inclination angle when the target image is tilted in the first direction; when the target image is tilted in the second direction, the angle between the border line determined by the fourth vertex and the first vertex and the horizontal reference line is calculated as the inclination angle.

[0093] In summary, on the one hand, the embodiment of the present application utilizes the Unet network to locate the area to be detected of the product image, generates a mask image describing the area to be detected, and then the method of cropping the image to be detected on the product image based on the mask image can improve the positioning accuracy of the area to be detected of the product while ensuring the cropping speed, and reduce the probability of being affected by interference items; on the other hand, the embodiment of the present application can effectively reduce the impact of different product inclination angles on the test results of the defect detection model by rotating the target image to the image straight, and avoid misjudgment of the product; on the other hand, the present application can increase the proportion of the product part in the picture by cropping the image to be detected including the area to be detected of the product from the product image, so as to facilitate the defect detection model to detect defects of small targets.

[0094] It should be noted that:

[0095] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description of the specific language above is for the purpose of disclosing the best mode of the application.

[0096] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0097] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0098] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0099] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0100] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the image to be measured area extraction device according to the embodiment of the present application. The present application can also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0101] For example, Fig. 9The structural schematic diagram of a product defect detection system according to an embodiment of the present application is shown. The target detection system 900 includes a camera 940, a neural network platform 950, a processor 910, and a memory 920 arranged to store computer executable instructions (computer readable program code). The camera 940 captures the product image of the product on the production line and sends the product image to the processor 910. The processor 910 executes the image to be detected area extraction method according to the computer executable instruction, and outputs the obtained image to be detected to the neural network platform 950. The neural network platform 950 uses the neural network model to determine whether the product on the production line has defects. Among them, the memory 920 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. The memory 920 has a storage space 930 for storing a computer readable program code 931 for executing any method step in the above method. For example, the storage space 930 for storing computer readable program code may include various computer readable program codes 931 for implementing various steps in the above method. Computer readable program code 931 can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards, or floppy disks. Such computer program products are typically, for example, Fig.10 The computer-readable storage medium.

[0102] Fig.10 A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application is shown. The computer-readable storage medium 1000 stores a computer-readable program code 931 for executing the method steps according to the present application, which can be read by the processor 910 of the target detection system 900. When the computer-readable program code 931 is executed by the target detection system 900, the target detection system 900 executes each step in the method described above. Specifically, the computer-readable program code 931 stored in the computer-readable storage medium can execute the method shown in any of the above embodiments. The computer-readable program code 931 can be compressed in an appropriate form.

[0103] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A method for extracting an image region to be detected, characterized in that: include: Acquire a product image, and generate a mask image for describing a to-be-detected area according to the product image; Perform channel-level splicing on the mask image and the product image to obtain a spliced ​​image, wherein the number of channels of the spliced ​​image is the sum of the number of channels of the mask image and the product image; In the stitched image, according to the corresponding relationship between the pixels of the mask image and the product image, the image to be detected including the area to be detected is extracted from the product image; Extracting the image to be detected including the area to be detected from the product image according to the corresponding relationship between the pixels of the mask image and the product image, comprising: Determine the pixel position of the area to be detected in the product image according to the corresponding relationship between the pixels of the mask image and the product image; Selecting a geometric figure including the area to be detected on the product image based on the pixel position, and cropping an image of the area where the geometric figure is located on the product image to obtain a target image; Calculating the tilt degree of the target image; According to the tilt degree, the target image is adjusted until the image is straight to obtain an image to be detected; The tilt degree refers to the tilt degree of the target image relative to the coordinate axis of the rectangular coordinate system constructed with the upper left corner pixel of the product image as the origin.

2. The method according to claim 1, characterized in that Generating a mask image for describing the area to be detected according to the product image, including: Pre-construct and train a shallow Unet network, which includes a convolution part and an upsampling part, wherein the feature map output by the upsampling part is a feature map obtained by channel-level concatenation with the feature map output by the convolution part; The product image is input into a trained shallow Unet network, and the contour features of the area to be detected in the product image are extracted using the shallow Unet network to obtain a mask image output by the shallow Unet network including the area to be detected.

3. The method according to claim 1, characterized in that Calculating the tilt degree of the target image includes: Calculating an angle between a border line of the geometric figure and a preset reference line, wherein the preset reference line includes a horizontal reference line or a vertical reference line, wherein the horizontal reference line is parallel to an X-axis of the rectangular coordinate system, and the vertical reference line is parallel to a Y-axis of the rectangular coordinate system; The included angle is used as the inclination angle of the target image, and the inclination degree of the target image relative to the preset reference line is represented by the inclination angle.

4. The method according to claim 3, characterized in that Adjusting the target image to be straight according to the tilt degree to obtain the image to be detected includes: When the tilt angle is zero, the target image is used as the image to be detected; When the tilt angle is not zero, the tilt direction of the target image is calculated according to the vertex coordinates of the geometric figure, and the target image is rotated according to the tilt direction until the tilt angle is zero, and the target image when the tilt angle is zero is used as the image to be detected.

5. The method according to claim 4, characterized in that When the tilt angle is not zero, rotating the target image according to the tilt direction until the tilt angle is zero, comprising: When the target image is tilted toward the first direction, rotating the target image clockwise until the tilt angle is zero; When the target image is tilted toward the second direction, the target image is rotated counterclockwise until the tilt angle is zero.

6. The method according to claim 4, characterized in that The geometric figure is the minimum circumscribed rectangle of the area to be detected, and the tilt direction of the target image is calculated according to the vertex coordinates of the geometric figure, including: In a rectangular coordinate system constructed with the pixel point at the upper left corner of the product image as the origin, mark the vertex with the largest ordinate value among the four vertices of the minimum circumscribed rectangle as the first vertex, and mark the other three vertices of the minimum circumscribed rectangle in a counterclockwise direction with the first vertex as the reference point as the second vertex, the third vertex, and the fourth vertex; The tilt direction of the target image is calculated according to the magnitude relationship between the abscissas of the first vertex and the third vertex, specifically: if the abscissa of the third vertex is greater than the abscissa of the first vertex, the target image tilts toward the first direction; if the abscissa of the third vertex is less than the abscissa of the first vertex, the target image tilts toward the second direction; Alternatively, the inclination direction of the target image is calculated according to the slope of the straight line determined by the first vertex and the third vertex, specifically: if the slope of the straight line is greater than zero, the target image is inclined in the first direction; if the slope of the straight line is less than zero, the target image is inclined in the second direction.

7. The method according to claim 6, characterized in that The preset reference lines include a horizontal reference line. The angle between a border line of the geometric figure and the horizontal reference line or the vertical reference line is calculated, including: When the target image is tilted toward the first direction, calculating the angle between the border line determined by the second vertex and the first vertex and the horizontal reference line as the tilt angle; When the target image is tilted toward the second direction, an angle between a border line determined by the fourth vertex and the first vertex and the horizontal reference line is calculated as the tilt angle.

8. A device for extracting an image region to be detected, characterized in that: include: An image segmentation unit, used for acquiring a product image and generating a mask image for describing a to-be-detected area according to the product image; An image stitching unit, used for performing channel-level stitching on the mask image and the product image to obtain a stitched image, wherein the number of channels of the stitched image is the sum of the number of channels of the mask image and the product image; An image extraction unit, configured to extract, in the stitched image, an image to be detected including the area to be detected from the product image according to a correspondence between pixels of the mask image and the product image; The image extraction unit is further used for: Determine the pixel position of the area to be detected in the product image according to the corresponding relationship between the pixels of the mask image and the product image; Selecting a geometric figure including the area to be detected on the product image based on the pixel position, and cropping an image of the area where the geometric figure is located on the product image to obtain a target image; Calculating the tilt degree of the target image; According to the tilt degree, the target image is adjusted until the image is straight to obtain an image to be detected; The tilt degree refers to the tilt degree of the target image relative to the coordinate axis of the rectangular coordinate system constructed with the upper left corner pixel of the product image as the origin.

9. A product defect detection system, characterized in that: Includes cameras, memory, processors, and neural network platforms; The camera captures product images of products on the production line and sends the product images to the processor; The memory stores computer executable instructions; The processor executes the method for extracting the image to be detected area according to any one of claims 1 to 7 according to the computer executable instructions, and outputs the obtained image to be detected to the neural network platform; The neural network platform uses a neural network model to determine whether there are defects in the products on the production line.

Images