Bubble detection method, terminal and computer readable storage medium

Abstract

The present invention provides a bubble detection method, a terminal and a computer-readable storage medium. In the present application, a glue spotting area map in the lens screen image is obtained by extracting features from the acquired lens screen image to be detected; a semantic segmentation model is then used to process the glue spotting area map to generate a bubble mask image; and then the bubble mask image is used to determine whether the bubble is qualified. The bubble detection method of the present application can automatically detect and extract the glue spotting area map without physical manual work, avoiding manual intervention in the detection results, and can improve both the detection accuracy and the detection efficiency; and the bubble mask image generated by the semantic segmentation network model is used to determine whether the bubble is qualified, which can improve the detection accuracy of the bubble type.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a bubble detection method, a terminal and a computer-readable storage medium. Background Art

[0002] When installing the screen on the lens, glue needs to be applied to a certain area around the screen so that the screen is fixed on the lens through glue. However, due to process reasons, the glue dispensing operation may result in no glue, less glue, or uneven glue dispensing. These situations will cause bubbles to form in the glue dispensing area inside the screen. If the bubbles are too large or too many, it will affect the quality of the entire product. Therefore, it is particularly important to detect whether the bubbles meet the specifications. Summary of the invention

[0003] The main technical problem solved by the present invention is to provide a bubble detection method, a terminal and a computer-readable storage medium to solve the problem in the prior art of how to judge from the screen image whether the bubbles in the dispensing area meet the requirements.

[0004] To solve the above technical problems, the first technical solution adopted by the present invention is: to provide a bubble detection method, which includes: obtaining a lens screen image to be detected; extracting a glue spot area map in the lens screen image; using a semantic segmentation model to process the glue spot area map to generate a bubble mask image; and determining whether the bubble is qualified through the bubble mask image.

[0005] Among them, the step of extracting the glue spot area map in the lens screen image includes: detecting the glue spot area map of the lens screen image by detecting the network model; determining the shape of the glue spot area map and the position information of the glue spot area map in the lens screen image; extracting the glue spot area map in the lens screen image according to the shape and position information.

[0006] Among them, the detection network model includes a first convolution layer, a first pooling layer, and a first upsampling layer which are cascaded in sequence; the step of detecting the glue dispensing area map of the lens screen image through the detection network model includes: the first convolution layer extracts features of the lens screen image to obtain a first feature map; the first pooling layer compresses the first feature map to obtain a second feature map; the first upsampling layer amplifies the second feature map to obtain a third feature map; the third feature map is connected with the first feature map to output a glue dispensing area map corresponding to the lens screen image.

[0007] Among them, the detection network model also includes a second convolutional layer, the second convolutional layer is arranged between the first pooling layer and the first upsampling layer, and the second convolutional layer is cascaded with the upsampling layer; the step of the first upsampling layer amplifying the second feature map to obtain the third feature map includes: the second convolutional layer extracts the feature map of the second feature map to obtain the fourth feature map; the first upsampling layer amplifies the fourth feature map to obtain the third feature map.

[0008] Among them, the detection network model is the YOLOv3-Tiny network deployed on the Caffe network framework.

[0009] Among them, the semantic segmentation model includes multiple upper convolution layers, pooling layers, upsampling layers and lower convolution layers; the semantic segmentation model is used to process the glue dispensing area map, and the steps of generating a bubble mask image specifically include: the upper convolution layer encodes and downsamples the glue dispensing area maps of different sizes to obtain feature maps of different sizes; the pooling layer compresses the feature maps of different sizes; the upsampling layer decodes the feature map compressed by the pooling layer; the feature map decoded by the upsampling layer is connected to the feature map encoded by the upper convolution layer; the feature map of the upsampling layer connected to the upper convolution layer is extracted through the lower convolution layer to obtain a bubble mask map of the glue dispensing area map.

[0010] Among them, the semantic segmentation model includes four down-convolutional layers, and the number of image channels of the four down-convolutional layers are 16, 32, 64, and 128 respectively.

[0011] Among them, the semantic segmentation model is a U-net network deployed on the OpenVINO network framework.

[0012] Among them, after the step of extracting the glue dot area in the lens screen image, it also includes: drawing a ray with the center of the lens screen image as the starting point, and calculating the pixels on the ray within the set direction range; judging whether the difference between the previous pixel point and the next pixel point of the current pixel point on the ray is greater than a preset value; if the difference between the previous pixel point and the next pixel point is greater than the preset value, determining that the current pixel point is an edge point.

[0013] Among them, after the step of extracting the glue-dotting area in the lens screen image, the step also includes: filtering and fitting multiple edge points according to the shape characteristics of the glue-dotting area to obtain the inner edge shape and position of the glue-dotting area.

[0014] Among them, the steps of filtering and fitting multiple edge points according to the shape characteristics of the glue spot area to obtain the inner edge shape and position of the glue spot area specifically include: dividing the glue spot area into multiple areas in the horizontal / vertical direction; counting the number of edge points in multiple areas; selecting two areas with the largest number of edge points in the horizontal / vertical direction, and inversely synthesizing the edge points in the two areas into two straight lines respectively.

[0015] Among them, after the step of extracting the glue dot area in the lens screen image, it also includes: judging whether the inner edge shape conforms to the preset shape; if the inner edge shape does not conform to the preset shape, reducing the shape level or performing noise reduction processing on the edge points and fitting the inner edge shape again.

[0016] Among them, the step of determining whether the bubble is qualified through the bubble mask image specifically includes: judging whether the distance between the bubble and the inner edge or the outer edge of the glue dispensing area is greater than a preset distance; if greater, the bubble is a central bubble; if not greater, the bubble is an edge bubble.

[0017] Among them, the step of determining whether the bubble is qualified through the bubble mask image specifically includes: judging whether the information of the bubble meets the preset information; if the information of the bubble does not meet the preset information, issuing an alarm.

[0018] Wherein, before the step of obtaining the lens screen image to be detected, the step includes: training the initial detection network model to obtain the detection network model.

[0019] Among them, the step of training the initial detection network model to obtain the detection network model includes: obtaining a first training sample set, the first training sample set includes multiple screen images, the screen images are images marked with the real position information and the real shape of the glue spot area; detecting the screen image through the initial detection network model to obtain the predicted position information and predicted shape of the glue spot area in the screen image; constructing a first loss function through the predicted position information and the real position information, the predicted shape and the real shape of the glue spot area; iteratively training the initial detection network model through the first loss function to obtain the detection network model.

[0020] Wherein, before the step of obtaining the lens screen image to be detected, the step includes: training the initial semantic segmentation model to obtain the semantic segmentation model.

[0021] Among them, the step of training the initial semantic segmentation model to obtain the semantic segmentation model includes: obtaining a second training sample set, the second training sample set includes multiple dispensing area images, and the dispensing area images are images marked with the real position of the bubbles and the real type of the bubbles; detecting the dispensing area images through the initial semantic segmentation model to obtain the predicted position and predicted type of the bubbles in the dispensing area images; constructing a second loss function through the predicted position and real position, predicted type and real type of the bubbles; iteratively training the initial semantic segmentation model through the second loss function to obtain the semantic segmentation model.

[0022] To solve the above technical problems, the second technical solution adopted by the present invention is: to provide a terminal, the terminal includes a memory, a processor and a computer program stored in the memory and running on the processor, and the processor is used to implement the steps of the above bubble detection method.

[0023] In order to solve the above technical problems, the third technical solution adopted by the present invention is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above bubble detection method are implemented.

[0024] The beneficial effects of the present invention are as follows: Different from the prior art, a bubble detection method, terminal and computer-readable storage medium are provided. In the present application, a glue spotting area map in the lens screen image is obtained by extracting features from the acquired lens screen image to be detected; the glue spotting area map is processed by a semantic segmentation model to generate a bubble mask image; and then the bubble mask image is used to determine whether the bubble is qualified. The bubble detection method of the present application can automatically detect and extract the glue spotting area map without physical manual operation, avoiding manual intervention in the detection results, and can improve both detection accuracy and detection efficiency; and the bubble mask image generated by the semantic segmentation model is used to determine whether the bubble is qualified, which can improve the detection accuracy of the bubble type. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic flow chart of the bubble detection method provided by the present invention;

[0026] Figure 2 It is a flow chart of a specific embodiment of the bubble detection method provided by the present invention;

[0027] Figure 3 yes Figure 2 A schematic flow chart of a specific embodiment of step S201 in the provided bubble detection method;

[0028] Figure 4 yes Figure 2 A flow chart of a specific embodiment of step S202 in the provided bubble detection method;

[0029] Figure 5 is a schematic block diagram of an implementation manner of a terminal provided by the present invention;

[0030] Figure 6 It is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0031] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0032] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0033] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, a bubble detection method provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] See also Figure 1 , Figure 1 Schematic diagram of the process of the bubble detection method provided by the present invention. In this embodiment, a bubble detection method is provided, and the bubble detection method comprises the following steps.

[0036] S11: Acquire the screen image of the lens to be detected.

[0037] Specifically, an image including a lens screen is collected by an image acquisition device. The image of the lens screen is collected under a normal lighting environment. The lens screen image includes glue-dispensing areas distributed around. The lens screen is fixed to the lens barrel through the glue-dispensing areas. The image acquisition device may be a camera. In a specific embodiment, the image is collected from a side of the lens screen away from the lens barrel.

[0038] S12: extracting a dispensing area map from the lens screen image.

[0039] Specifically, the glue dispensing area map of the lens screen image is detected by the detection network model. The shape of the glue dispensing area map and the position information of the glue dispensing area map in the lens screen image are determined; and the glue dispensing area map in the lens screen image is extracted according to the shape and position information. Among them, the detection network model is a YOLOv3-Tiny network deployed on the Caffe network framework.

[0040] In a specific embodiment, after detecting the dispensing area map of the lens screen image, the method further includes: drawing a ray with the center of the lens screen image as the starting point, calculating the pixels on the ray within the set direction range; judging whether the difference between the previous pixel and the next pixel of the current pixel on the ray is greater than a preset value; if the difference between the previous pixel and the next pixel is greater than the preset value, determining that the current pixel is an edge point. In an optional embodiment, filtering and fitting multiple edge points according to the shape characteristics of the dispensing area obtains the inner edge shape and position of the dispensing area.

[0041] S13: The dispensing area map is processed using a semantic segmentation model to generate a bubble mask image.

[0042] Specifically, the semantic segmentation model includes multiple upper convolutional layers, pooling layers, upsampling layers and lower convolutional layers. Among them, the semantic segmentation model includes four lower convolutional layers, and the number of image channels of the four lower convolutional layers is 16, 32, 64, and 128 respectively. The semantic segmentation model is a U-net network deployed on the OpenVINO network framework. In an optional embodiment, the upper convolutional layer encodes and downsamples the dispensing area maps of different sizes to obtain feature maps of different sizes; the pooling layer compresses the feature maps of different sizes; the upsampling layer decodes the feature map compressed by the pooling layer; the feature map decoded by the upsampling layer is connected to the feature map encoded by the upper convolutional layer; the feature map of the upsampling layer connected to the upper convolutional layer is extracted through the lower convolutional layer to obtain a bubble mask map of the dispensing area map.

[0043] S14: Determine whether the bubble is qualified through the bubble mask image.

[0044] Specifically, it is determined whether the information of the bubble meets the preset information; if the information of the bubble does not meet the preset information, an alarm is issued to remind the lens that the bubble in the screen image does not meet the requirements.

[0045] In the bubble detection method provided in this embodiment, the glue spotting area map in the lens screen image is obtained by extracting features from the acquired lens screen image to be detected; the glue spotting area map is then processed using a semantic segmentation model to generate a bubble mask image; and the bubble mask image is used to determine whether the bubble is qualified. The bubble detection method can automatically detect and extract the glue spotting area map without physical manual work, avoiding manual intervention in the detection results, and can improve both detection accuracy and detection efficiency; and the bubble mask image generated by the semantic segmentation model is used to determine whether the bubble is qualified, which can improve the detection accuracy of the bubble type.

[0046] See also Figure 2 , Figure 2 1 is a flow chart of a specific embodiment of the bubble detection method provided by the present invention. In this embodiment, a bubble detection method is provided, and the bubble detection method includes the following steps.

[0047] S201: Train the initial detection network model to obtain a detection network model.

[0048] Specifically, the initial detection network model includes a first convolutional layer, a first pooling layer, and a first upsampling layer that are cascaded in sequence. The initial detection network model may also include a second convolutional layer, which is arranged between the first pooling layer and the first upsampling layer, and the second convolutional layer is cascaded with the upsampling layer. By setting the second convolutional layer, it is possible to meet the need to merge surrounding features and strengthen the expression of semantic features. The accuracy of the network is increased without increasing a lot of time. The initial detection network model is iteratively trained by selecting a first training sample set, and then the parameters of each layer are adjusted to obtain a detection network model. Among them, the initial detection network model is a YOLOv3-Tiny network deployed on the Caffe network framework, and it is lightweight improved. Please refer to Figure 3 , Figure 3 yes Figure 2 The flow chart of step S201 of the bubble detection method provided is a specific embodiment, which specifically includes the following steps.

[0049] S2011: Acquire a first training sample set, where the first training sample set includes a plurality of screen images, where the screen images are images annotated with real position information and real shapes of the dispensing areas.

[0050] Specifically, the first training sample set may include a plurality of screen images. The screen images are annotated with the real position information of the dispensing area and the image of the real shape formed by the distribution of the dispensing area. In an optional embodiment, the real shape of the dispensing area may be a circle or a rounded rectangle.

[0051] S2012: Detect the screen image through the initial detection network model to obtain the predicted position information and predicted shape of the glue dispensing area in the screen image.

[0052] Specifically, the first convolution layer extracts features from the screen image to obtain a first feature map; the first pooling layer compresses the first feature map to obtain a second feature map; the first upsampling layer amplifies the second feature map to obtain a third feature map; the third feature map is connected to the first feature map to output a glue spot area map corresponding to the screen image, and then the predicted position and predicted shape of the glue spot area are detected based on the glue spot area map.

[0053] S2013: Construct a first loss function through the predicted position information and the actual position information, the predicted shape and the actual shape of the dispensing area.

[0054] Specifically, the loss function is used to calculate the error between the predicted position information and the actual position information, and the predicted shape and the actual shape of the dispensing area. In a specific embodiment, the first loss function includes MSE (Mean Squared Error) Loss and BCE (Binary Cross-Entropy) Loss.

[0055] S2014: Iteratively train the initial detection network model using the first loss function to obtain a detection network model.

[0056] Specifically, the initial detection network model is iteratively trained by using the error values ​​between the predicted position information and the actual position information, and the predicted shape and the actual shape to obtain the detection network model.

[0057] In an optional embodiment, the result of the initial detection network model is back-propagated, and the weight of the initial detection network model is modified according to the loss value fed back by the first loss function. In an optional embodiment, the parameters in the initial detection network model can also be modified to achieve the training of the initial detection network model.

[0058] The screen image is input into the initial detection network model, and the initial detection network model predicts the position and shape in the screen image. When the error value between the predicted position information and the actual position information, and the predicted shape and the actual shape is less than a preset threshold value, which can be set by yourself, such as 1%, 5%, etc., the training of the initial detection network model is stopped and the detection network model is obtained.

[0059] S202: Train the initial semantic segmentation model to obtain a semantic segmentation model.

[0060] Specifically, the initial semantic segmentation model includes multiple upper convolutional layers, pooling layers, upsampling layers, and lower convolutional layers. The initial semantic segmentation model includes four lower convolutional layers, and the number of image channels of the four lower convolutional layers is 16, 32, 64, and 128 respectively. The initial semantic segmentation model is a U-net network deployed on the OpenVINO network framework. The U-net network is combined with the OpenVINO network framework to accelerate the network forward process. Please refer to Figure 4 , Figure 4 yes Figure 2 A flow chart of a specific embodiment of step S202 in the provided bubble detection method specifically includes the following steps.

[0061] S2021: Obtain a second training sample set, where the second training sample set includes multiple dispensing area images, where the dispensing area images are images marked with the actual position and actual type of the bubbles.

[0062] Specifically, the second training sample set may include multiple dispensing area images. The dispensing area images are annotated with images of the real positions and real types of bubbles. The dispensing area images may include one bubble or multiple bubbles. The size and shape of the bubbles may vary, with some bubbles occupying hundreds of pixels and some bubbles occupying only a few pixels.

[0063] S2022: Detect the dispensing area image through the initial semantic segmentation model to obtain the predicted position and predicted type of bubbles in the dispensing area image.

[0064] Specifically, the upper convolution layer encodes and downsamples the glue dispensing area maps of different sizes to obtain feature maps of different sizes; the pooling layer compresses the feature maps of different sizes; the upsampling layer decodes the feature map compressed by the pooling layer; the feature map decoded by the upsampling layer is connected to the feature map encoded by the upper convolution layer; the feature map of the upsampling layer connected to the upper convolution layer is extracted through the lower convolution layer to obtain a bubble mask map of the glue dispensing area map, and then the predicted position and predicted type of the bubble are detected based on the bubble mask map.

[0065] S2023: Construct a second loss function through the predicted position and true position, predicted type and true type of the bubble.

[0066] Specifically, a cross-entropy loss function is used to calculate the error value between the predicted position and the actual position, and the predicted type and the actual type of the bubble. In a specific embodiment, the second loss function is a cross-entropy loss Cross-entropyLoss.

[0067] S2024: Iteratively train the initial semantic segmentation model using a second loss function to obtain a semantic segmentation model.

[0068] Specifically, the initial semantic segmentation model is iteratively trained through the error values ​​between the predicted position and the true position, and the predicted type and the true type of the bubble to obtain the semantic segmentation model.

[0069] In an optional embodiment, the result of the initial semantic segmentation model is back-propagated, and the weight of the initial semantic segmentation model is modified according to the loss value fed back by the second loss function. In an optional embodiment, the parameters in the initial semantic segmentation model can also be modified to achieve the training of the initial semantic segmentation model.

[0070] The dispensing area image is input into the initial semantic segmentation model, which predicts the position and type of the bubbles in the dispensing area. When the error between the predicted position and the actual position, and the predicted type and the actual type of the bubble is less than a preset threshold, which can be set by yourself, such as 1%, 5%, etc., the training of the initial semantic segmentation model is stopped and the semantic segmentation model is obtained.

[0071] S203: Acquire the lens screen image to be detected.

[0072] Specifically, the image acquisition device acquires an image including a lens screen. The acquisition environment of the lens screen image is performed under a normal lighting environment. The lens screen image includes glue-dispensing areas distributed around. The lens screen is fixed to the lens barrel through the glue-dispensing areas. The image acquisition device may be a camera. In a specific embodiment, the image is acquired from a side of the lens screen away from the lens barrel.

[0073] S204: Detecting a dispensing area map of the lens screen image by using a detection network model.

[0074] Specifically, the lens screen image is detected by the trained detection network model to obtain a dispensing area map. The first convolution layer extracts features from the lens screen image to obtain a first feature map; the first pooling layer compresses the first feature map to obtain a second feature map; the first upsampling layer amplifies the second feature map to obtain a third feature map; the third feature map is connected with the first feature map to output a dispensing area map corresponding to the lens screen image. In a specific embodiment, the second convolution layer extracts features from the second feature map to obtain a fourth feature map; the first upsampling layer amplifies the fourth feature map to obtain a third feature map. The detection network model is a YOLOv3-Tiny network deployed on the Caffe network framework.

[0075] S205: Determine the shape of the glue dispensing area diagram and the position information of the glue dispensing area diagram in the lens screen image.

[0076] Specifically, a ray is drawn with the center of the lens screen image as the starting point, and the pixels on the ray within the set direction range are calculated; it is determined whether the difference between the previous pixel and the next pixel of the current pixel on the ray is greater than the preset value; if the difference between the previous pixel and the next pixel is greater than the preset value, the current pixel is determined to be the inner edge point. In order to reduce the amount of calculation, only the ray in the 180-degree direction needs to be calculated, and the other half is the same. Since the distance between the inner edge and the outer edge of the dispensing area is equal, when the inner edge point of the dispensing area is determined, the outer edge point of the dispensing area is also determined.

[0077] S206: Filter and fit multiple edge points according to the shape characteristics of the glue spotting area to obtain the inner edge shape and position of the glue spotting area.

[0078] Specifically, the glue dispensing area is divided into multiple areas in the horizontal / vertical direction; and the number of edge points in the multiple areas is counted. In an optional embodiment, the superellipse calculation formula can be used to calculate the edge points. Fit directly. However, the superellipse formula is too computationally intensive to fit, and high-order operations are time-consuming. Therefore, the rounded rectangle is split into four semicircles and four edges. Using the histogram method, the left side of the vertical axis is divided into several regions according to 10 pixels. The two regions with the largest number of edge points in the horizontal / vertical axis directions are selected, and the edge points in the two regions are respectively inversely synthesized into two straight lines. Then the four vertices where the two straight lines distributed horizontally and vertically intersect are calculated, and the points in the rectangle are counted. By judging the distance from the point to the four vertices, the point sets belonging to the four rounded corners can be divided. Then the least squares method is used to fit the circular parameters of the four rounded corners. The angle bisector is calculated using the two edges connected by the rectangle, and the centers of the four rounded circles are projected onto the angle bisector respectively. The parameters of the rounded corners are adjusted to achieve the best, and then a rounded rectangle is formed by fitting. Finally, the difference between the four rounded corners and the four edges is judged to determine whether the rounded corner is successfully fitted. If the rounded rectangle fails to fit, the fitted rectangle is used. According to the rules of the rectangle, it is judged whether it conforms to the rectangular shape. If it fails, the edge detection fails.

[0079] In another optional embodiment, according to the rounded rectangle calculation formula (xa) 2 +(yb) 2 =1 fits a circle. Since there are noise points at the edge points, the first fit of the circle edge will be too large, so the edge points outside the first fit circle are filtered out and the second fit is performed again. The second fit of the circle edge will have a better effect.

[0080] The inner edge shape of the glue-dispensing area is obtained by filtering and fitting multiple edge points through at least one of the above methods. It is judged whether the inner edge shape conforms to the preset shape; if the inner edge shape does not conform to the preset shape, the shape level is reduced or the edge points are subjected to noise reduction processing and the inner edge shape is fitted again. If the fitted inner edge shape conforms to the preset shape, the preset shape is determined to be the shape of the glue-dispensing area in the lens screen image to be detected.

[0081] S207: Extracting a dispensing area map in the lens screen image according to the shape and position information.

[0082] Specifically, a glue spotting area map is extracted from the lens screen image according to the position information of the glue spotting area and the shape of the glue spotting area, and other parts except the glue spotting area map are eliminated.

[0083] S208: Processing the dispensing area map using a semantic segmentation model to generate a bubble mask image.

[0084] Specifically, the glue dispensing area maps of different sizes are encoded and down-sampled through the upper convolution layer to obtain feature maps of different sizes; the pooling layer compresses the feature maps of different sizes; the upsampling layer decodes the feature map compressed by the pooling layer; the feature map decoded by the upsampling layer is connected to the feature map encoded by the upper convolution layer; the feature map of the upsampling layer connected to the upper convolution layer is extracted through the lower convolution layer to obtain a bubble mask map of the glue dispensing area map.

[0085] S209: Determine whether the distance between the bubble and the inner edge or the outer edge of the dispensing area is greater than a preset distance.

[0086] Specifically, the connected domain is used for detection, and each bubble in the dispensing area is extracted separately, and then the relevant data such as the center of mass, center and roundness of the bubble are calculated.

[0087] If the distance between the bubble and the inner edge or the outer edge of the dispensing area is greater than the preset distance, then jump directly to step S210. If the distance between the bubble and the inner edge or the outer edge of the dispensing area is not greater than the preset distance, then jump directly to step S211.

[0088] In a specific embodiment, it is determined whether the distance between the center of mass of the bubble and the inner edge or the outer edge is less than a threshold value. The threshold value can be set according to the needs. For example, the threshold value can be 20 pixels.

[0089] S210: The bubble belongs to the central bubble.

[0090] Specifically, if the distance between the bubble and the inner edge or the outer edge of the dispensing area is greater than a preset distance, the bubble is determined to be a central bubble. In a specific embodiment, if the distance between the center of mass of the bubble and the inner edge or the outer edge is not less than 20 pixels, the bubble can be confirmed to be a central bubble.

[0091] S211: The bubble is an edge bubble.

[0092] Specifically, if the distance between the bubble and the inner edge or the outer edge of the dispensing area is not greater than a preset distance, the bubble is determined to be an edge bubble. In a specific embodiment, if the distance between the center of mass of the bubble and the inner edge or the outer edge is less than 20 pixels, the bubble can be confirmed to be an edge bubble.

[0093] S212: Determine whether the bubble information conforms to the preset information.

[0094] Specifically, if the information of the bubble meets the preset information, the process directly jumps to step S214. If the information of the bubble does not meet the preset information, the process directly jumps to step S213.

[0095] When the bubble is a central bubble, in a specific embodiment, it is determined whether the width of the bubble is greater than the first threshold. If the width of the bubble is greater than the first threshold, it is directly jumped to step S213; if the width of the bubble is not greater than the first threshold, it is directly jumped to step S214. In another specific embodiment, it is determined whether the number of bubbles in the central area is greater than the second threshold. If the number of bubbles in the central area is greater than the second threshold, it is directly jumped to step S213; if the number of bubbles in the central area is not greater than the second threshold, it is directly jumped to step S214. In another specific embodiment, it is determined whether the area of ​​a single bubble is greater than a third threshold. If the area of ​​a single bubble is greater than the third threshold, it is directly jumped to step S213; if the area of ​​a single bubble is not greater than the third threshold, it is directly jumped to step S214. In another specific embodiment, it is determined whether the distance between the centroids of two bubbles is less than a fourth threshold. If the distance between the centroids of two bubbles is less than the fourth threshold, it is directly jumped to step S213; if the distance between the centroids of two bubbles is not less than the fourth threshold, it is directly jumped to step S214.

[0096] When the bubble is an edge bubble, in a specific embodiment, it is determined whether the width of the bubble is greater than the first threshold. If the width of the bubble is greater than the first threshold, it directly jumps to step S213; if the width of the bubble is not greater than the first threshold, it directly jumps to step S214. In another specific embodiment, it is determined whether the number of bubbles in the central area is greater than the second threshold. If the number of bubbles in the central area is greater than the second threshold, it directly jumps to step S213; if the number of bubbles in the central area is not greater than the second threshold, it directly jumps to step S214. In another specific embodiment, it is determined whether the length of a single bubble is greater than half of the length of the dispensing area. If the length of a single bubble is greater than half of the length of the dispensing area, it directly jumps to step S213; if the length of a single bubble is not greater than half of the length of the dispensing area, it directly jumps to step S214. In another specific embodiment, it is determined whether the area of ​​a single bubble is greater than a third threshold. If the area of ​​a single bubble is greater than the third threshold, it directly jumps to step S213; if the area of ​​a single bubble is not greater than the third threshold, it directly jumps to step S214. In another specific embodiment, it is determined whether the length of the continuous bubble is greater than half of the circumference of the glue dispensing area. If the length of the continuous bubble is greater than half of the circumference of the glue dispensing area, jump directly to step S213; if the length of the continuous bubble is not greater than half of the circumference of the glue dispensing area, jump directly to step S214.

[0097] S213: an alarm is sounded.

[0098] Specifically, if the information of the bubble does not conform to the preset information, an alarm is issued.

[0099] When the bubble belongs to the central bubble, when the width of the bubble is greater than the first threshold, an alarm is issued. When the number of bubbles in the central area is greater than the second threshold, an alarm is issued. When the area of ​​a single bubble is greater than the third threshold, an alarm is issued. When the distance between the centroids of two bubbles is less than the fourth threshold, an alarm is issued.

[0100] When the bubble is an edge bubble, an alarm is issued when the width of the bubble is greater than the first threshold. When the number of bubbles in the central area is greater than the second threshold, an alarm is issued. When the length of a single bubble is greater than half of the length of the dispensing area, an alarm is issued. When the area of ​​a single bubble is greater than the third threshold, an alarm is issued. When the length of continuous bubbles is greater than half of the perimeter of the dispensing area, an alarm is issued.

[0101] S214: Output OK.

[0102] Specifically, if the information of the bubble meets the preset information, OK is output, indicating that the bubble meets the requirements.

[0103] When the bubble belongs to the central bubble, if the width of the bubble is not greater than the first threshold, then output OK. When the number of bubbles in the central area is not greater than the second threshold, then output OK. When the area of ​​a single bubble is not greater than the third threshold, then output OK. When the distance between the centroids of two bubbles is not less than the fourth threshold, then output OK.

[0104] When the bubble is an edge bubble, an alarm is issued when the width of the bubble is not greater than the first threshold. When the number of bubbles in the central area is not greater than the second threshold, an alarm is issued. When the length of a single bubble is not greater than half of the length of the dispensing area, an alarm is issued. When the area of ​​a single bubble is not greater than the third threshold, an alarm is issued. When the length of continuous bubbles is not greater than half of the perimeter of the dispensing area, an alarm is issued.

[0105] In this embodiment, the glue spotting area map in the lens screen image is obtained by extracting features from the acquired lens screen image to be detected; the glue spotting area map is then processed using a semantic segmentation model to generate a bubble mask image; and the bubble mask image is used to determine whether the bubble is qualified. The bubble detection method of the present application can automatically detect and extract the glue spotting area map without physical manual work, avoiding manual intervention in the detection results, which can improve both detection accuracy and detection efficiency; and the bubble mask image generated by the semantic segmentation model is used to determine whether the bubble is qualified, which can improve the detection accuracy of the bubble type.

[0106] See also Figure 5 , Figure 5 1 is a schematic block diagram of an embodiment of a terminal provided by the present invention. Figure 5As shown, the terminal 70 in this embodiment includes: a processor 71, a memory 72, and a computer program stored in the memory 72 and executable on the processor 71. When the computer program is executed by the processor 71, the above-mentioned trajectory tracking control method is implemented. To avoid repetition, they are not described one by one here.

[0107] See also Figure 6 , Figure 6 It is a schematic block diagram of an embodiment of a computer-readable storage medium provided by the present invention.

[0108] A computer-readable storage medium 90 is also provided in an embodiment of the present application. The computer-readable storage medium 90 stores a computer program 901. The computer program 901 includes program instructions. The processor executes the program instructions to implement any target tracking method provided in the embodiment of the present application.

[0109] The computer-readable storage medium 90 may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium 90 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0110] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A bubble detection method, characterized in that: The bubble detection method comprises: Acquire a lens screen image to be detected; the lens screen image includes a glue-dispensing area; the lens screen is fixed on the lens barrel through the glue-dispensing area; Extracting a dispensing area map from the lens screen image; The dispensing area map is processed using a semantic segmentation model to generate a bubble mask image; Determine whether each of the bubbles in the dispensing area map is qualified by using the bubble mask image; Wherein, after the step of extracting the glue dispensing area in the lens screen image, the method further includes: Draw a ray with the center of the lens screen image as the starting point, and calculate the pixels on the ray within a set direction range; Determine whether the difference between the previous pixel and the next pixel of the current pixel on the ray is greater than a preset value; If the difference between the previous pixel and the next pixel is greater than the preset value, determining that the current pixel is an edge point; Wherein, the step of extracting the dispensing area map in the lens screen image includes: Detecting a dispensing area map of the lens screen image by using a detection network model; Determine the shape of the glue dispensing area map and the position information of the glue dispensing area map in the lens screen image; Extracting the glue dispensing area map in the lens screen image according to the shape and the position information; After the step of extracting the dispensing area in the lens screen image, the following steps are further included: Divide the dispensing area into a plurality of areas in the horizontal and vertical directions respectively; Counting the number of edge points in the plurality of regions; Select two regions with the largest number of edge points in the region corresponding to the horizontal axis and the vertical axis, respectively, and inversely synthesize the edge points in the two regions into two straight lines respectively; Based on the intersection of the two straight lines fitted in the horizontal axis direction and the two straight lines fitted in the vertical axis direction, the inner edge shape and position of the glue dispensing area are determined.

2. The bubble detection method according to claim 1, characterized in that: The detection network model includes a first convolutional layer, a first pooling layer, and a first upsampling layer which are cascaded in sequence; The step of detecting the dispensing area map of the lens screen image by detecting the network model comprises: The first convolutional layer extracts features from the lens screen image to obtain a first feature map; The first pooling layer compresses the first feature map to obtain a second feature map; The first upsampling layer amplifies the second feature map to obtain a third feature map; The third characteristic map is connected with the first characteristic map to output the glue dispensing area map corresponding to the lens screen image.

3. The bubble detection method according to claim 2, characterized in that: The detection network model further includes a second convolutional layer, which is arranged between the first pooling layer and the first upsampling layer, and the second convolutional layer is cascaded with the upsampling layer; The step of amplifying the second feature map by the first upsampling layer to obtain a third feature map comprises: The second convolutional layer performs feature map extraction on the second feature map to obtain a fourth feature map; The first upsampling layer amplifies the fourth feature map to obtain the third feature map.

4. The bubble detection method according to claim 1, characterized in that: The detection network model is a YOLOv3-Tiny network deployed on the Caffe network framework.

5. The bubble detection method according to claim 1, characterized in that: The semantic segmentation model includes multiple up-convolution layers, pooling layers, up-sampling layers and down-convolution layers; The step of using the semantic segmentation model to process the dispensing area map to generate a bubble mask image specifically includes: The upper convolution layer encodes and downsamples the dispensing area maps of different sizes to obtain feature maps of different sizes; The pooling layer compresses the feature maps of different sizes; The upsampling layer decodes the feature map compressed by the pooling layer; Connecting the feature map obtained by decoding the upsampling layer to the feature map obtained by encoding the upper convolutional layer; The feature map of the upsampling layer connected to the upper convolution layer is subjected to feature extraction through the lower convolution layer to obtain the bubble mask map of the dispensing area map.

6. The bubble detection method according to claim 5, characterized in that: The semantic segmentation model includes four lower convolutional layers, and the numbers of image channels of the four lower convolutional layers are 16, 32, 64, and 128 respectively.

7. The bubble detection method according to claim 5, characterized in that: The semantic segmentation model is a U-net network deployed on the OpenVINO network framework.

8. The bubble detection method according to claim 1, characterized in that: After the step of extracting the dispensing area in the lens screen image, the following steps are further included: Determining whether the inner edge shape conforms to a preset shape; If the inner edge shape does not conform to the preset shape, the shape level is reduced or the edge points are subjected to noise reduction processing and then the inner edge shape is fitted again.

9. The bubble detection method according to claim 8, characterized in that: The step of determining whether each bubble in the dispensing area map is qualified by using the bubble mask image specifically includes: Determine whether the distance between the bubble and the inner edge or the outer edge of the dispensing area is greater than a preset distance; If it is greater than, the bubble belongs to the central bubble; If it is not greater than, the bubble is an edge bubble.

10. The bubble detection method according to claim 9, characterized in that: The step of determining whether each bubble in the dispensing area map is qualified by using the bubble mask image specifically includes: Determine whether the information of the bubble meets the preset information; If the information of the bubble does not conform to the preset information, an alarm is issued.

11. The bubble detection method according to claim 1, characterized in that: The step of obtaining the lens screen image to be detected includes: The initial detection network model is trained to obtain the detection network model.

12. The bubble detection method according to claim 11, characterized in that: The step of training the initial detection network model to obtain the detection network model comprises: Acquire a first training sample set, wherein the first training sample set includes a plurality of screen images, wherein the screen images are images with real position information and real shapes of the dispensing areas marked; Detecting the screen image through the initial detection network model to obtain predicted position information and predicted shape of the glue dispensing area in the screen image; Constructing a first loss function through the predicted position information and the actual position information, the predicted shape and the actual shape of the dispensing area; The detection network model is obtained by iteratively training the initial detection network model using the first loss function.

13. The bubble detection method according to claim 1, characterized in that: The step of obtaining the lens screen image to be detected includes: The initial semantic segmentation model is trained to obtain the semantic segmentation model.

14. The bubble detection method according to claim 13, characterized in that: The step of training the initial semantic segmentation model to obtain the semantic segmentation model comprises: Acquire a second training sample set, wherein the second training sample set includes a plurality of dispensing area images, wherein the dispensing area images are images marked with the actual positions and types of bubbles; Detecting the dispensing area image by using the initial semantic segmentation model to obtain the predicted position and predicted type of the bubble in the dispensing area image; Constructing a second loss function through the predicted position and the true position of the bubble, the predicted type and the true type; The initial semantic segmentation model is iteratively trained using the second loss function to obtain the semantic segmentation model.

15. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to execute program data to implement the steps in the bubble detection method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the bubble detection method according to any one of claims 1 to 14 are implemented.

Images