Method, device, equipment and storage medium for determining glue defects

Abstract

The present application discloses a method, device, equipment and storage medium for determining glue defects, and relates to the field of artificial intelligence technology. The method comprises: obtaining a first point image of a sample, wherein the first point image shows some components in the sample and the glue between some components, and the first point image is an image taken at the first point of the sample; determining the glue area where the glue is located in the first point image according to the feature information of the first point image extracted by the semantic segmentation model, and the feature information is the deep learning information extracted by the semantic segmentation model; determining the glue defect existing at the first point of the sample according to the distribution of the glue area. The glue area determined by the feature information has higher accuracy and is closer to the actual situation. Therefore, the glue defect determined based on the more realistic distribution of the glue area is also more accurate.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for determining glue defects. Background Art

[0002] With the development of artificial intelligence technology, cameras have gradually become an important role in the field of artificial intelligence. Generally speaking, a camera module has multiple different subdivided components, and these multiple subdivided components are connected by glue. Therefore, whether there are defects in the glue between the subdivided components is the key to judging whether a camera module is a good product. Generally, the glue of the camera module is photographed, and the area where the glue is located is determined based on the captured image.

[0003] In the related art, the glue area of ​​the image is determined by binarization, that is, the values ​​of the pixels corresponding to the glue part and the values ​​of the pixels corresponding to other parts are set to different values, so as to obtain the glue area. According to the determined glue area, the defects are judged.

[0004] However, in the above-mentioned related technologies, due to the problem of shooting angle and the possibility of reflection of the glue itself, the accuracy of the determined glue defects is not high, and there may be a situation that is greatly different from the actual situation. Summary of the invention

[0005] The embodiment of the present application provides a method, device, equipment and storage medium for determining glue defects. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a method for determining a glue defect is provided, the method comprising:

[0007] Acquire a first point image of the sample, wherein the first point image shows some components in the sample and glue between the some components, and the first point image is an image obtained by photographing the first point of the sample;

[0008] Determine a glue area where the glue in the first point image is located according to feature information of the first point image extracted by the semantic segmentation model, wherein the feature information is deep learning information extracted by the semantic segmentation model;

[0009] According to the distribution of the glue area, the glue defect existing at the first point of the sample is determined.

[0010] According to one aspect of an embodiment of the present application, a device for determining glue defects is provided, the device comprising:

[0011] An image acquisition module, used for acquiring a first point image of the sample, wherein the first point image shows some components in the sample and glue between the components, and the first point image is an image obtained by photographing the first point of the sample;

[0012] A region determination module, configured to determine a glue region where the glue in the first point image is located according to feature information of the first point image extracted by a semantic segmentation model, wherein the feature information is deep learning information extracted by the semantic segmentation model;

[0013] The defect determination module is used to determine the glue defect existing in the sample at the first point according to the distribution of the glue area.

[0014] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above method.

[0015] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above method.

[0016] According to one aspect of the embodiments of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method.

[0017] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0018] First, an image of the sample at the point (which may be called the first point) where glue defects need to be detected is obtained (also referred to as the first point image), and then the feature information of the image is obtained through the semantic segmentation model, so that the glue area of ​​the first point image can be determined. After the glue area is determined, the glue defects of the sample at the first point are determined based on its distribution. The technical solution provided in the embodiment of the present application uses a semantic segmentation model to extract the deep learning information of the first point image, and determines the glue area with the help of the deep learning information. The determination of the glue area is not limited by the reflection of the glue, and the glue area determined by the feature information has higher accuracy and is closer to the actual situation. Therefore, based on the distribution of the more realistic glue area, the determined glue defects are also more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a method for determining glue defects provided by an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a method for determining sample defects provided by an embodiment of the present application;

[0022] Figure 4 is a flow chart of a method for determining glue defects provided by an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a glue area provided by an embodiment of the present application;

[0024] Figure 6 is a flow chart of a method for determining glue defects provided by another embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a semantic segmentation model provided by an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a foreground area provided by an embodiment of the present application;

[0027] Fig. 9 is a flow chart of a method for determining glue defects provided by another embodiment of the present application;

[0028] Fig.10 is a schematic diagram of fitting edge points provided by an embodiment of the present application;

[0029] Fig.11 is a block diagram of a device for determining glue defects provided by one embodiment of the present application;

[0030] Fig.12 is a block diagram of a device for determining glue defects provided by another embodiment of the present application;

[0031] Fig.13 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0033] Before introducing the technical solution of this application, some background technical knowledge involved in this application is first introduced and explained. The following related technologies can be combined arbitrarily with the technical solution of the embodiment of this application as optional solutions, and they all belong to the protection scope of the embodiment of this application. The embodiment of this application includes at least part of the following contents.

[0034] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0035] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes natural language processing technology, machine learning / deep learning and other major directions.

[0036] Computer vision technology (CV for short) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, virtual reality, augmented reality, simultaneous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0037] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0038] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0039] The solutions provided in the embodiments of the present application involve technologies such as artificial intelligence computer vision and machine learning, which are specifically illustrated by the following embodiments.

[0040] Before introducing the technical solution of the present application, some terms involved in the present application are explained. The following related explanations can be combined arbitrarily with the technical solution of the embodiment of the present application as optional solutions, and they all belong to the protection scope of the embodiment of the present application. The embodiment of the present application includes at least part of the following contents.

[0041] Deep learning: The concept of deep learning originates from the study of artificial neural networks. A multi-layer perceptron with multiple hidden layers is a deep learning structure. Deep learning discovers distributed feature representations of data by combining low-level features to form more abstract high-level representations of attribute categories or features.

[0042] Convolutional Neural Networks (CNN): A type of feedforward neural network (FNN) that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can classify input information in a translation-invariant manner according to their hierarchical structure. Generally, the basic structure of CNN includes two layers. One is the feature extraction layer. The input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local feature is extracted, its positional relationship with other features is also determined; the second is the feature mapping layer. Each computing layer of the network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. CNN consists of an input layer, a convolution layer, an activation function, a pooling layer, and a fully connected layer.

[0043] Neck glue: The camera module is assembled from multiple different subdivided components, which are connected by glue. When the glue is in the neck part of the camera module, the glue is called neck glue.

[0044] VGG (Visual Geometry Group) 16 network: The 16 in the VGG16 network means that it contains 16 convolutional layers and fully connected layers, with a total of 138 million parameters. The VGG16 network is a type of neural network, but at the same time the VGG16 network simplifies the neural network, and the VGG16 network uses a unified convolution kernel size: 3*3.

[0045] Mask image: An image after a mask operation. The mask operation of an image refers to recalculating the value of each pixel in the image through a mask kernel operator. The mask kernel operator describes the degree of influence of the domain pixel on the new pixel value, and performs a weighted average of the pixels according to the weight factor in the mask operator. Image mask operations are often used in areas such as image smoothing, edge detection, and feature analysis.

[0046] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include: a terminal device 10 and a server 20.

[0047] The terminal device 10 includes but is not limited to mobile phones, tablet computers, intelligent voice interaction devices, game consoles, wearable devices, multimedia playback devices, PCs (Personal Computers), vehicle terminals, smart home appliances and other electronic devices. The client of the target application can be installed in the terminal device 10.

[0048] In an embodiment of the present application, the above-mentioned target application can be any application that can provide defect detection. Typically, the application is a glue defect application. Of course, in addition to glue defect applications, other types of applications can also provide glue defect determination services. For example, quality inspection applications, product application applications, browser applications, virtual reality (Virtual Reality, VR for short) applications, augmented reality (Augmented Reality, AR for short) applications, etc., the present application embodiment does not limit this. In addition, for different applications, the defects that need to be detected are different, so the defects can be glue defects or other defects. For example, whether the components are complete, whether the parts are missing, etc., the present application embodiment does not limit this. Optionally, a client of the above-mentioned application is running in the terminal device 10.

[0049] The server 20 is used to provide background services for the client of the target application in the terminal device 10. For example, the server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but is not limited to these.

[0050] The terminal device 10 and the server 20 can communicate with each other via a network, which can be a wired network or a wireless network.

[0051] The method provided in the embodiment of the present application may be performed by a computer device. The computer device may be any electronic device that has the ability to store and process data. For example, the computer device may be Figure 1 The server 20 in may be Figure 1 The terminal device 10 in the example may also be another device other than the terminal device 10 and the server 20.

[0052] Please refer to Figure 2 , which shows a schematic diagram of a method for determining glue defects provided by an embodiment of the present application.

[0053] First, take a picture of the glue at the first point of the camera subdivision component. Figure 2The first point image in the figure, where 100 is the location of the neck glue of the camera, it is not difficult to find that the glue photographed has a serious reflection problem. Therefore, if the relevant technology is used to binarize the image according to the color characteristics of the neck glue to extract the glue area of ​​the neck glue, the neck glue will have such a problem. Figure 2 The reflection problem shown makes it difficult to extract the complete glue area of ​​the neck glue by binarization, which brings great trouble to the subsequent determination of glue defects based on the glue area.

[0054] The technical solution provided by the embodiment of the present application extracts the feature information of the first point image, wherein the feature information is deep learning information, and determines the glue area of ​​the first point image based on the feature information. Then determine different defect types according to the glue area. In the related art, there is also a method of extracting the features of glue and then judging the defect type. However, in the related art, samples with defects are usually used to train the model, so that when the first point image of the sample is input, the defect type can be determined according to the image. However, this method in the related art, since there are relatively few samples with defects in actual industrial production, the training samples of the model of the related art are also relatively few. In the case of fewer training samples, the training effect of the model is not very ideal. The detection effect of the glue defect based on the first point image determined by the model is not very ideal. In addition, there is also an automated visual AI defect detection method to detect defects, but considering that most glue defects are caused by unreasonable distribution areas of the glue itself, such as overflow, leakage, and breakage, the glue areas are irregular, so it will bring great challenges to automated visual AI defect detection.

[0055] The difference is that the technical solution provided in the embodiment of the present application is to train the model through a large number of first-point images of genuine products. The training semantic segmentation model 110 can extract the complete glue area. Since there are a large number of genuine products in actual industrial production, the training samples of the model of the present application are very rich. Therefore, the model trained based on the rich training samples has a relatively better ability to extract the glue area. Therefore, the technical solution provided in the embodiment of the present application, the glue area 120 of the first point extracted is closer to the actual glue situation. Based on the glue area 120, the glue defect can be further determined.

[0056] Therefore, the technical solution provided in the embodiment of the present application extracts the deep learning information of the first point according to the semantic segmentation model, and then determines the glue area in the first point image of the sample, thereby determining the possible glue defects. The final glue defect detection effect is better, closer to the actual situation, and has higher accuracy.

[0057] Please refer to Figure 3 , which shows a schematic diagram of a method for determining sample defects provided by an embodiment of the present application.

[0058] The technical solution provided in the embodiment of the present application can be used as part of the quality inspection capability and applied to the camera module quality inspection in the industrial AI quality inspection platform to analyze and determine the abnormal neck glue, and give a defect judgment of the sample together with the analysis results of other points.

[0059] Figure 3 This is a possible application scenario of the present application, that is, the determination of the final defects of samples in industry. Here, the final defects are determined by combining the defects of multiple points. Figure 3 The camera shown in the figure includes multiple components: a light hole, a lens barrel, a lens, petals, a lens holder, a flexible circuit board, and a connector. The glue at the connection between the lens holder and the flexible circuit board can be called neck glue. There is no restriction on the points in this application. All positions that need to be detected can be called points, such as Figure 3 The sub-images (a), (b), and (c) in the figure are three point images obtained by shooting at three different points. Among them, sub-image (b) is the point where the neck glue is located. Therefore, neck glue detection is performed on sub-image (b), that is, the glue defect at this position is determined using the technical solution provided in the embodiment of the present application.

[0060] Therefore, the technical solution provided in the embodiment of the present application can be used to detect defects at each point, and other defect detection methods can also be used, which is not limited in the present application. Finally, it is determined whether the sample is a bad sample based on the defect conditions of multiple points. If it is a good sample, it is determined as a good product. If it is a bad sample, it is considered to be a sample that needs to be destroyed or destroyed, and the yield rate of a batch of products can be counted.

[0061] Please refer to Figure 4 , which shows a flow chart of a method for determining glue defects provided by an embodiment of the present application. The execution subject of each step of the method may be Figure 1 The terminal device 10 or server 20 in the implementation environment of the solution shown. In the following method embodiment, for the convenience of description, only the execution subject of each step is introduced as "computer device". The method may include at least one of the following steps (320-360):

[0062] Step 320, obtaining a first point image of the sample, wherein the first point image shows some components in the sample and glue between some components, and the first point image is an image obtained by photographing the first point of the sample.

[0063] Sample: A product to be tested, in which possible defects are not yet known. This application does not limit the type of sample, which can be a camera module or other products, such as a robot module, a microscope module, a sensor module, and other products that need to be connected with glue. The technical solution provided in the embodiment of this application can be used to detect glue defects.

[0064] First point image: an image taken at the first point of the sample. The first point can be manually set in advance, or it can be determined based on the specific circumstances of the sample, and this application does not limit this. The first point includes but is not limited to the location of the neck glue. In some embodiments, the first point image includes but is not limited to grayscale images, color images, and depth images. In some embodiments, the first point image is a grayscale image, and each pixel in the image corresponds to a grayscale value. In some embodiments, the first point image is a color image, and each pixel in the image corresponds to the value of the three channels of red, green, and blue. In some embodiments, the first point image is a depth image, and each pixel in the image corresponds to a depth value. This application does not limit the type of the first point image.

[0065] In some embodiments, the shooting angle of the first point image should be fixed and standard, that is, the sample shot can be horizontal. Therefore, if the shooting angle is wrong, the first point image shot may not be horizontal, which will also cause the glue area subsequently determined to be non-horizontal, and there may be tilt and other situations that are inconsistent with the actual situation. Therefore, in the case of the non-horizontal first point image caused by the shooting angle problem, the final determined glue area can be closer to the real glue area by reshooting, rotating and correcting the first point image shot, rotating and correcting the determined glue area, etc. Therefore, the problem of inaccurate detection of glue defects caused by shooting angle problems can be avoided.

[0066] Step 340, determining a glue area where the glue in the first point image is located according to feature information of the first point image extracted by the semantic segmentation model, wherein the feature information is deep learning information extracted by the semantic segmentation model.

[0067] Semantic segmentation model: The semantic segmentation model in the embodiments of the present application is a type of neural network model, and the present application does not limit the specific structure of the semantic segmentation model. There is no limitation on the number of convolutional layers and pooling layers included in the semantic segmentation model, and there is no limitation on the size of the convolution kernel. A model that can extract the glue area through feature information can be considered as the semantic segmentation model in the embodiments of the present application. Generally speaking, the semantic segmentation model is pre-trained. In some embodiments, the semantic segmentation model is pre-trained using multiple labeled first point images. Figure 2 The semantic segmentation model 110 shown in , calculates the value of the cross entropy loss function according to the difference between the predicted result (which can also be considered as the predicted feature map) and the label, and then adjusts the parameters of the model. Optionally, the semantic segmentation model includes but is not limited to deeplab v3, cascaded mask rcnn and other models.

[0068] Feature information: used to characterize the features of the first point image. The form of the feature information is not limited here, and it can be in the form of a feature vector or a distance. In addition, the present application does not limit the type of feature information, and it can be detail feature information, semantic feature information, or detail feature information superimposed on semantic feature information. See the following embodiments for details, which will not be elaborated here.

[0069] Glue area: The first point image is predicted by the semantic segmentation model to obtain the glue area. The glue area in the embodiment of the present application refers to the glue area predicted based on the image rather than the area where the actual glue is located. In some embodiments, the glue area is highlighted according to the result of the model prediction.

[0070] Step 360: Determine the glue defect existing at the first point of the sample according to the distribution of the glue area.

[0071] Glue defect: If the glue of the sample is insufficient compared with the glue of the genuine product or compared with itself, it is considered to be a glue defect. The present application does not limit the type of glue defect, which can be at least one of insufficient glue, glue leakage, glue breakage, and glue overflow. The distribution of the glue area can be considered to be the size of the glue area, whether the glue area is complete, whether the glue area is symmetrical at both ends, etc. All situations that can be used to determine glue defects can be considered as the distribution of the glue area.

[0072] refer to Figure 5, which shows a schematic diagram of the glue area provided by an embodiment of the present application. Wherein sub-image (a) and sub-image (b) can be considered as the first point image, in which the circled position represents the position of the glue. Sub-image (c) and sub-image (d) are the glue areas determined based on sub-image (a) and sub-image (b). Sub-image (c) can be considered as the glue area of ​​the genuine product, and the glue area in sub-image (d) is compared with the glue area in sub-image (c). The widths of the left and right ends of the glue area in sub-image (d) are not consistent, so it can be considered that the glue in sub-image (b) has glue defects.

[0073] The technical solution provided in the embodiment of the present application first obtains an image (also referred to as the first point image) of the sample at the point (also referred to as the first point) where glue defects need to be detected, and then obtains the feature information of the image through a semantic segmentation model, thereby determining the glue area of ​​the first point image. After determining the glue area, the glue defects of the sample at the first point are determined based on their distribution. The technical solution provided in the embodiment of the present application uses a semantic segmentation model to extract the deep learning information of the first point image, and determines the glue area with the help of the deep learning information. The determination of the glue area is not limited by the reflection of the glue, and the glue area determined by relying on the feature information has higher accuracy and is closer to the actual situation. Therefore, based on the distribution of the more realistic glue area, the determined glue defects are also more accurate.

[0074] Please refer to Figure 6 , which shows a flow chart of a method for determining glue defects provided by another embodiment of the present application. The execution subject of each step of the method may be Figure 1 The terminal device 10 or server 20 in the implementation environment of the solution shown. In the following method embodiment, for the convenience of description, only the execution subject of each step is introduced as "computer device". The method may include at least one of the following steps (320-360):

[0075] Step 320, obtaining a first point image of the sample, wherein the first point image shows some components in the sample and glue between some components, and the first point image is an image obtained by photographing the first point of the sample.

[0076] Step 342: The semantic segmentation model includes a feature extraction sub-model and a prediction sub-model. The feature extraction sub-model is used to obtain feature information of the first point image.

[0077] Feature extraction submodel: used to extract features of the first point image. In some embodiments, the feature extraction submodel is a VGG16 network. Optionally, the feature extraction submodel includes multiple convolutional layers and multiple pooling layers.

[0078] Prediction sub-model: used to determine the prediction result of the first point image according to the feature information of the first point image. The prediction result here can also be a prediction feature map, or the probability that each pixel belongs to different categories. The category corresponding to the maximum probability is determined as the prediction result of the current pixel. Optionally, the prediction sub-model is a prediction head module, including a convolution layer.

[0079] In some embodiments, the shallow features and deep features of the first point image are extracted through the feature extraction sub-model, the shallow features are used to characterize the detail features of the first point image, and the deep features are used to characterize the semantic features of the first point image. The shallow features are fused with the deep features to obtain feature information of the first point image. In some embodiments, the detail features contain more pixel information of the first point image, and the semantic features contain more deep abstract information, which can be used to characterize the category information.

[0080] refer to Figure 7 , which shows a schematic diagram of a semantic segmentation model provided by an embodiment of the present application, wherein the semantic segmentation model 700 includes a feature extraction sub-model, a prediction sub-model, and a decoder sub-model. Among them, the feature extraction sub-model includes a plurality of convolutional layers. Optionally, there are 5 convolutional layers. From left to right, they are the first convolutional layer to the fifth convolutional layer. In some embodiments, the output of the third convolutional layer is considered to be a shallow feature, and the output of the fifth convolutional layer is considered to be a deep feature. It is not difficult to find that the shallow feature is closer to the first point image, and therefore has more detail information, so it is used to characterize the detail features of the first point image. The deep feature is far away from the first point image, so there is less detail information in the deep feature, and more semantic information, that is, the deep feature is used to characterize the semantic features of the image, that is, the features that characterize the category to which each pixel in the image belongs. After the shallow feature passes through two convolutional layers, the deep feature is obtained, that is, the shallow feature is downsampled to obtain the deep feature, and there is a difference in the size of the two. Therefore, when fusing shallow features and deep features, the deep features can be upsampled to restore them to the same size as the shallow features, and then the features are fused. In some embodiments, the decoder submodule is used to fuse the features. In some embodiments, the fusion of features can be the superposition of dimensions, or it can be obtained after specific operations, including but not limited to addition, subtraction, multiplication and division.

[0081] Step 344: Obtain a mask image corresponding to the first point image according to the feature information through the prediction sub-model.

[0082] In some embodiments, two types of objects are set in advance in the model, the glue area is 1, and other components or background areas are 0. When the prediction sub-model detects the fused features, it determines the probability of the category corresponding to each pixel. When the probability of belonging to the glue area is greater than the probability of belonging to the background area, the value corresponding to the pixel is determined to be 1, otherwise, the value corresponding to the pixel is determined to be 0. Therefore, a mask image can be obtained, in which different pixels in the image correspond to different values, and the value indicates the category to which the pixel belongs.

[0083] Step 346, determining the glue area where the glue in the first point image is located according to the area where the pixel points whose values ​​meet the conditions in the mask image are located.

[0084] In the present application, the condition is not limited, and the value may be equal to a preset value, or may be greater than a threshold value, etc. In some embodiments, the area where the pixel point with a value of 1 is located is determined as the glue area where the glue is located in the first point image.

[0085] In some embodiments, the area where the pixels in the mask image that meet the conditions are located is determined as the foreground area, which includes at least one closed area. The closed area with the largest area in the foreground area is determined as the glue area where the glue is located.

[0086] refer to Figure 8 , which shows a schematic diagram of a foreground area provided by an embodiment of the present application. Optionally, the area where the pixel points whose values ​​satisfy the conditions in the mask image are located is determined as the foreground area, such as Figure 8 The foreground region includes at least one closed region, and the region with the largest area in the closed region, that is, region 81, is determined as the glue region where the glue is located. In some embodiments, the glue region in the foreground region can be determined according to the maximum connected domain extraction algorithm.

[0087] Step 360: Determine the glue defect existing at the first point of the sample according to the distribution of the glue area.

[0088] The technical solution provided in the embodiment of the present application, by subdividing the semantic segmentation model into a feature extraction sub-model and a prediction sub-model, uses the feature extraction sub-model to extract feature information, and uses the prediction sub-model to predict the category to which the pixel point belongs. This can make the principle of the model clearer, determine the division of labor of the model, and thus better determine the glue area based on the feature information.

[0089] The technical solution provided in the embodiment of the present application can extract shallow features and deep features of the first point image through a feature extraction sub-model, and can combine detail features with semantic features, so that the model prediction results can be more accurate and closer to the actual situation with as little loss of detail features as possible.

[0090] The technical solution provided in the embodiment of the present application determines the glue area where the glue is located by the largest closed area in the foreground area, and excludes some closed areas with smaller areas, which can effectively reduce some errors. Of course, when there is more than one closed area, it means that the glue at the first point has certain defects, such as broken glue. The specific determination of glue defects is shown in the following embodiments. Therefore, by extracting the closed area with the largest area, the error can be reduced to a certain extent, paving the way for subsequent work and reducing the amount of data processing to a certain extent.

[0091] Please refer to Fig. 9 , which shows a flow chart of a method for determining glue defects provided by another embodiment of the present application. The execution subject of each step of the method may be Figure 1 The terminal device 10 or server 20 in the implementation environment of the solution shown. In the following method embodiment, for the convenience of description, only the execution subject of each step is introduced as "computer device". The method may include at least one of the following steps (320-366):

[0092] Step 320, obtaining a first point image of the sample, wherein the first point image shows some components in the sample and glue between some components, and the first point image is an image obtained by photographing the first point of the sample.

[0093] Step 340, determining a glue area where the glue in the first point image is located according to feature information of the first point image extracted by the semantic segmentation model, wherein the feature information is deep learning information extracted by the semantic segmentation model.

[0094] Step 362, determining a plurality of edge points of the glue area, wherein the plurality of edge points constitute the outline of the glue area.

[0095] In some embodiments, the mask image is scanned horizontally, and the upper and lower endpoints are used as the contour points of the glue area. In some embodiments, if the mask image is not horizontal, it needs to be rotated and corrected.

[0096] Step 364 , predicting a plurality of fitting edge points of the glue based on the plurality of edge points, wherein the plurality of fitting edge points are closer to the edge of the glue than the plurality of edge points.

[0097] In some embodiments, since there may be multiple abnormal points among multiple edge points, which can also be understood as noise points, the edge points may be unreliable. In this case, the multiple edge points need to be fitted so that the fitted edge points are closer to the actual situation.

[0098] In some embodiments, the fitting edge points are predicted using Huber regression. Huber regression is a type of linear regression, but linear regression has been changed. In the related art, linear regression uses the MSE (Mean Square Error) loss function, which is not robust to abnormal points. When there are many noise points, the fitting results will have large deviations. Since the neck glue itself has irregular shapes, this loss function that is not robust to noise can easily lead to large regression errors. Therefore, the technical solution provided in the present application adopts Huber regression. This regression method uses the MAE (Mean Absolute Error) loss function when the error between the predicted result (fitting edge point) and the real point (edge ​​point) is large, thereby reducing the gradient signal caused by the abnormal point, so that the optimization result is closer to the optimal value, that is, the predicted multiple fitting edge points are closer to the actual glue edge. The specific formula is as follows:

[0099]

[0100] Wherein, L represents the value of the loss function, δ is an adjustable parameter, the larger the value, the closer it is to the MSE loss function, and the smaller the value, the closer it is to the MAE loss function, and the value depends on the actual proportion of outliers. f(x) represents the predicted fitting edge point, and y represents the edge point. This application does not limit the linear regression method, and any linear regression method that can predict the fitting edge point can be used in the technical solution provided in this application.

[0101] Step 366, determining the glue defect existing at the first point of the sample according to the distribution of the multiple fitting edge points.

[0102] In some embodiments, the glue defect existing at the first point of the sample is determined based on the glue area enclosed by a plurality of fitting edge points.

[0103] If the ratio of the area of ​​the region enclosed by the multiple fitting edge points to the glue area of ​​the genuine product at the first point is greater than the third value, it is determined that the sample has overflow glue at the first point; if the ratio of the area of ​​the region enclosed by the multiple fitting edge points to the glue area of ​​the genuine product at the first point is less than the fourth value, it is determined that the sample has less glue at the first point; wherein the third value is greater than the fourth value. In some embodiments, the third value is 1.5, that is, the ratio of the area of ​​the region enclosed by the multiple fitting edge points to the glue area of ​​the genuine product at the first point is greater than 1.5, and the area of ​​the region enclosed by the multiple fitting edge points is at least 50% larger than the glue area of ​​the genuine product at the first point, indicating that the sample has overflow glue at the first point. In some embodiments, the fourth value is 0.5, that is, the ratio of the area of ​​the region enclosed by the multiple fitting edge points to the glue area of ​​the genuine product at the first point is less than 0.5, and the area of ​​the region enclosed by the multiple fitting edge points is at least 50% smaller than the glue area of ​​the genuine product at the first point, indicating that the sample has less glue at the first point. The present application does not limit the specific values ​​of the third value and the fourth value.

[0104] In some embodiments, the glue defect existing at the first point of the sample is determined based on the positions of the plurality of fitting edge points.

[0105] If the average width of the area surrounded by multiple fitting edge points is greater than the average width of the glue of the genuine product at the first point, it is determined that the sample has overflow glue at the first point; if the average width of the area surrounded by multiple fitting edge points is less than the average width of the glue of the genuine product at the first point, it is determined that the sample has insufficient glue at the first point. In some embodiments, the width between the fitting edge points of the glue in each column is measured, and the average width is finally determined based on the width of each column. In some embodiments, if the ratio of the average width of the area surrounded by multiple fitting edge points to the average width of the glue of the genuine product at the first point is greater than the fifth value, it is determined that the sample has overflow glue at the first point. In some embodiments, if the ratio of the average width of the area surrounded by multiple fitting edge points to the average width of the glue of the genuine product at the first point is less than the sixth value, it is determined that the sample has overflow glue at the first point. Among them, the fifth value is greater than the sixth value. In some embodiments, the fifth value is 1.3, that is, the ratio of the average width of the area surrounded by multiple fitting edge points to the average width of the glue of the genuine product at the first point is greater than 1.3, that is, the average width of the area surrounded by multiple fitting edge points is at least 30% larger than the average width of the glue of the genuine product at the first point, and it is determined that the sample has overflow glue at the first point. In some embodiments, the sixth value is 0.7, that is, the ratio of the average width of the area surrounded by multiple fitting edge points to the average width of the glue of the genuine product at the first point is less than 0.7, that is, the average width of the area surrounded by multiple fitting edge points is at least 30% smaller than the average width of the glue of the genuine product at the first point, and it is determined that the sample has insufficient glue at the first point.

[0106] In some embodiments, outliers in the edge points are determined based on the edge points and the fitted edge points. When the ratio of the number of outliers to the number of all edge points is greater than a seventh value, it is considered that there is less glue. Optionally, the seventh value is 0.2, and when the ratio of the number of outliers to the number of all edge points is greater than 0.2, it is considered that there is less glue.

[0107] refer to Fig.10 , which shows a schematic diagram of fitting edge points provided by an embodiment of the present application. Fig.10 The neutron graph (a) shows that there are a large number of abnormal points, and it can be considered that there is less glue in some areas.

[0108] In some embodiments, the glue defect existing at the first point of the sample is determined based on the symmetry of multiple fitting edge points. Based on multiple fitting edge points, determine whether the glue of the sample at the first point is symmetrical; if it is asymmetrical, it is determined that the sample has insufficient glue at the first point. In some embodiments, the widths of the abnormal points on both sides of the neck glue are calculated separately: for multiple fitting edge points, scan from both ends to the middle to find the position of the first abnormality, and from this position, calculate the distance to the next non-abnormal point as the width of the abnormal point at one end. Assuming that the widths of the abnormal points at the two ends are a and b, respectively, limit max(a,b) / min(a,b)<eighth value. Optionally, the eighth value is 2. If it exceeds this range, it means that there is asymmetry at both ends, then one of the endpoints is likely to have insufficient glue, where a and b are positive numbers. Fig.10 From the neutron graph (b), we can find that the width of the abnormal point on the right is much larger than that of the abnormal point on the left, so it can be determined that there is less glue on the right.

[0109] The technical solution provided in the embodiment of the present application determines the fitting edge points based on the edge points. The fitting edge points can effectively avoid the noise points in some edge points, and obtain the glue edge that is relatively close to the actual situation, which is less affected by the noise points and thus has better robustness.

[0110] The technical solution provided in the embodiment of the present application determines glue defects based on the distribution of multiple fitting edge points. Since the fitting edge points themselves are closer to the actual glue edges, the accuracy of the determination results of the glue defects can be improved, making the detection effect better and more in line with the actual situation.

[0111] The technical solution provided in the embodiment of the present application determines different defects according to the specific conditions of the edge points after fitting, and the accuracy of the determination result is relatively high. The final glue defect is determined according to the glue area surrounded by multiple fitting edge points, the position of multiple fitting edge points, and the symmetry of multiple fitting edge points. Different conditions have different defect determination criteria, so the defect determination is more standardized and more in line with actual defect determination.

[0112] The technical solution provided in the embodiment of the present application defines glue defects as intrinsic defects. For such defects, the complete glue area is first extracted through a deep image segmentation method (which can also be considered as a semantic segmentation model), and then the incomplete or overflowing parts are detected through a geometric analysis method, and the degree of the defect is qualitatively determined according to the degree of distortion. The technical solution provided in the embodiment of the present application makes full use of the shape characteristics of normal neck glue (authentic) and can correctly analyze the defects of the glue at both ends or in the middle. First, Huber regression is used to model the long side of the neck glue, and the abnormality of the neck glue is determined based on the distribution of abnormal points in the linear regression results. Therefore, there is no need to use a large number of sample images with glue defects for training, and a higher detection rate can be achieved in actual use.

[0113] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0114] Please refer to Fig.11 , which shows a block diagram of a device for determining glue defects provided by an embodiment of the present application. The device has the function of implementing the above method example, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device introduced above, or it can be set in a computer device. Fig.11 As shown, the device 1400 may include: an image acquisition module 1410 , a region determination module 1420 , and a defect determination module 1430 .

[0115] The image acquisition module 1410 is used to acquire a first point image of the sample, wherein the first point image displays some components in the sample and the glue between the components, and the first point image is an image captured at the first point of the sample.

[0116] The area determination module 1420 is used to determine the glue area where the glue is located in the first point image according to the feature information of the first point image extracted by the semantic segmentation model, and the feature information is the deep learning information extracted by the semantic segmentation model.

[0117] The defect determination module 1430 is used to determine the glue defect existing at the first point of the sample according to the distribution of the glue area.

[0118] In some embodiments, the semantic segmentation model includes a feature extraction sub-model and a prediction sub-model.

[0119] In some embodiments, Fig.12 As shown, the region determination module 1420 includes a feature acquisition unit 1422 , an image determination unit 1424 and a region determination unit 1426 .

[0120] The feature acquisition unit 1422 acquires feature information of the first point image through the feature extraction sub-model.

[0121] The image determination unit 1424 obtains a mask image corresponding to the first point image according to the feature information through the prediction sub-model.

[0122] The region determining unit 1426 determines the glue region where the glue in the first point image is located according to the region where the pixels whose values ​​satisfy the condition in the mask image are located.

[0123] In some embodiments, the feature acquisition unit 1422 is used to extract shallow features and deep features of the first point image through the feature extraction sub-model, the shallow features are used to characterize the detail features of the first point image, and the deep features are used to characterize the semantic features of the first point image.

[0124] The feature acquisition unit 1422 is used to fuse the shallow features with the deep features to obtain feature information of the first point image.

[0125] In some embodiments, the region determination unit 1426 is used to determine the region where the pixels in the mask image whose values ​​meet the conditions are located as a foreground region, and the foreground region includes at least one closed region.

[0126] The region determining unit 1426 is used to determine the largest closed region in the foreground region as the glue region where the glue is located.

[0127] In some embodiments, Fig.12 As shown, the defect determination module 1430 includes an edge point determination unit 1432 and a defect determination unit 1434 .

[0128] The edge point determination unit 1432 is used to determine a plurality of edge points of the glue area, where the plurality of edge points constitute the outline of the glue area.

[0129] The edge point determination unit 1432 is used to predict a plurality of fitting edge points of the glue according to the plurality of edge points, wherein the plurality of fitting edge points are closer to the edge of the glue than the plurality of edge points.

[0130] The defect determination unit 1434 is used to determine the glue defect existing in the sample at the first point according to the distribution of the multiple fitting edge points.

[0131] In some embodiments, the defect determination unit 1434 is used to determine the glue defect existing at the first point of the sample based on the glue area enclosed by the multiple fitting edge points; or, to determine the glue defect existing at the first point of the sample based on the position of the multiple fitting edge points; or, to determine the glue defect existing at the first point of the sample based on the symmetry of the multiple fitting edge points.

[0132] In some embodiments, the glue defect includes at least one of insufficient glue and overflowing glue.

[0133] The defect determination unit 1434 is used to determine that there is overflow of glue on the sample at the first point if the ratio of the area of ​​the area enclosed by the multiple fitting edge points to the glue area of ​​the authentic product at the first point is greater than a third value; and to determine that there is insufficient glue on the sample at the first point if the ratio of the area of ​​the area enclosed by the multiple fitting edge points to the glue area of ​​the authentic product at the first point is less than a fourth value; wherein the third value is greater than the fourth value.

[0134] The defect determination unit 1434 is used to determine that there is overflow of glue on the sample at the first point if the average width of the area enclosed by the multiple fitting edge points is greater than the average width of the glue of the authentic product at the first point; and to determine that there is insufficient glue on the sample at the first point if the average width of the area enclosed by the multiple fitting edge points is less than the average width of the glue of the authentic product at the first point.

[0135] The defect determination unit 1434 is used to determine whether the glue of the sample at the first point is symmetrical based on the multiple fitting edge points; if it is not symmetrical, it is determined that there is insufficient glue on the sample at the first point.

[0136] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0137] Fig.13 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown.

[0138] Typically, the computer device 1500 includes a processor 1501 and a memory 1502 .

[0139] Processor 1501 may include one or more processing cores, such as a 4-core processor, a 15-core processor, etc. Processor 1501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, processor 1501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, processor 1501 may also include an AI processor, which is used to process computing operations related to machine learning.

[0140] The memory 1502 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1502 stores a computer program, which is loaded and executed by the processor 1501 to implement the method for determining glue defects provided in the above-mentioned method embodiments.

[0141] Those skilled in the art will understand that Fig.13The structure shown in the figure does not constitute a limitation on the computer device 1500, and the computer device 1500 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0142] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the method for determining a glue defect is implemented.

[0143] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drives) or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0144] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned method for determining a glue defect.

[0145] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0146] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining glue defects, characterized in that: The method comprises: Acquire a first point image of the sample, wherein the first point image shows some components in the sample and glue between the some components, and the first point image is an image obtained by photographing the first point of the sample; Determine a glue area where the glue in the first point image is located according to feature information of the first point image extracted by the semantic segmentation model, wherein the feature information is deep learning information extracted by the semantic segmentation model; Determine a plurality of edge points of the glue area, wherein the plurality of edge points constitute a contour of the glue area; According to the multiple edge points, a plurality of fitting edge points of the glue are predicted by using Huber regression, wherein the multiple fitting edge points are closer to the edge of the glue than the multiple edge points, and the Huber regression method is used for, when the absolute value of the error between the fitting edge point and the edge point is less than or equal to δ, dividing the square of the error by 2 as a loss function, and when the absolute value of the error is greater than δ, multiplying the result obtained by subtracting δ / 2 from the absolute value of the error and δ as a loss function, wherein δ is determined by the proportion of abnormal points in the multiple edge points; If the ratio of the area of ​​the region enclosed by the multiple fitting edge points to the glue area of ​​the authentic product at the first point is greater than a third value, it is determined that the sample has overflowed glue at the first point; if the ratio of the area of ​​the region enclosed by the multiple fitting edge points to the glue area of ​​the authentic product at the first point is less than a fourth value, it is determined that the sample has insufficient glue at the first point; wherein the third value is greater than the fourth value; or, If the average width of the area surrounded by the multiple fitting edge points is greater than the average width of the glue of the authentic product at the first point, it is determined that there is glue overflow in the sample at the first point; if the average width of the area surrounded by the multiple fitting edge points is less than the average width of the glue of the authentic product at the first point, it is determined that there is insufficient glue in the sample at the first point; or, According to the multiple fitting edge points, it is determined whether the glue of the sample at the first point is symmetrical; if it is not symmetrical, it is determined that there is less glue in the sample at the first point.

2. The method according to claim 1, characterized in that The semantic segmentation model includes a feature extraction sub-model and a prediction sub-model; The step of determining the glue region where the glue is located in the first point image according to the feature information of the first point image extracted by the semantic segmentation model includes: Acquiring feature information of the first point image through the feature extraction sub-model; Obtaining, by the prediction sub-model according to the feature information, a mask image corresponding to the first point image; The glue region where the glue in the first point image is located is determined according to the region where the pixels whose values ​​satisfy the conditions in the mask image are located.

3. The method according to claim 2, characterized in that The step of obtaining feature information of the first point image by using the feature extraction sub-model includes: Extracting shallow features and deep features of the first point image through the feature extraction sub-model, wherein the shallow features are used to characterize detail features of the first point image, and the deep features are used to characterize semantic features of the first point image; The shallow features are fused with the deep features to obtain feature information of the first point image.

4. The method according to claim 2, characterized in that: The determining, according to the area where the pixel points whose values ​​satisfy the condition in the mask image are located, the glue area where the glue is located in the first point image comprises: Determine an area where pixels whose values ​​satisfy a condition are located in the mask image as a foreground area, wherein the foreground area includes at least one closed area; The largest closed area in the foreground area is determined as the glue area where the glue is located.

5. A device for determining glue defects, characterized in that: The device comprises: An image acquisition module, used for acquiring a first point image of the sample, wherein the first point image shows some components in the sample and glue between the components, and the first point image is an image obtained by photographing the first point of the sample; A region determination module, configured to determine a glue region where the glue in the first point image is located according to feature information of the first point image extracted by a semantic segmentation model, wherein the feature information is deep learning information extracted by the semantic segmentation model; a defect determination module, for determining a plurality of edge points of the glue area, wherein the plurality of edge points constitute the outline of the glue area; and predicting a plurality of fitting edge points of the glue using Huber regression according to the plurality of edge points, wherein the plurality of fitting edge points are closer to the edge of the glue than the plurality of edge points, and wherein the Huber regression method is used to, when the absolute value of the error between the fitting edge point and the edge point is less than or equal to δ, use the square of the error divided by 2 as a loss function, and when the absolute value of the error is greater than δ, use the product of the absolute value of the error minus δ / 2 and δ as a loss function, wherein δ is determined by the proportion of abnormal points in the plurality of edge points; if the ratio of the area of ​​the area enclosed by the plurality of fitting edge points to the glue area of ​​the genuine product at the first point is greater than a third value, It is determined that there is overflow of glue at the first point of the sample; if the ratio of the area of ​​the area enclosed by the multiple fitting edge points to the glue area of ​​the authentic product at the first point is less than the fourth value, it is determined that there is insufficient glue at the first point of the sample; wherein the third value is greater than the fourth value; or, if the average width of the area enclosed by the multiple fitting edge points is greater than the average width of the glue of the authentic product at the first point, it is determined that there is overflow of glue at the first point of the sample; if the average width of the area enclosed by the multiple fitting edge points is less than the average width of the glue of the authentic product at the first point, it is determined that there is insufficient glue at the first point of the sample; or, based on the multiple fitting edge points, it is determined whether the glue of the sample at the first point is symmetrical; if it is asymmetrical, it is determined that there is insufficient glue at the first point of the sample.

6. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the method according to any one of claims 1 to 4.

Images