Target area identification method, solder paste defect detection method, device, equipment and medium

Abstract

The present application discloses a method for identifying a target area, a method, device, equipment and medium for detecting solder paste defects, and the present application belongs to the technical field of optical detection equipment. The method includes: obtaining a height image and a texture image of the area to be tested; performing feature extraction on the height image to obtain height image features, and performing feature extraction on the texture image to obtain texture image features; performing a fusion operation on the height image features and the texture image features, and using a recognition model to obtain the probability that each module in the area to be tested is a target area; determining the target area in the area to be tested based on a predetermined confidence threshold and the probability corresponding to each module. The present technical solution can accurately identify the target area in different environments, thereby increasing the scope of application of the solution and making it more robust.

Description

Technical Field

[0001] The present application belongs to the technical field of optical detection equipment, and specifically relates to a target area recognition method, solder paste defect detection method, device, equipment and medium. Background Art

[0002] Currently, optical inspection equipment is generally composed of a light source system, an optical imaging system, an image acquisition system, an image processing system, a control system, a mechanical conveying track, a display system, and a data storage system. After soldering the components to the PCB (Printed Circuit Board), the PCB can be transferred to the optical inspection equipment through the mechanical conveying track for optical inspection.

[0003] Among them, solder paste defect detection is an important goal of optical inspection. In the solder paste defect detection process, solder paste identification must be performed first. The accuracy of solder paste identification directly affects the accuracy of solder paste detection. At the same time, solder paste identification is also a difficulty and focus in solder paste detection. At present, solder paste identification and detection are mainly based on manual operation and machine recognition. Among them, manual operation efficiency is low, and manual observation often introduces subjective factors, and machine recognition needs to be based on color recognition of the captured image. However, due to the inconsistent formulation of solder paste raw materials from different manufacturers, there are large differences in solder paste color, and there are scenes where the color of the pad and solder paste are very close. Therefore, the accuracy of solder paste identification through image recognition technology is poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a target area recognition method, solder paste defect detection method, device, equipment and medium, so as to accurately identify the target area based on texture images and height images, based on the aggregation results of the two image features. The target area can be accurately identified in different environments, thereby improving the scope of application of the solution and making it more robust.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying a target area, the method comprising:

[0006] Acquire the height image and texture image of the area to be measured;

[0007] Performing feature extraction on the height image to obtain height image features, and performing feature extraction on the texture image to obtain texture image features;

[0008] Performing a fusion operation on the height image features and the texture image features, and using a recognition model to obtain a probability that each module in the test area is a target area;

[0009] The target area in the area to be tested is determined according to a predetermined confidence threshold and a probability corresponding to each module.

[0010] The beneficial effect of this solution is that feature extraction can be performed on the height image and texture image of the area to be tested respectively to obtain height image features and texture image features. The recognition model performs recognition based on the aggregated features of the height image features and the texture image features, outputs a probability map of the area to be tested, and then determines the position of the target area in combination with a preset threshold. The recognition model of this application recognizes the target area based on multiple features and fusion features of the two images, with higher recognition accuracy and efficiency.

[0011] Further, performing feature extraction on the height image to obtain height image features, and performing feature extraction on the texture image to obtain texture image features, including:

[0012] Extracting detail features and semantic features from the height image to obtain height image features, wherein the height image features include detail features and semantic features of the height image;

[0013] Detail feature extraction and semantic feature extraction are performed on the texture image to obtain texture image features, wherein the texture image features include detail features and semantic features of the texture image.

[0014] This scheme extracts detail features and semantic features from height images and texture images respectively. The height image features include detail features and semantic features, and the texture image features include detail features and semantic features, so that the height image features and texture image features are richer. The detail features can reflect the detail information of the target area, and the semantic features can reflect the neighborhood information of the target area. The target area is identified based on at least four features, which further improves the recognition accuracy, has a good scope of application, and is more robust.

[0015] Furthermore, feature extraction is performed on the height image to obtain height image features, including:

[0016] Extracting features from the height image through a recognition model to obtain detail features and semantic features of the height image;

[0017] Aggregating the height image detail feature and the height image semantic feature to obtain a height image feature;

[0018] Extracting features from the texture image to obtain texture image features includes:

[0019] Extracting features of the texture image through a recognition model to obtain detail features and semantic features of the texture image;

[0020] The texture image detail features and the texture image semantic features are aggregated to obtain texture image features.

[0021] The beneficial effect of this solution is that, by extracting detail features and semantic features from each image separately through the recognition model, and then fusing the detail features with the texture features, texture image features and height image features containing detail features and semantic features are obtained, and the target area can be identified from different feature dimensions, thereby improving the recognition accuracy.

[0022] Furthermore, the height image detail feature and the height image semantic feature are aggregated to obtain the height image feature, including:

[0023] Performing depth-separable convolution processing and conventional convolution processing on the height image detail features to obtain a first detail feature map;

[0024] Performing conventional convolution processing and pooling processing on the height image detail features to obtain a second detail feature map;

[0025] Performing conventional convolution processing and upsampling processing on the height image semantic features to obtain a first semantic feature map;

[0026] Performing depth-separable convolution processing and conventional convolution processing on the height image semantic features to obtain a second semantic feature map;

[0027] The first detail feature map is fused with the first semantic feature map to obtain a first aggregation factor; and the second detail feature map is fused with the second semantic feature map to obtain a second aggregation factor;

[0028] Performing an aggregation operation on the first aggregation factor and the second aggregation factor to obtain a height image feature;

[0029] Aggregating the texture image detail features and the texture image semantic features to obtain texture image features includes:

[0030] Performing depth-separable convolution processing and conventional convolution processing on the detail features of the texture image to obtain a third detail feature map;

[0031] Performing conventional convolution processing and pooling processing on the detail features of the texture image to obtain a fourth detail feature map;

[0032] Performing conventional convolution processing and upsampling processing on the semantic features of the texture image to obtain a third semantic feature map;

[0033] Performing depth-separable convolution processing and conventional convolution processing on the semantic features of the texture image to obtain a fourth semantic feature map;

[0034] fusing the third detail feature map with the third semantic feature map to obtain a third aggregation factor; and fusing the fourth detail feature map with the fourth semantic feature map to obtain a fourth aggregation factor;

[0035] An aggregation operation is performed on the third aggregation factor and the fourth aggregation factor to obtain a texture image feature.

[0036] The beneficial effect of this solution is that after obtaining the detail features and semantic features of each image, two different methods are used to extract feature maps for the detail features and semantic features respectively, thereby realizing a processing method of first fusing and then aggregating different extraction methods, so that when obtaining the features of the current image, the difference between the target area and the background area can be better shown, thereby improving the recognition accuracy.

[0037] Further, the second detail feature map and the second semantic feature map are fused to obtain a second aggregation factor, including:

[0038] The second detail feature map is fused with the second semantic feature map, and scaled according to a preset scaling factor to obtain a second aggregation factor, so that the size of the second aggregation factor is consistent with the size of the first aggregation factor;

[0039] The fourth detail feature map is fused with the fourth semantic feature map to obtain a fourth aggregation factor, including:

[0040] The fourth detail feature map and the fourth semantic feature map are fused and scaled according to a preset scaling factor to obtain a fourth aggregation factor, so that the size of the fourth aggregation factor is consistent with the size of the third aggregation factor.

[0041] The beneficial effect of this solution is that before the aggregation operation, the two aggregation factors are aligned first, which can improve the subsequent aggregation accuracy and avoid the calculation problems and aggregation accuracy problems caused by the inconsistency between the sizes of the two aggregation factors and the mismatch between the aggregated points.

[0042] Furthermore, the training process of the recognition model includes:

[0043] Acquire a data set consisting of a preset number of data pairs; wherein each data pair includes a height image and a texture image;

[0044] Using the initial model to annotate a target area of ​​a portion of the data in the data set, and obtaining an initial training result of the initial model;

[0045] The target area of ​​the data pairs in the data set is annotated using the initial training results, and parameters are fine-tuned according to feedback information of the annotation results to obtain the final training results as a recognition model.

[0046] The beneficial effect of this solution is that by using part of the data for initial training and using other data or all the data for fine-tuning training, the training efficiency of the model can be improved, the accuracy of the model training can be ensured, and the model convergence speed can be increased.

[0047] Furthermore, after obtaining the recognition model, the method further includes:

[0048] extracting a random number of samples from the data set;

[0049] Performing highly abnormal filtering on the random number of samples to obtain filtered data;

[0050] The recognition model is optimized using the filtered data to obtain an optimized recognition model.

[0051] The beneficial effect of this solution is that by filtering out outliers, the training accuracy of the recognition model can be improved, making the parameters of the recognition model more accurate, thus providing a guarantee for the accuracy of subsequent target recognition.

[0052] Furthermore, the initial model is a large visual detection model; the large visual detection model labels the target area of ​​part of the data based on pre-configured initialization information.

[0053] The beneficial effect of this solution is that by using a large model and configuring the initialization information, the training speed of the recognition model can be increased and the overall performance of the recognition model can be improved.

[0054] In a second aspect, an embodiment of the present application provides a solder paste defect detection method, wherein the solder paste defect detection method identifies a solder paste area according to the target area identification method described in the first aspect, and the solder paste defect detection method includes:

[0055] Get the parameters to be tested in the solder paste area;

[0056] Determine whether the parameters to be tested in the solder paste area meet the preset threshold;

[0057] When the parameter to be measured in the solder paste area meets a preset threshold, it is determined that a solder paste defect exists.

[0058] The beneficial effect of this solution is that the identification method based on the above-mentioned target area can accurately identify the solder paste area of ​​the PCB board and collect the parameters to be measured in the solder paste area, which can provide a data basis for solder paste defect identification and provide accurate data information for identifying whether there are defects in the solder paste area, while also reducing the calculation amount of solder paste defect identification.

[0059] In a third aspect, an embodiment of the present application provides a device for identifying a target area, the device comprising:

[0060] An image acquisition module is used to acquire a height image and a texture image of the identified object;

[0061] A feature extraction module, used to extract features from the height image to obtain height image features, and to extract features from the texture image to obtain texture image features;

[0062] A probability generation module, used for fusing the height image features and the texture image features, and obtaining the probability of each module in the tested area being the target area by using a recognition model;

[0063] The target area identification module is used to determine the target area in the area to be tested according to a predetermined confidence threshold and the probability corresponding to each module.

[0064] The beneficial effect of this solution is that feature extraction can be performed on the height image and texture image of the area to be tested respectively to obtain height image features and texture image features. The recognition model performs recognition based on the aggregated features of the height image features and the texture image features, outputs a probability map of the area to be tested, and then determines the position of the target area in combination with a preset threshold. The recognition model of this application recognizes the target area based on multiple features and fusion features of the two images, with higher recognition accuracy and efficiency.

[0065] In a fourth aspect, an embodiment of the present application provides a solder paste defect detection device, wherein the solder paste defect detection device identifies a solder paste area according to the target area identification device as described in the third aspect, and the solder paste defect detection device includes:

[0066] The module for obtaining the parameters to be tested obtains the parameters to be tested in the solder paste area;

[0067] A judgment module is used to judge whether the parameters to be tested in the solder paste area meet the preset threshold value;

[0068] The solder paste defect detection module is used to determine the presence of a solder paste defect when the parameter to be measured in the solder paste area meets a preset threshold.

[0069] The beneficial effect of this solution is that the identification method based on the above-mentioned target area can accurately identify the solder paste area of ​​the PCB board and collect the parameters to be measured in the solder paste area, which can provide a data basis for solder paste defect identification and provide accurate data information for identifying whether there are defects in the solder paste area, while also reducing the calculation amount of solder paste defect identification.

[0070] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the target area recognition method as described in the first aspect, or implements the solder paste defect detection method as described in the second aspect.

[0071] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the target area identification method as described in the first aspect is implemented, or the solder paste defect detection method as described in the second aspect is implemented.

[0072] In the seventh aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the target area identification method as described in the first aspect, or to implement the solder paste defect detection method as described in the second aspect.

[0073] In an embodiment of the present application, a height image and a texture image of the area to be tested are obtained; feature extraction is performed on the height image to obtain height image features, and feature extraction is performed on the texture image to obtain texture image features; the height image features and the texture image features are fused, and the probability of each module in the area to be tested being the target area is obtained using a recognition model; the target area in the area to be tested is determined based on a predetermined confidence threshold and the probability corresponding to each module. The technical solution provided in the embodiment of the present application can include both the height image and the texture image in the input data of the model, and extract detail features and semantic features respectively, obtain a mask image after aggregating the two features, and then combine the actual recognition needs to accurately identify the solder paste area of ​​the PCB board, with a better scope of application and stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of a target area identification method provided in an embodiment of the present application;

[0075] Figure 2 is a schematic diagram of obtaining a height image provided in an embodiment of the present application;

[0076] Figure 3It is a schematic diagram of the internal processing flow of the polymerization unit provided in the embodiment of the present application;

[0077] Figure 4 is a schematic diagram of the recognition process of the recognition model provided in the embodiment of the present application;

[0078] Figure 5 is a schematic diagram comparing the recognition effects of the recognition models provided in the embodiments of the present application;

[0079] Figure 6 It is a flowchart of the recognition model training provided in the embodiment of the present application;

[0080] Figure 7 It is a flow chart of a solder paste defect detection method provided in an embodiment of the present application;

[0081] Figure 8 is a schematic diagram of the structure of a target area recognition device provided in an embodiment of the present application;

[0082] Fig. 9 is a schematic structural diagram of a solder paste defect detection device provided in an embodiment of the present application;

[0083] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only part of the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0086] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0087] In conjunction with the accompanying drawings, the target area identification method, solder paste defect detection method, device, equipment and medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0088] In the prior art, SPI (Solder Paste Inspection) is a device that detects common defects encountered in the solder paste printing process of PCB boards based on optical principles. Currently, before testing, the device first takes a photo of a standard PCB board and creates a corresponding PCB board inspection program as an inspection standard. In the actual test process, after the board to be tested flows into the device, the device takes a photo of the board and compares it with the standard board based on the prepared inspection program, and then determines whether the board to be tested is qualified.

[0089] In the PCBA (Printed Circuit Board Assembly, the entire process of PCB blank board through SMT loading, or DIP plug-in), solder paste is an important medium to connect components to pads, and it is also the first process, which directly affects the connection quality between components and pads in the subsequent component placement process. According to the amount of tin applied and the printing position, the common defects of solder paste include insufficient tin, excessive tin, continuous tin, and irregular shapes. Severe solder paste defects will lead to batch accidents and failure of the entire PCB board. The solder paste defect detection process mainly includes three links: solder paste identification, solder paste property calculation, and defect screening based on property values. Among them, the accuracy of solder paste identification directly affects the calculation accuracy of solder paste properties. At the same time, solder paste identification is also a difficulty and focus in SPI detection. The inconsistent formulation of solder paste raw materials from different manufacturers leads to large differences in solder paste color, and there are scenarios where the pad and solder paste color are very close, which puts higher requirements on the robustness and generalization of solder paste identification algorithms. At present, there is no relatively mature, accurate, and universal algorithm to solve this problem. The present invention proposes an AI method for identifying solder paste areas based on RGBD, which solves the problems of poor solder paste recognition stability, troublesome debugging and poor recognition effect in the solder paste defect detection process.

[0090] The color-based method is time-consuming and labor-intensive. The color of solder paste varies from manufacturer to manufacturer and production line to production line, requiring operators to frequently debug equipment. In addition, the color cannot be accurately extracted, resulting in a high false alarm rate. The height-based method for identifying solder paste relies entirely on height information and cannot accurately identify the location of the solder paste. The recognition effect of solder paste ligation and excessive soldering is poor. The AI-based method is relatively simple to program and easy to operate, but the existing method only uses RGB information as input and cannot accurately and robustly identify solder paste in complex scenarios.

[0091] The embodiment of the present invention realizes solder paste recognition based on a method of fusing depth and texture information. It combines the rich texture information and the spatial information of depth, can adapt to the influence of color, shape, light, position changes, etc., and has the advantages of high precision, good stability, strong versatility, etc.

[0092] Figure 1 FIG. 1 is a flow chart of a target area identification method provided in an embodiment of the present application. Figure 1 As shown, the specific steps include:

[0093] S101, obtaining a height image and a texture image of the area to be measured;

[0094] First, the use scenario of this solution can be a scenario of accurately locating the target area of ​​the object to be measured. Based on the above use scenario, it can be understood that the execution subject of this solution can be an intelligent terminal with image recognition and data processing capabilities, such as: mobile phones, tablet computers and desktop computers, etc., which are not limited here.

[0095] As an example, the object to be tested in this solution may refer to a PCB (Printed Circuit Board), which is a basic circuit board used to support and connect electronic components. The area to be tested may be the entire PCB image or a partial area in the PCB image, such as the left 1 / 2 area of ​​the PCB image.

[0096] The height image may refer to an image in which each pixel includes height value information; correspondingly, the measuring device may refer to a device that can acquire the height image of the object to be measured.

[0097] In one embodiment, the image height measurement is performed to obtain the height image output by each measuring device, and structured light imaging technology, laser ranging technology, radar ranging technology, etc. Among them, structured light imaging technology is the mainstream technology used for three-dimensional reconstruction of the PCB board to be measured. Structured light imaging technology projects a structured light source and uses a camera to capture the deformation information of the object surface, thereby realizing three-dimensional reconstruction of the object; accordingly, the above-mentioned measuring device can be a camera used by structured light imaging technology to capture structured light projected onto the object.

[0098] Figure 2 Schematic diagram of obtaining a height image provided by an embodiment of the present application. Figure 2 As shown, there are three measuring devices, namely measuring device 1, measuring device 2 and measuring device 3, and each measuring device can output a height image. By combining the three height images, a height image of the object to be measured, such as a height image of a PCB board, can be obtained. Moreover, the height image can avoid the problem that the height value of some pixels is affected by the presence of protruding objects on the object to be measured.

[0099] The texture image may be an image captured by an RGB camera. As an implementation method, a color camera may be placed directly above the object to be measured to capture the object. For example, after the PCB board flows into the device, its center is aligned with the bottom of the RGB camera to capture the texture image through the RGB camera.

[0100] S102, performing feature extraction on the height image to obtain height image features, and performing feature extraction on the texture image to obtain texture image features;

[0101] A feature extraction algorithm or a feature extraction model is used to extract features from the height image and the texture image of the area to be measured, and the height image features and the texture image features are obtained respectively.

[0102] The feature extraction model may be a model with two feature extraction branches. Each feature extraction branch may correspond to one or more processing methods for image data. After performing feature extraction respectively, features of different images may be obtained. For example, in this solution, it may be regarded as having a height image processing branch and a texture image processing branch.

[0103] The height image feature may be a feature obtained by extracting features from the height image, and the texture image feature may be a feature obtained by extracting features from the texture image. The feature extraction method may include convolution, pooling, upsampling, downsampling, and activation function processing. The features obtained here can be used to reflect the features of distinguishing foreground pixels from background pixels based on the height image, and the features of distinguishing foreground pixels from background pixels based on the texture image.

[0104] S103, performing a fusion operation on the height image feature and the texture image feature, and obtaining a probability that each module in the test area is a target area by using a recognition model;

[0105] Optionally, a fusion operation can be performed on the height image features and the texture image features using a fusion algorithm or model. The fusion operation can be understood as the process of fusing the corresponding pixel points between the two image features. For example, for a pixel point at the same position, the probability value of being identified as a target area in the height image feature is 30%, and the probability value of being identified as a target area in the texture image feature is 50%. After fusion, the two probability values ​​can be weighted averaged. Since the weights are not specifically limited, it can be obtained that the probability value of the pixel point after fusion can be 40%.

[0106] It can be understood that after fusion, the probability of each module in the entire test area being the target area can be obtained, and all modules can carry their own probability information and position information to form a whole image with probability information, which can be used as a probability map. It should be noted that other forms can also be selected here, and are not limited to the form of using a graph, such as the form of a tabular data composed of the pixel position and the probability of each position, which is not too limited here.

[0107] The area to be tested is composed of at least one module, and the module can be composed of one pixel or multiple pixels. If it is one pixel, the probability value of the module is the probability value of the pixel. If it is multiple pixels, the probability value of the module can be the average of the probability values ​​of all pixels within the module. In addition, in the process of determining the module, it can also be divided according to a pre-set probability value range, for example, the probability value is between 95%-100%, and adjacent pixels can be used as a module, and the probability value is between 90%-95%, and adjacent pixels can be used as a module. By analogy, each module in the area to be tested can be obtained.

[0108] S104: Determine a target area in the area to be tested according to a predetermined confidence threshold and a probability corresponding to each module.

[0109] The confidence threshold can be determined by the user based on actual detection accuracy requirements or historical data. For example, if the detection accuracy requirement for the identification of the target area is extremely high, the confidence threshold can be set to 95%, that is, only when the probability of identifying a certain module or pixel as the target area is greater than or equal to 95%, can it be determined as the target area. In this solution, since the modules are divided in advance, the target area in the area to be tested can be determined based on the comparison result of the predetermined confidence threshold and the probability corresponding to each module.

[0110] It can be understood that the confidence threshold can be determined by the user according to the needs or historical data. After the confidence threshold is determined, the same confidence threshold can be used in a board inspection to identify the solder paste area of ​​a batch of PCB boards.

[0111] In another case, if the user does not input the confidence threshold, a default value may be used, for example, the default value is 80%.

[0112] In this solution, whether each module is a target area can be determined based on the comparison result between the confidence threshold and the probability corresponding to each module, thereby obtaining the target area in the entire area to be tested.

[0113] The technical solution provided in the embodiment of the present application is to obtain a height image and a texture image of the area to be tested; perform feature extraction on the height image to obtain height image features, and perform feature extraction on the texture image to obtain texture image features; perform fusion operation on the height image features and the texture image features, and use a recognition model to obtain the probability that each module in the area to be tested is the target area; and determine the target area in the area to be tested according to a predetermined confidence threshold and the probability corresponding to each module. The technical solution provided in the embodiment of the present application includes both a height image and a texture image in the input data of the model, and the recognition model identifies the target area based on multiple features and fusion features of the two images, and then combines the actual recognition needs to accurately identify the target area, such as the solder paste area of ​​a PCB board, with a better scope of application and stronger robustness.

[0114] In an optional embodiment, performing feature extraction on the height image to obtain height image features, and performing feature extraction on the texture image to obtain texture image features, comprises:

[0115] Extracting detail features and semantic features from the height image to obtain height image features, wherein the height image features include detail features and semantic features of the height image;

[0116] Detail feature extraction and semantic feature extraction are performed on the texture image to obtain texture image features, wherein the texture image features include detail features and semantic features of the texture image.

[0117] Among them, detail features can refer to more specific, small and local characteristics or attributes of things or phenomena. For example, in a picture, detail features may include color changes, line thickness and curvature, brushstrokes, etc. Taking a picture of a car as an example, its detail features can be the pattern of the wheel hub, the shape of the door handle, the decoration on the car logo, etc. Semantic features can express the meaning, concept and information contained in things. Semantic features focus on the understanding and interpretation of things. For example, in a picture, semantic features may include a kind of fruit composed of certain pixels, a dining table composed of certain pixels, and a wall composed of certain pixels, etc.

[0118] This scheme extracts detail features and semantic features from height images and texture images respectively. The height image features include detail features and semantic features, and the texture image features include detail features and semantic features. The detail features can reflect the detail information of the target area, and the semantic features can reflect the neighborhood information of the target area. The extraction of detail features and semantic features makes the height image features and texture image features richer. The target area is identified based on at least four features, which further improves the recognition accuracy, has a good scope of application, and is more robust.

[0119] Optionally, extracting features from the height image using a recognition model to obtain height image features includes:

[0120] Extracting features from the height image through a recognition model to obtain detail features and semantic features of the height image;

[0121] Aggregating the height image detail feature and the height image semantic feature to obtain a height image feature;

[0122] Extracting features of the texture image through a recognition model to obtain texture image features includes:

[0123] Extracting features of the texture image through a recognition model to obtain detail features and semantic features of the texture image;

[0124] The texture image detail features and the texture image semantic features are aggregated to obtain texture image features.

[0125] Here, detail features are features used to represent image texture details, contrast details, gradient details, etc. Semantic features can be features used to reflect that different objects in an image have different semantic information. For example, in an image, the left half is a forest and the right half is land. The pixels in the left half can be given semantic information of the forest, and the pixels in the right half can be given semantic information of the land.

[0126] After the recognition model obtains the detail features and the semantic features, an aggregation unit may be used to perform an aggregation operation on the detail features and the semantic features.

[0127] It can be understood that the aggregation unit here is a processing unit for performing aggregation operations, which may include one or more algorithms. The aggregation operation mode of the aggregation unit may be to correspond the detail features with the semantic features according to the pixels, and to determine the output result of the pixel by combining the detail feature information and the semantic feature information of each pixel.

[0128] Among them, after each pixel is aggregated, the recognition model can perform recognition based on the aggregated features, and after recognition, a value can be output, as well as the probability of the pixel taking the value. For example: if it is a target area, it takes 1; if it is a background area, it takes 0, then the output result of a certain pixel can be "1, 85%", where 1 means that the pixel is very likely to be the target area, and 85% means that the probability that the pixel is actually the target area is 85%. Alternatively, the recognition model outputs the probability that each pixel is the target area, such as: 85%, indicating that the probability that the pixel is the target area is 85%.

[0129] This solution can identify the target area from different feature dimensions by identifying the detail features and semantic features of each image respectively, thereby improving the recognition accuracy.

[0130] Optionally, aggregating the height image detail feature and the height image semantic feature to obtain the height image feature includes:

[0131] Performing depth-separable convolution processing and conventional convolution processing on the height image detail features to obtain a first detail feature map;

[0132] Performing conventional convolution processing and pooling processing on the height image detail features to obtain a second detail feature map;

[0133] Performing conventional convolution processing and upsampling processing on the height image semantic features to obtain a first semantic feature map;

[0134] Performing depth-separable convolution processing and conventional convolution processing on the height image semantic features to obtain a second semantic feature map;

[0135] The first detail feature map is fused with the first semantic feature map to obtain a first aggregation factor; and the second detail feature map is fused with the second semantic feature map to obtain a second aggregation factor;

[0136] Performing an aggregation operation on the first aggregation factor and the second aggregation factor to obtain a height image feature;

[0137] Aggregating the texture image detail features and the texture image semantic features to obtain texture image features includes:

[0138] Performing depth-separable convolution processing and conventional convolution processing on the detail features of the texture image to obtain a third detail feature map;

[0139] Performing conventional convolution processing and pooling processing on the detail features of the texture image to obtain a fourth detail feature map;

[0140] Performing conventional convolution processing and upsampling processing on the semantic features of the texture image to obtain a third semantic feature map;

[0141] Performing depth-separable convolution processing and conventional convolution processing on the semantic features of the texture image to obtain a fourth semantic feature map;

[0142] fusing the third detail feature map with the third semantic feature map to obtain a third aggregation factor; and fusing the fourth detail feature map with the fourth semantic feature map to obtain a fourth aggregation factor;

[0143] An aggregation operation is performed on the third aggregation factor and the fourth aggregation factor to obtain a texture image feature.

[0144] Among them, it can be understood that the above-mentioned aggregation process can include the calculation process of four channels, which are respectively used to process the height image detail features, the height image semantic features, the texture image detail features and the texture image semantic features. Specifically, the first channel is used to process the height image detail features to obtain the first detail feature map; the second channel is used to process the height image detail features to obtain the second detail feature map; the first channel is used to process the height image semantic features to obtain the first semantic feature map; the second channel is used to process the height image semantic features to obtain the second semantic feature map; and the first detail feature map and the first semantic feature map are fused to obtain the first aggregation factor; and the second detail feature map and the second semantic feature map are fused to obtain the second aggregation factor, and then the aggregation operation is performed to obtain the height image features.

[0145] Using a similar processing method, texture image features can be obtained.

[0146] Here, each channel is processed differently. For example, each channel may include one or more of a convolution layer, a pooling layer, an upsampling layer, and a downsampling layer. The specific convolution method, such as the size and step length of the convolution kernel, may also be adjusted as required.

[0147] Through such a setting, this scheme can fuse the detail features and semantic features of the pixels before aggregation in the process of aggregation after feature extraction, so that the aggregation result obtained by this scheme is more in line with the actual situation, and different channels can be used for different processing. The processing dimensions of detail features and semantic features are more controllable, which helps to improve the recognition accuracy of the recognition model.

[0148] Figure 3 Schematic diagram of the internal processing flow of the polymerization unit provided in the embodiment of the present application. Figure 3 As shown, the two channels on the left are the first channel and the second channel, and the input parameters of the first channel and the second channel are both detail features.

[0149] The two channels on the right are the third channel and the fourth channel respectively, and the input parameters of the third channel and the fourth channel are both semantic features.

[0150] Thus, the first detail feature map obtained by the first channel and the first semantic feature map obtained by the third channel have the same size and can be fused to obtain a first aggregation factor. The second detail feature map obtained by the second channel and the second semantic feature map obtained by the fourth channel have the same size and can be fused to obtain a second aggregation factor.

[0151] The two aggregation factors are then aggregated. Here, since the first aggregation factor carries detail information and semantic information, and the second aggregation factor also carries detail information and semantic information, the aggregation result may also carry detail information and semantic information.

[0152] Similarly, after performing the above processing on the height image, the above processing may also be performed on the texture image, and finally the height image features and the texture image features may be obtained respectively.

[0153] It should be noted that, for a conventional convolution layer, assuming that the input layer is a 64×64 pixel, three-channel color image, after a convolution layer containing 4 filters, 4 feature maps are finally output, and the size is the same as the input layer. The depthwise separable convolution layer decomposes a complete convolution operation into two steps, namely Depthwise Convolution and Pointwise Convolution. Among them, Depthwise Convolution is to first perform the first convolution operation on a 64×64 pixel, three-channel color image. The difference is that this convolution is completely performed in a two-dimensional plane, and the number of filters is the same as the depth of the previous layer. Therefore, a three-channel image generates 3 feature maps after the operation, and one of the filters only contains a kernel of size 3×3. The number of feature maps after the completion of Depthwise Convolution is the same as the depth of the input layer, but this operation ends after performing convolution operations on each channel of the input layer independently, and does not effectively utilize the information of different maps at the same spatial position. Therefore, it is necessary to add another step to combine these maps to generate a new feature map, which is the next Pointwise Convolution. The operation of Pointwise Convolution is very similar to the conventional convolution operation, the difference is that the size of the convolution kernel is 1×1×M, where M is the depth of the previous layer. Therefore, the convolution operation here will perform a weighted combination of the maps of the previous step in the depth direction to generate a new feature map. There are as many feature maps as there are filters, and it uses a 1×1 convolution method. After Pointwise Convolution, 4 feature maps are also output, which is the same as the output dimension of conventional convolution.

[0154] The number of parameters of conventional convolution is:

[0155] ;

[0156] The parameters of Separable Convolution are obtained by adding two parts:

[0157] ;

[0158] ;

[0159] ;

[0160] For the same input, we get 4 feature maps. The number of parameters of Separable Convolution is about 1 / 3 of that of conventional convolution. Therefore, under the premise of the same number of parameters, the number of neural network layers using Separable Convolution can be deeper.

[0161] On the basis of the above technical solutions, optionally, the second detail feature map and the second semantic feature map are fused to obtain a second aggregation factor, including:

[0162] The second detail feature map is fused with the second semantic feature map, and scaled according to a preset scaling factor to obtain a second aggregation factor, so that the size of the second aggregation factor is consistent with the size of the first aggregation factor;

[0163] The fourth detail feature map is fused with the fourth semantic feature map to obtain a fourth aggregation factor, including:

[0164] The fourth detail feature map and the fourth semantic feature map are fused and scaled according to a preset scaling factor to obtain a fourth aggregation factor, so that the size of the fourth aggregation factor is consistent with the size of the third aggregation factor.

[0165] The size of the first aggregation factor is H×W×C; the size of the first aggregation factor is H / 4×W / 4×C; therefore, before performing the aggregation operation on the first aggregation factor and the second aggregation factor, the second aggregation factor is Similarly, the fourth aggregation factor also needs to be upsampled to be consistent with the size of the third aggregation factor.

[0166] In another embodiment, the first aggregation factor and the third aggregation factor may also be The downsampling process is used to align the pixels between the two aggregation factors.

[0167] In this scheme, the second aggregation factor is The upsampling process can realize the pixel alignment operation between the two aggregation factors, avoiding the need for other processing to complete the aggregation operation. This setting of the scheme can not only retain the extracted feature information, but also ensure the normal calculation, and improve the recognition accuracy of the target area.

[0168] It should be noted that Figure 3The BN in stands for BN layer, i.e., Batch Normalization layer, which has the main functions of normalizing data, accelerating training, allowing the use of higher learning rates, playing a certain regularization effect, and reducing sensitivity to initial weights. The Batch Normalization (BN) layer is mainly used to solve the problem of numerical instability during the training of deep neural networks. It normalizes each batch of data so that the mean of the data is close to 0 and the variance is close to 1, which helps to solve the problem of internal covariate shift. The BN layer makes the network easier to train because the network no longer needs to constantly adapt to changes in the distribution of input data, so faster convergence speed and less training time can be achieved. In addition, the BN layer can also use a higher learning rate because the internal covariate shift is reduced, and the network can be trained more stably. This technology also has a certain regularization effect, which helps prevent the model from overfitting by introducing noise during the training process. Moreover, the BN layer reduces the sensitivity of the network to the initial weights, making the network less sensitive to the choice of initial weights, which helps to improve the robustness of the model.

[0169] Figure 4 Schematic diagram of the recognition process of the recognition model provided in the embodiment of the present application. Figure 4 As shown, a texture image, i.e., an RGB image, and a height image, i.e., a depth image, are input. Specifically, feature recognition can be performed on the texture image to obtain texture image features, and feature recognition can be performed on the height image to obtain height image features. After obtaining the texture image features and the height image features, the two are fused to obtain a probability map.

[0170] In this solution, through such a setting, the texture image and the height image can be taken into account respectively in the process of extracting detail features and semantic features, so that the obtained detail features and semantic features have information provided by the two images, making the detail feature extraction and semantic feature extraction more comprehensive.

[0171] The model architecture adopted in this solution can be understood as a SegFast_RGBD model. Specifically, it is a dual-stream encoding CNN structure that extracts feature information of different depths. Combined with the feature-guided aggregation unit, it effectively fuses the two different dimensional features of RGB and Depth. The features extracted by each module are as follows: Figure 4 As shown in the figure, the dual-stream encoding CNN structure: RGB image data and Depth image data are data of different modalities and cannot be directly fused. Moreover, the information contained in the two is inconsistent. The RGB image contains rich texture and structural information of solder paste, which is more helpful for learning details. The Depth image contains the spatial features of solder paste, which is more helpful for segmenting the foreground and background.

[0172] This method customizes two branches, detail extraction and semantic extraction, according to the characteristics of different input data. The detail extraction branch uses high resolution and low channel design. The purpose of high resolution is to learn the low-dimensional texture information of the data and improve the segmentation accuracy. The purpose of low channel is to reduce the number of parameters and ensure the real-time calculation. The semantic extraction branch uses low resolution and high channel design. The purpose of low resolution is to obtain a larger receptive field and richer neighborhood information. The purpose of high channel is to balance the information loss of low resolution.

[0173] The aggregation unit is an important component in the recognition model. Its purpose is to effectively aggregate the feature representations of the detail extraction branch and the semantic extraction branch. The specific implementation details are mainly composed of DW-Conv (deep separable convolution) + upsample (upsampling) + Sigmoid activation function + BN layer. The aggregation unit uses a bidirectional aggregation method, that is, the high-level semantic information in the semantic branch is used to guide the low-level detail information in the detail branch, thereby achieving an effective combination of two types of features. In this way, it can ensure that while retaining the detail information, the semantic information can also be fully utilized for more accurate pixel-level classification. This fusion strategy helps to improve the accuracy of segmentation because it allows the network to take into account both local and global information when making decisions.

[0174] Figure 5 is a comparative schematic diagram of the recognition effect of the recognition model provided in the embodiment of the present application. Figure 5 As shown in the figure, the SegFast_RGBD model that integrates RGBD information is compared with the feature map and inference results that only use RGB information. It can be seen that after integrating Depth information, the accuracy of the model has been greatly improved.

[0175] Figure 6 Schematic diagram of the process of training the recognition model provided in the embodiment of the present application. Figure 6 As shown, the specific steps include:

[0176] S601, obtaining a data set consisting of a preset number of data pairs; wherein each data pair includes a height image and a texture image;

[0177] The data pair may be a texture image and a height image obtained by photographing the same PCB board, wherein the resolution of the texture image and the height image may be the same, that is, they have the same pixel points and arrangement.

[0178] The preset number of data pairs may be tens of thousands of data pairs. In some cases where sample data is insufficient, data enhancement processing may be performed, such as flipping, rotating, scaling, shifting, and blurring the image data, to increase the number of data pairs.

[0179] S602, using the initial model to mark the target area of ​​part of the data in the data set, and obtaining an initial training result of the initial model;

[0180] In this solution, an initial model can be pre-built, for example, the initial model can include a visual detection model. The visual detection model is an artificial intelligence model specially designed for image segmentation tasks. Image segmentation is one of the core tasks of machine vision.

[0181] The visual inspection large model is trained on a large amount of data, including promptable model training on 11 million images, using promptable segmentation tasks that can achieve strong zero-shot generalization. This training method gives the visual inspection large model strong generalization capabilities and the ability to learn general visual knowledge extracted from large-scale datasets. Because the visual inspection large model learns general visual knowledge extracted from other large-scale datasets and prompts learning through image samples in the field of visual inspection, it has strong generalization capabilities and high accuracy and efficiency in detecting image defects.

[0182] The partial data may be a small part of the data set, for example, if the data set includes 60,000 data pairs, then the partial data may be 3,000 to 5,000 data pairs. The target area of ​​the partial data may be labeled by using the initial model to identify the target area in the texture image and / or depth image in the partial data, combined with manual verification, and the data pairs with errors in the verification process are used as feedback data for the initial model to train.

[0183] In a feasible solution, optionally, the initial model is a large visual detection model; the large visual detection model marks the target area of ​​part of the data based on pre-configured initialization information.

[0184] Among them, initialization information is an AI prompt word, which is a method of using natural language to guide or inspire AI models to complete specific tasks. It is a new paradigm that has emerged in the field of natural language processing (NLP) in recent years, also known as "pretrain-prompt-predict". The role of initialization information is to prompt the AI ​​model with the context of input information and parameter information of the input model. When training supervised learning or unsupervised learning models, initialization information can help the model better understand the intent of the input and respond. In addition, initialization information can also help improve the interpretability and accessibility of the model.

[0185] This solution can accurately identify the target area through manually prompted initialization information without the need for fine-tuning. Combined with this marking method, the marking efficiency of the solder paste area can be increased by 10 times.

[0186] Among them, the batch training model can be the result obtained by training the initial model using part of the data, which can be regarded as an intermediate result.

[0187] S603, using the initial training results to mark the target area of ​​the data pairs in the data set, and fine-tuning the parameters according to the feedback information of the marking results to obtain the final training results as the recognition model.

[0188] After the initial model is trained with partial data, the model can be further trained with the remaining data in the data set to obtain a recognition model. Because the data pairs with deviations in the recognition of the initial model have been learned in combination with the feedback information of manual verification, that is, the parameters of the initial model have been adjusted, higher accuracy can be achieved in the recognition process of a large amount of remaining data. For example, partial data can be 3,000 data pairs, and the remaining data can be 57,000 data pairs. In the case of a large number of remaining data, because the recognition accuracy is higher, it will not take too much time in the manual verification process, so that the labeling efficiency of the target area of ​​the overall data set shows a significant improvement trend.

[0189] In one example, the model was used to pre-label the remaining data, i.e., 57,000 data pairs, or all the data, i.e., 60,000 data pairs, to generate labels, and finally, combined with manual rechecking, the inaccurately segmented data was fine-tuned. While ensuring the quality of the data set annotation, the speed was greatly improved, and the original 30-day annotation time could be shortened to 7 days.

[0190] As an optional embodiment, after the recognition model is trained, the model is converted to TensorRT format and connected to the solder paste defect detection software. TensorRT optimizes the deep learning model structure, can provide high throughput and low latency reasoning deployment, and can achieve reasoning acceleration. In the solder paste detection process, the user sets the foreground confidence threshold, and inputs the RGB image and Depth image as required. The model outputs the mask image of the solder paste area based on the recognition result of the target area, which is used for subsequent calculation of solder paste attribute values, such as height, area, and volume, etc., and accurately screens out defective samples based on the comparison of the attribute value with the threshold.

[0191] The technical solution provided in this embodiment uses part of the data for initial training and uses other data or all the data for fine-tuning training, which can improve the training efficiency of the model, ensure the accuracy of the model training, and improve the model convergence speed.

[0192] In one embodiment, optionally, after obtaining the recognition model, the method further includes:

[0193] extracting a random number of samples from the data set;

[0194] Performing highly abnormal filtering on the random number of samples to obtain filtered data;

[0195] The recognition model is optimized using the filtered data to obtain an optimized recognition model.

[0196] The random number samples may be randomly selected from the entire data set, and the random number samples are subjected to highly abnormal filtering to obtain filtered data. Here, the number of random number samples may be 500, and 200 samples with high abnormality may be filtered out to obtain 300 data as filtered data.

[0197] After obtaining the filtered data, the filtered data can be used to train the recognition model. Specifically, the recognition model can be used to mark the target area of ​​the filtered data, and after obtaining the marking results, it can be provided to manual verification, so that inaccurate markings can be found in the manual verification process, and the actual range of the target area can be drawn, and fed back to the recognition model for learning, so as to adjust the parameters of the recognition model and obtain the adjusted recognition model.

[0198] Through such a setting, this scheme can use filtered data to re-train the recognition model, so as to avoid the recognition model carrying highly abnormal information in the data, which may cause some parameters of the recognition model to be biased, affecting the training results of the recognition model and affecting the accuracy of the target area output using the recognition model.

[0199] In a specific solution, the training phase mainly involves data preprocessing, data enhancement and training strategy.

[0200] Data preprocessing: Calculate the RGB normalization information of the data set, and count the mean and standard deviation of different channels of all RGB images in the data set. Determining a reasonable and robust normalization parameter can improve the generalization ability of the model and improve the training speed and stability of the model; Depth uses the min-max formula to normalize the data to the range of [0,1];

[0201] Data enhancement: Use RandomCrop, RandomFlip, RandomBlur, ColorJitter and other technologies to enhance the data and improve the diversity of data and the generalization of the model during training;

[0202] The training strategy adopts a small batch plus two-stage fine-tuning strategy. Based on the SegFast_RGBD model, a small batch of data is used to train the model. The model can accurately identify solder paste in most scenarios, but if abnormal height appears in the Depth data, the recognition accuracy will drop sharply. A two-stage fine-tuning strategy is adopted. On the basis of the obtained recognition model, the data set data_filter that filters out height anomalies is used to fine-tune the recognition model to obtain the final version of the recognition model, that is, the final model. The abnormal height area is noise, which affects the performance of the height feature module and causes the model to learn wrong knowledge.

[0203] This method uses a height anomaly recognition algorithm to automatically calculate the base plate height and solder paste height, identify whether there are collapsed and pointed areas, and if there are abnormal areas, remove the image index from the training list. After fine-tuning, the accuracy of model recognition can be increased by 5%, greatly improving recognition accuracy.

[0204] Figure 7 Schematic diagram of the process of solder paste defect detection method provided in the embodiment of the present application. Figure 7 As shown, the solder paste defect detection method identifies the solder paste area according to the target area identification method described in the above embodiments, and the solder paste defect detection method includes:

[0205] S701, obtaining parameters to be measured in the solder paste area;

[0206] The parameters to be measured may be height parameters of each position of the solder paste area, solder paste shape parameters of the solder paste area, and range parameters of the solder paste area, etc.

[0207] The height parameter of each position in the solder paste area may be obtained by acquiring the height data of each pixel point corresponding to the solder paste area from the height image, thereby obtaining the height data of each position in the solder paste area.

[0208] The solder paste shape parameter of the solder paste area can be the shape of the range covered by the entire solder paste area on the PCB board extracted as a shape parameter, or it can be a parameter that determines the spatial shape of the entire solder paste in the solder paste area in combination with the height of each pixel point in the solder paste area. This solution can determine the shape of the range covered by the solder paste area on the PCB board by obtaining the pixel coordinates of the solder paste area, or obtain the height data of each position and the pixel data of each position to obtain the spatial shape of the entire solder paste area.

[0209] The range parameter of the solder paste area may be the range of the solder paste area in the entire PCB image. Specifically, the area size of the solder paste area may be determined by the pixel position of each pixel point in the solder paste area.

[0210] S702, determining whether the parameters to be tested in the solder paste area meet a preset threshold;

[0211] The preset threshold may be a height parameter threshold, a shape parameter threshold, a range parameter threshold, etc. Specifically, when the height parameter is greater than a preset height parameter threshold, it may be determined that the preset height parameter threshold is satisfied; when the shape parameter is inconsistent with a preset shape parameter threshold, it may be determined that the preset shape parameter threshold is satisfied; and when the range parameter is greater than a preset range parameter threshold, it may be determined that the preset range parameter threshold is satisfied.

[0212] S703: When the parameter to be measured in the solder paste area meets a preset threshold, determine that a solder paste defect exists.

[0213] In this solution, the presence of a solder paste defect can be determined when the height parameter is greater than a preset height parameter threshold. The presence of a solder paste defect can also be determined when the shape parameter is inconsistent with the preset shape parameter threshold, such as when the original top of the solder paste should be a flat surface, but the measured top of the solder paste is an arc-shaped spherical surface. The presence of a solder paste defect can also be determined when the range parameter is greater than a preset range parameter threshold.

[0214] The technical solution provided in this embodiment, after identifying the solder paste area using the target area identification method described in the above embodiments, obtains the parameters to be measured in the solder paste area; determines whether the parameters to be measured in the solder paste area meet the preset threshold; and determines that there is a solder paste defect when the parameters to be measured in the solder paste area meet the preset threshold. By adopting this technical solution, the solder paste area of ​​the PCB board can be accurately identified based on the above-mentioned target area identification method, and the parameters to be measured in the solder paste area can be collected, which can provide a data basis for solder paste defect identification, provide accurate data information for identifying whether there are defects in the solder paste area, and also reduce the calculation amount of solder paste defect identification.

[0215] Figure 8 Schematic diagram of the structure of the target area recognition device provided in the embodiment of the present application. Figure 8 As shown, specifically including the following:

[0216] Image acquisition module 801, used to acquire a height image and a texture image of an identified object;

[0217] A feature extraction module 802 is used to extract features from the height image to obtain height image features, and to extract features from the texture image to obtain texture image features;

[0218] A probability generation module 803 is used to perform a fusion operation on the height image feature and the texture image feature, and obtain the probability of each module in the test area being the target area by using a recognition model;

[0219] The target region identification module 804 is used to determine the target region in the region to be detected according to a predetermined confidence threshold and the probability corresponding to each module.

[0220] The technical solution provided in this embodiment includes an image acquisition module for acquiring a height image and a texture image of an object to be identified; a feature extraction module for performing feature extraction on the height image to obtain height image features, and a feature extraction on the texture image to obtain texture image features; a probability generation module for performing a fusion operation on the height image features and the texture image features, and obtaining the probability that each module in the area to be tested is a target area using a recognition model; and a target area identification module for determining the target area in the area to be tested based on a predetermined confidence threshold and the probability corresponding to each module. By adopting the above technical solution, the height image and the texture image can be included in the input data at the same time, and detail features and semantic features can be extracted respectively. After the two features are aggregated, a mask image is obtained, and then combined with actual recognition needs, the solder paste area of ​​the PCB board can be accurately identified, which has a better scope of application and stronger robustness.

[0221] The target area recognition device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0222] The target area recognition device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0223] The target area recognition device provided in the embodiment of the present application can implement each process implemented by the above-mentioned method embodiments, and will not be described again here to avoid repetition.

[0224] Fig. 9 Schematic diagram of the structure of the solder paste defect detection device provided in the embodiment of the present application. The solder paste defect detection device identifies the solder paste area according to the target area recognition device as described above, such as Fig. 9 As shown, the solder paste defect detection device comprises:

[0225] The test parameter acquisition module 901 acquires the test parameters of the solder paste area;

[0226] A judgment module 902 is used to judge whether the parameter to be tested in the solder paste area meets a preset threshold;

[0227] The solder paste defect detection module 903 is used to determine the presence of a solder paste defect when the parameter to be measured in the solder paste area meets a preset threshold.

[0228] The technical solution provided in this embodiment is to identify the solder paste area through the target area identification device as described above, and the test parameter acquisition module is used to obtain the test parameters of the solder paste area; the judgment module is used to judge whether the test parameters of the solder paste area meet the preset threshold; the solder paste defect detection module is used to determine the existence of solder paste defects when the test parameters of the solder paste area meet the preset threshold. This technical solution can accurately identify the solder paste area of ​​the PCB board based on the above-mentioned target area identification method, and collect the test parameters of the solder paste area, which can provide a data basis for solder paste defect identification, provide accurate data information for identifying whether there are defects in the solder paste area, and also reduce the calculation amount of solder paste defect identification.

[0229] Fig.10 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig.10 As shown, an embodiment of the present application also provides an electronic device 1000, including a processor 1001, a memory 1002, and a program or instruction stored in the memory 1002 and executable on the processor 1001. When the program or instruction is executed by the processor 1001, each process of the above-mentioned target area recognition device embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0230] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0231] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned target area identification method embodiment and solder paste defect detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0232] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0233] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned target area identification method embodiment and the solder paste defect detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0234] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0235] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0236] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0237] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0238] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for identifying a target area, characterized in that: The method comprises: Acquire the height image and texture image of the area to be measured; Performing feature extraction on the height image to obtain height image features, and performing feature extraction on the texture image to obtain texture image features; The height image features and the texture image features are fused, and a recognition model is used to obtain the probability that each module in the tested area is a target area; wherein the recognition model is obtained by training the target area recognition based on a sample data pair, and the sample data pair is composed of a height image and a texture image; The target area in the area to be tested is determined according to a predetermined confidence threshold and a probability corresponding to each module.

2. The target area recognition method according to claim 1, characterized in that: Performing feature extraction on the height image to obtain height image features includes: Extracting features from the height image through a recognition model to obtain detail features and semantic features of the height image; Aggregating the height image detail feature and the height image semantic feature to obtain a height image feature; Extracting features from the texture image to obtain texture image features includes: Extracting features of the texture image through a recognition model to obtain detail features and semantic features of the texture image; The texture image detail features and the texture image semantic features are aggregated to obtain texture image features.

3. The target area recognition method according to claim 2, characterized in that: Aggregating the height image detail feature and the height image semantic feature to obtain a height image feature includes: Performing depth-separable convolution processing and conventional convolution processing on the height image detail features to obtain a first detail feature map; Performing conventional convolution processing and pooling processing on the height image detail features to obtain a second detail feature map; Performing conventional convolution processing and upsampling processing on the height image semantic features to obtain a first semantic feature map; Performing depth-separable convolution processing and conventional convolution processing on the height image semantic features to obtain a second semantic feature map; The first detail feature map is fused with the first semantic feature map to obtain a first aggregation factor; and the second detail feature map is fused with the second semantic feature map to obtain a second aggregation factor; Performing an aggregation operation on the first aggregation factor and the second aggregation factor to obtain a height image feature; Aggregating the texture image detail features and the texture image semantic features to obtain texture image features includes: Performing depth-separable convolution processing and conventional convolution processing on the detail features of the texture image to obtain a third detail feature map; Performing conventional convolution processing and pooling processing on the detail features of the texture image to obtain a fourth detail feature map; Performing conventional convolution processing and upsampling processing on the semantic features of the texture image to obtain a third semantic feature map; Performing depth-separable convolution processing and conventional convolution processing on the semantic features of the texture image to obtain a fourth semantic feature map; fusing the third detail feature map with the third semantic feature map to obtain a third aggregation factor; and fusing the fourth detail feature map with the fourth semantic feature map to obtain a fourth aggregation factor; An aggregation operation is performed on the third aggregation factor and the fourth aggregation factor to obtain a texture image feature.

4. The target area recognition method according to claim 3, characterized in that: The second detail feature map and the second semantic feature map are fused to obtain a second aggregation factor, including: The second detail feature map is fused with the second semantic feature map, and scaled according to a preset scaling factor to obtain a second aggregation factor, so that the size of the second aggregation factor is consistent with the size of the first aggregation factor; The fourth detail feature map is fused with the fourth semantic feature map to obtain a fourth aggregation factor, including: The fourth detail feature map and the fourth semantic feature map are fused and scaled according to a preset scaling factor to obtain a fourth aggregation factor, so that the size of the fourth aggregation factor is consistent with the size of the third aggregation factor.

5. The target area recognition method according to claim 1, characterized in that: The training process of the recognition model includes: Acquire a data set consisting of a preset number of data pairs; wherein each data pair includes a height image and a texture image; Using the initial model to annotate a target area of ​​a portion of the data in the data set, and obtaining an initial training result of the initial model; The target area of ​​the data pairs in the data set is annotated using the initial training results, and parameters are fine-tuned according to feedback information of the annotation results to obtain the final training results as a recognition model.

6. The target area recognition method according to claim 5, characterized in that: After obtaining the recognition model, the method further includes: extracting a random number of samples from the data set; Performing highly abnormal filtering on the random number of samples to obtain filtered data; The recognition model is optimized using the filtered data to obtain an optimized recognition model.

7. The target area recognition method according to claim 5, characterized in that: The initial model is a large visual detection model; the large visual detection model marks the target area of ​​part of the data based on pre-configured initialization information.

8. A method for detecting solder paste defects, characterized in that: The solder paste defect detection method identifies the solder paste area according to the target area identification method according to any one of claims 1 to 7, and the solder paste defect detection method includes: Get the parameters to be tested in the solder paste area; Determine whether the parameters to be tested in the solder paste area meet the preset threshold; When the parameter to be measured in the solder paste area meets a preset threshold, it is determined that a solder paste defect exists.

9. A target area recognition device, characterized in that: The device comprises: An image acquisition module is used to acquire a height image and a texture image of the identified object; A feature extraction module is used to extract features from the height image to obtain height image features, and to extract features from the texture image to obtain texture image features; A probability generation module, used to perform a fusion operation on the height image features and the texture image features, and obtain the probability of each module in the test area being a target area by using a recognition model; wherein the recognition model is obtained by training the target area recognition based on a sample data pair, and the sample data pair consists of a height image and a texture image; The target area identification module is used to determine the target area in the area to be tested according to a predetermined confidence threshold and the probability corresponding to each module.

10. A solder paste defect detection device, characterized in that: The solder paste defect detection device is used to identify the solder paste area according to the target area identification method described in any one of claims 1 to 7, and the solder paste defect detection device includes: The module for obtaining the parameters to be tested obtains the parameters to be tested in the solder paste area; A judgment module is used to judge whether the parameters to be tested in the solder paste area meet the preset threshold value; The solder paste defect detection module is used to determine the presence of a solder paste defect when the parameter to be measured in the solder paste area meets a preset threshold.

11. An electronic device, characterized in that: It includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for identifying a target area as described in any one of claims 1 to 7 is implemented, or the method for detecting solder paste defects as described in claim 8 is implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by the processor, the target area identification method described in any one of claims 1 to 7 is implemented, or the solder paste defect detection method described in claim 8 is implemented.

Images