Height data restoration method and device, equipment and medium

Abstract

The invention discloses a height data restoration method and device, equipment and a medium. The invention belongs to the technical field of image processing. The method comprises the steps that height data of a to-be-measured element and image data of the to-be-measured element are acquired, and the image data comprise RGB images; performing semantic segmentation of the to-be-detected element on the RGB image to obtain semantic segmentation data; and processing the semantic segmentation data, the height data and the RGB image by adopting a pre-trained repair model to obtain a repair result of the height data. By adopting the technical scheme, the semantic segmentation data, the height data and the RGB image are input into the repair model to be processed, the repair result of the height data can be obtained, and rapid, efficient and accurate height data anomaly repair is realized.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a method, device, equipment and medium for repairing height data. Background Art

[0002] Currently, PCB boards are one of the indispensable core components of various smart devices. 3D reconstruction technology is mainly used for PCB boards. By performing 3D reconstruction on PCB boards, a data basis can be provided for subsequent PCB board inspection. However, this technology is very sensitive to light signals. If the material is easily reflective or has gradient changes, the light signal may not be able to form a loop, resulting in imaging anomalies in the 3D reconstruction image based on structured light, which often cannot meet the needs of subsequent inspections.

[0003] In the prior art, a filtering algorithm is used to repair the areas with imaging anomalies in the 3D reconstructed image. However, the filtering algorithm can only compensate for very small areas with imaging anomalies, and cannot meet the needs of repairing areas with large-area collapse or peak anomalies. Therefore, how to achieve fast, efficient and accurate abnormal repair of height data is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, equipment and medium for repairing height data, which can solve the problem of imaging anomalies in height data and achieve fast, efficient and accurate abnormal repair of height data.

[0005] In a first aspect, an embodiment of the present application provides a method for repairing height data, the method comprising:

[0006] Acquire height data of the component to be tested and image data of the component to be tested, wherein the image data includes an RGB image;

[0007] Performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data;

[0008] The semantic segmentation data, the height data and the RGB image are processed using a pre-trained restoration model to obtain a restoration result of the height data.

[0009] This solution, by inputting semantic segmentation data, height data and RGB images into the repair model for processing, can obtain the repair result of the height data and realize fast, efficient and accurate abnormal repair of the height data.

[0010] Further, performing semantic segmentation of the DUT on the RGB image to obtain semantic segmentation data includes:

[0011] The RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested as semantic segmentation data; wherein the semantic segmentation data includes the electronic component area, the tin creeping area, and the PCB board area;

[0012] Accordingly, the semantic segmentation data, the height data and the RGB image are processed using a pre-trained restoration model to obtain a restoration result of the height data, including:

[0013] The semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the tin creeping area in the height data.

[0014] The beneficial effect of this solution is that by using an edge information extraction algorithm to extract edge information of the component to be tested, the boundaries of the body area and the connection area of ​​the component to be tested can be maintained intact, and at the same time a strong prior knowledge is provided for the repair model, so that the repair model can keep the boundaries of the body area and the connection area of ​​the component to be tested in the height data consistent with the boundaries of the body area and the connection area of ​​the component to be tested in reality.

[0015] Furthermore, the RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested, including:

[0016] Convert the RGB image into a target space image; wherein the target space image includes an L channel, an A channel, and a B channel;

[0017] For the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel;

[0018] Determine the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel and the B channel;

[0019] All scales are traversed, and the electronic components, tin creeping areas, and edge information between any two of the electronic components, tin creeping areas, and PCB boards included in the component to be tested are determined according to the gradient information of each scale.

[0020] The beneficial effect of this solution is that by determining the edge information of the component to be tested based on the gradient information in different directions on each channel at each scale, the detection capability for weak edges can be improved, and the edge detection capability in directions other than the horizontal and vertical directions can be improved, thereby better adapting the repair model of this solution.

[0021] Further, for the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel includes:

[0022] Constructing a structure tuple; wherein the structure tuple includes a structure element in at least one direction;

[0023] Gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel is determined based on the structure tuple.

[0024] The beneficial effect of this scheme is that by constructing a structure tuple and determining the gradient information of the target space image in different directions on each channel based on the structure tuple, the edge features can be described more comprehensively, the robustness to interference factors such as illumination changes and noise can be enhanced, and the determined edge information can be made more stable.

[0025] Further, determining the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel, and the B channel includes:

[0026] For any channel, taking the mean value of the gradient information of the current channel in the at least one direction as the gradient information of the current channel;

[0027] Traverse each channel to obtain the gradient information of each channel at the current scale;

[0028] The maximum value of the gradient information of the three channels at the current scale is used as the gradient information of the current scale.

[0029] The beneficial effect of this scheme is that by taking the mean of the gradient information of each channel in different directions as the gradient information of the channel, the noise or local changes in individual directions can be suppressed. By taking the maximum value of the gradient information of the three channels at the current scale as the gradient information of the current scale, the noise interference on other channels can be suppressed, helping to retain the strongest edge information.

[0030] Furthermore, all scales are traversed, and the edge information between the electronic components, the tin creeping area, and any two in the PCB board included in the component to be tested is determined according to the gradient information of each scale, including:

[0031] All scales are traversed, and the edge information between the electronic components, the tin creeping area, and any two in the PCB board included in the component to be tested is determined according to the mean value of the gradient information of each scale.

[0032] The beneficial effect of this scheme is that smaller-scale gradient information can better detect subtle edge details, while larger-scale gradient information can help capture a larger range of edge structures. By determining the mean of the gradient information at each scale as the edge information of the component to be tested, the accuracy and robustness of the edge information extraction algorithm can be improved.

[0033] Furthermore, based on the image data, semantic segmentation data between any two of the electronic components, the solder creeping area and the PCB board included in the component to be tested is obtained, including:

[0034] The RGB image is input into a pre-trained semantic segmentation model to obtain the electronic component area, the solder creeping area and the PCB board area; wherein the semantic segmentation model adopts a lightweight network FDDWNET architecture, uses Lovasz-Softmax Loss as the loss function, uses the distillation strategy of the Teacher-Student algorithm, and is obtained based on supervised training.

[0035] The beneficial effect of this solution is that by inputting image data into a pre-trained semantic segmentation model, the main body area and connection area of ​​the component to be tested can be obtained, providing clues for the subsequent repair model to repair the height data, thereby improving efficiency and accuracy.

[0036] Furthermore, the repair network adopts a dual encoding structure;

[0037] Accordingly, the semantic segmentation data, the height data, the RGB image and the edge information are input as input data to the repair network of the repair model to obtain the repair result of the height data, including:

[0038] The height data and the RGB image are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a first encoding structure to obtain a first encoding result;

[0039] The height data and the first encoding result are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a second encoding structure to obtain a second encoding result;

[0040] A fusion result is obtained by performing a fusion calculation based on the first encoding result and the second encoding result, and the fusion result is used as a restoration result.

[0041] The beneficial effect of this solution is that by using a restoration network with a dual encoding structure, semantic segmentation data, height data, RGB images and edge information can be deeply combined to improve the restoration efficiency and the accuracy of the restoration results.

[0042] Further, after obtaining a fusion result based on the fusion of the first encoding result and the second encoding result, the method further includes:

[0043] The fusion result and the height data are input into a decoding and redefinition network, and an output result based on the decoding and redefinition network is used as a restoration result.

[0044] The beneficial effect of this solution is that by inputting the fusion result and the height data into the decoding and redefinition network, and the decoding and redefinition network outputs the repair result, the problem of the overall height information of the fusion result being too large can be solved, and the accuracy of the repair result can be improved.

[0045] In a second aspect, an embodiment of the present application provides a device for repairing height data, the device comprising:

[0046] A data acquisition module, used to acquire height data of the component to be tested and image data of the component to be tested, wherein the image data includes an RGB image;

[0047] A semantic segmentation module, used to perform semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data;

[0048] The restoration result determination module is used to process the semantic segmentation data, the height data and the RGB image using a pre-trained restoration model to obtain a restoration result of the height data.

[0049] This solution, by inputting semantic segmentation data, height data and RGB images into the repair model for processing, can obtain the repair result of the height data and realize fast, efficient and accurate abnormal repair of the height data.

[0050] Furthermore, the semantic segmentation module is specifically used for:

[0051] The RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested as semantic segmentation data; wherein the semantic segmentation data includes the electronic component area, the tin creeping area, and the PCB board area;

[0052] Accordingly, the repair result determination module is specifically used for:

[0053] The semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the tin creeping area in the height data.

[0054] The beneficial effect of this solution is that by using an edge information extraction algorithm to extract edge information of the component to be tested, the boundaries of the body area and the connection area of ​​the component to be tested can be maintained intact, and at the same time a strong prior knowledge is provided for the repair model, so that the repair model can keep the boundaries of the body area and the connection area of ​​the component to be tested in the height data consistent with the boundaries of the body area and the connection area of ​​the component to be tested in reality.

[0055] Furthermore, the semantic segmentation module is specifically used for:

[0056] Convert the RGB image into a target space image; wherein the target space image includes an L channel, an A channel, and a B channel;

[0057] For the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel;

[0058] Determine the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel and the B channel;

[0059] All scales are traversed, and the electronic components, tin creeping areas, and edge information between any two of the electronic components, tin creeping areas, and PCB boards included in the component to be tested are determined according to the gradient information of each scale.

[0060] The beneficial effect of this solution is that by determining the edge information of the component to be tested based on the gradient information in different directions on each channel at each scale, the detection capability for weak edges can be improved, and the edge detection capability in directions other than the horizontal and vertical directions can be improved, thereby better adapting the repair model of this solution.

[0061] Furthermore, the semantic segmentation module is specifically used for:

[0062] Constructing a structure tuple; wherein the structure tuple includes a structure element in at least one direction;

[0063] Gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel is determined based on the structure tuple.

[0064] The beneficial effect of this scheme is that by constructing a structure tuple and determining the gradient information of the target space image in different directions on each channel based on the structure tuple, the edge features can be described more comprehensively, the robustness to interference factors such as illumination changes and noise can be enhanced, and the determined edge information can be made more stable.

[0065] Furthermore, the semantic segmentation module is specifically used for:

[0066] For any channel, taking the mean value of the gradient information of the current channel in the at least one direction as the gradient information of the current channel;

[0067] Traverse each channel to obtain the gradient information of each channel at the current scale;

[0068] The maximum value of the gradient information of the three channels at the current scale is used as the gradient information of the current scale.

[0069] The beneficial effect of this scheme is that by taking the mean of the gradient information of each channel in different directions as the gradient information of the channel, the noise or local changes in individual directions can be suppressed. By taking the maximum value of the gradient information of the three channels at the current scale as the gradient information of the current scale, the noise interference on other channels can be suppressed, helping to retain the strongest edge information.

[0070] Furthermore, the semantic segmentation module is specifically used for:

[0071] All scales are traversed, and the edge information between the electronic components, the tin creeping area, and any two in the PCB board included in the component to be tested is determined according to the mean value of the gradient information of each scale.

[0072] The beneficial effect of this scheme is that smaller-scale gradient information can better detect subtle edge details, while larger-scale gradient information can help capture a larger range of edge structures. By determining the mean of the gradient information at each scale as the edge information of the component to be tested, the accuracy and robustness of the edge information extraction algorithm can be improved.

[0073] Furthermore, the semantic segmentation module is specifically used for:

[0074] The RGB image is input into a pre-trained semantic segmentation model to obtain the electronic component area, the solder creeping area and the PCB board area; wherein the semantic segmentation model adopts a lightweight network FDDWNET architecture, uses Lovasz-Softmax Loss as the loss function, uses the distillation strategy of the Teacher-Student algorithm, and is obtained based on supervised training.

[0075] The beneficial effect of this solution is that by inputting image data into a pre-trained semantic segmentation model, the main body area and connection area of ​​the component to be tested can be obtained, providing clues for the subsequent repair model to repair the height data, thereby improving efficiency and accuracy.

[0076] Furthermore, the repair network adopts a dual encoding structure;

[0077] Accordingly, the repair result determination module is specifically used for:

[0078] The height data and the RGB image are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a first encoding structure to obtain a first encoding result;

[0079] The height data and the first encoding result are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a second encoding structure to obtain a second encoding result;

[0080] A fusion result is obtained by performing a fusion calculation based on the first encoding result and the second encoding result, and the fusion result is used as a restoration result.

[0081] The beneficial effect of this solution is that by using a restoration network with a dual encoding structure, semantic segmentation data, height data, RGB images and edge information can be deeply combined to improve the restoration efficiency and the accuracy of the restoration results.

[0082] Furthermore, the repair result determination module is specifically used to:

[0083] The fusion result and the height data are input into a decoding and redefinition network, and an output result based on the decoding and redefinition network is used as a restoration result.

[0084] The beneficial effect of this solution is that by inputting the fusion result and the height data into the decoding and redefinition network, and the decoding and redefinition network outputs the repair result, the problem of the overall height information of the fusion result being too large can be solved, and the accuracy of the repair result can be improved.

[0085] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0086] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0087] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0088] In an embodiment of the present application, the height data of the component to be tested and the image data of the component to be tested are obtained, and the image data includes an RGB image; the RGB image is semantically segmented to obtain semantic segmentation data; the semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the height data. The above-mentioned method for repairing height data can obtain a repair result of the height data by inputting the semantic segmentation data, the height data and the RGB image into the repair model for processing, thereby realizing fast, efficient and accurate abnormal repair of height data. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 is a flow chart of a method for repairing height data provided in an embodiment of the present application;

[0090] Figure 2 is a flow chart of a method for repairing height data provided in an embodiment of the present application;

[0091] Figure 3 is a flow chart of a method for repairing height data provided in an embodiment of the present application;

[0092] Figure 4 is a schematic diagram of a first coding structure and a second coding structure provided in an embodiment of the present application;

[0093] Figure 5 This is a schematic diagram of the decoding redefinition network input and output provided in an embodiment of the present application;

[0094] Figure 6 is a schematic diagram of the structure of a height data repair device provided in an embodiment of the present application;

[0095] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all of the content is shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0097] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0098] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0099] In the prior art, a filtering algorithm is used to repair the areas with imaging anomalies in the 3D reconstructed image. The filtering algorithm is a technology used for signal processing, which can be used to remove noise from the signal, smooth the signal, enhance the signal characteristics, or extract information of interest. However, the filtering algorithm can only compensate for very small areas with imaging anomalies, and cannot meet the needs of repairing areas with large-area collapse or peak anomalies.

[0100] In an embodiment of the present invention, semantic segmentation is first performed based on the RGB image of the component to be tested to obtain an accurate three-dimensional reconstructed image of the area where imaging abnormalities exist. By inputting the semantic segmentation data, height data and RGB image into a repair model based on a deep learning algorithm and a convolutional neural network block framework for processing, fast, efficient and accurate height data repair can be achieved.

[0101] Figure 1 FIG. 1 is a flow chart of a method for repairing height data provided in an embodiment of the present application. Figure 1 As shown, the specific steps include:

[0102] S101, acquiring height data of a component under test and image data of the component under test, wherein the image data includes an RGB image;

[0103] First, the application scenario of this solution can be to use structured light imaging technology for 3D reconstruction, and there are imaging anomalies. Based on the above application scenarios, it can be understood that the execution subject of this application can be an electronic device with image segmentation, image recognition and data processing capabilities, such as: smart terminals such as mobile phones, tablet computers, desktop computers and large conference display panels.

[0104] Among them, the component to be tested can be an object that needs to be reconstructed in three dimensions. At present, PCB boards are one of the indispensable core components of various smart devices. Three-dimensional reconstruction technology is mainly used in PCB boards. The component to be tested can refer to the produced PCB boards. By reconstructing the PCB boards in three dimensions, it can provide a data basis for subsequent testing of the PCB boards. Among them, the PCB board is a basic circuit board used to support and connect electronic components.

[0105] The height data may refer to the height values ​​of various positions on the component to be tested.

[0106] In one embodiment, the height data of the component to be measured can be obtained by using structured light imaging technology. Structured light imaging technology is a method for measuring and reconstructing three-dimensional objects. A light projector is used to project a specific light structure onto the component to be measured, and a camera is used to capture the light structure pattern formed on the surface of the component to be measured. By analyzing the deformation of the light structure on the surface of the component to be measured, the height data of the component to be measured can be determined.

[0107] The image data may refer to a color image containing the component to be tested, and may include an RGB image. Specifically, an RGB image is an image containing three channel information of R (red), G (green) and B (blue), and presents rich colors and details by combining the brightness values ​​of these three color channels.

[0108] In one embodiment, the image data can be collected by using a camera. The basic process of collecting image data by a camera includes: the optical sensor located behind the camera lens converts the light entering through the lens into an electrical signal; the photosensitive element in the optical sensor detects the intensity and color of the light and converts it into a voltage signal; the analog-to-digital converter converts the continuously changing analog voltage signal into a digital form for further processing and storage; the image processor processes and enhances the digital signal, and the processing may include color correction, contrast adjustment, and noise reduction to improve the image quality and clarity; the camera compresses the image data to reduce the space and bandwidth required for storage and transmission; the processed and compressed image data is output to the microprocessor chip of the computer through an interface (such as USB, HDMI, and network cable, etc.).

[0109] S102, performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data;

[0110] Semantic segmentation aims to segment an image into different regions or objects with linguistic meaning. Specifically, semantic segmentation can be used to determine the main body region and connection region of the component under test in the RGB image. Semantic segmentation data is the corresponding pixel region of the main body region and connection region of the component under test in the RGB image obtained after semantic segmentation of the RGB image.

[0111] In one embodiment, the semantic segmentation data may be obtained by inputting an RGB image into a pre-trained semantic segmentation model, and the semantic segmentation model outputs the main body area and the connection area of ​​the component to be tested.

[0112] In a feasible embodiment, optionally, based on the image data, obtaining semantic segmentation data between any two of the electronic components, the tin creeping area and the PCB board included in the component to be tested includes:

[0113] The RGB image is input into a pre-trained semantic segmentation model to obtain the electronic component area, the solder creeping area and the PCB board area; wherein the semantic segmentation model adopts a lightweight network FDDWNET architecture, uses Lovasz-Softmax Loss as the loss function, uses the distillation strategy of the Teacher-Student algorithm, and is obtained based on supervised training.

[0114] Among them, the main body area may refer to the area occupied by the electronic component area and the PCB board area in the RGB image, and the connection area may refer to the area occupied by the tin creeping area in the RGB image. For the PCB board, the tin creeping area may be the part where the three-dimensional reconstruction imaging anomaly exists. Specifically, the main body area may be the area occupied by the entire PCB board in the RGB image, or the area occupied by the electronic components in the RGB image, and the connection area refers to the area occupied by the tin creeping area of ​​the electronic components (chip components) of the PCB board in the RGB image. Among them, the tin creeping area of ​​the chip component refers to the contact area formed between the solder and the component pin or pad during the welding process.

[0115] Among them, the pre-trained semantic segmentation model can refer to a model that is trained on a large-scale image dataset and has learned the semantic segmentation task of the image. The model uses a deep learning algorithm, such as a convolutional neural network, and is trained with a large amount of labeled data to learn the mapping from the image to the pixel-level semantic label. The semantic segmentation model of this scheme adopts a lightweight network FDDWNET architecture, uses Lovasz-Softmax Loss as the loss function, adopts the distillation strategy of the Teacher-Student algorithm, and is obtained based on supervised training. The semantic segmentation model can eventually achieve an MIou of 0.92, and the average time to obtain the semantic segmentation data of each RGB image can be less than 3 milliseconds, realizing fast and accurate segmentation of the main body area and the connection area. Among them, Mlou (Mean Intersection over Union) is an indicator used to evaluate the performance of an image segmentation algorithm, which is used to measure the similarity between the image segmentation result and the true segmentation annotation.

[0116] Among them, the lightweight network FDDWNET architecture is a lightweight neural network architecture that uses FDDWC (factorized dilated depth-wise separable convolutions) to learn feature representations of receptive fields of different scales. Lightweight means having low computational complexity and number of parameters, and the neural network architecture refers to the overall structure and organization of the neural network model, including the hierarchical structure of the network, the connection between layers, and the choice of activation functions.

[0117] Among them, Lovasz-Softmax Loss is a loss function for multi-category classification tasks. The loss function is a function used in machine learning and deep learning to measure the difference between the model prediction results and the true label. The goal of the loss function is to minimize the prediction error of the model, so that the model can fit the training data more accurately and generalize to new unseen data.

[0118] Among them, the distillation strategy of the Teacher-Student algorithm is a model compression method that improves the performance of the student model by transferring the knowledge of a complex model (teacher model) to a simpler model (student model). The basic idea of ​​the distillation strategy is to let the student model learn the output distribution and decision boundary of the teacher model, rather than simply fitting the label of the teacher model.

[0119] Among them, supervised training is a commonly used learning paradigm in machine learning, which is used to learn the mapping relationship between input and output from labeled data. In supervised training, the training data set consists of a set of input samples and corresponding labels or target values, and the model performs prediction or classification tasks by learning the association between these samples and labels.

[0120] In one embodiment, the body area and the connection area of ​​the component to be tested can be obtained by inputting the RGB image into a pre-trained semantic segmentation model, and the computer calculates and feeds back the body area and the connection area of ​​the component to be tested.

[0121] This solution, by inputting image data into a pre-trained semantic segmentation model, can obtain the body area and connection area of ​​the component to be tested, providing clues for the subsequent repair model to repair the height data, thereby improving efficiency and accuracy.

[0122] S103, processing the semantic segmentation data, the height data and the RGB image using a pre-trained restoration model to obtain a restoration result of the height data.

[0123] The repair model may be a neural network model for repairing the height data of the component under test. The neural network model may be a machine learning model composed of multiple neurons that transmit and process information through connection weights. The neural network model may learn to extract features from input data and make predictions or decisions by adjusting connection weights.

[0124] The repair result of the height data is the output result of the repair model. Specifically, the repair result of the height data may refer to height data close to the actual height value.

[0125] In one embodiment, the method for obtaining the restoration result of the height data is to input the semantic segmentation data, the height data and the RGB image into the restoration model, and the computer calculates and feeds back the restoration result of the height data.

[0126] In a feasible embodiment, optionally, performing semantic segmentation of the DUT on the RGB image to obtain semantic segmentation data includes:

[0127] The RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested as semantic segmentation data; wherein the semantic segmentation data includes the electronic component area, the tin creeping area, and the PCB board area;

[0128] Accordingly, the semantic segmentation data, the height data and the RGB image are processed using a pre-trained restoration model to obtain a restoration result of the height data, including:

[0129] The semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the tin creeping area in the height data.

[0130] In the field of computer vision and image processing, edge information refers to information about the boundaries or transition areas between pixels in an image. An edge refers to a boundary or significant change between areas such as objects, textures, or colors in an image.

[0131] Among them, edge information extraction algorithm is an algorithm used to detect and extract edge information in images in the field of computer vision and image processing. Algorithm refers to a series of clearly defined steps or rules used to solve a specific problem or perform a specific task.

[0132] In one embodiment, the edge information of the component to be tested can be obtained by inputting the RGB image into a function program using an edge information extraction algorithm, and the computer calculates and feeds back the edge information of the component to be tested.

[0133] After obtaining the edge information, semantic segmentation data can be determined based on the edge information, that is, the electronic component area, solder creep area, and PCB board area that are actually obtained.

[0134] Accordingly, after the above data is input into the pre-trained repair model for processing, the repair result of the tin creeping area in the height data can be obtained, that is, the height repair of the tin creeping area can be achieved, avoiding large errors in the height recognition of the tin creeping area.

[0135] In one embodiment, the method for obtaining the restoration result of the height data is to input the semantic segmentation data, the height data, the RGB image and the edge information into the restoration model, and the computer calculates and feeds back the restoration result of the height data.

[0136] This scheme, by using an edge information extraction algorithm to extract edge information of the component to be tested, can maintain the boundaries of the body area and the connection area of ​​the component to be tested intact, and at the same time provide a strong prior knowledge for the repair model, so that the repair model can keep the boundaries of the body area and the connection area of ​​the component to be tested in the height data consistent with the boundaries of the body area and the connection area of ​​the component to be tested in reality.

[0137] The technical solution provided in the embodiment of the present application obtains the height data of the component to be tested and the image data of the component to be tested, wherein the image data includes an RGB image; the RGB image is semantically segmented to obtain semantic segmentation data; the semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the height data. The above-mentioned method for repairing height data can obtain a repair result of the height data by inputting the semantic segmentation data, the height data and the RGB image into the repair model for processing, thereby realizing fast, efficient and accurate abnormal repair of the height data.

[0138] As another specific solution, this solution can also be summarized as follows:

[0139] Acquire three-dimensional reconstruction result data of the component to be tested containing the solder creep area, and image data of the component to be tested, wherein the image data includes an RGB image;

[0140] Performing semantic segmentation of the component under test on the RGB image data to obtain semantic segmentation data of the solder creep area;

[0141] The semantic segmentation data, the three-dimensional reconstruction result data and the image data are processed by using a pre-trained restoration model to obtain a restoration result of the three-dimensional reconstruction result data of the tin creeping area.

[0142] The tin creeping area may refer to the tin creeping area of ​​the surface-mount component, specifically refers to the contact area formed between the solder and the component pin or pad during the welding process.

[0143] That is, this solution is to perform height repair on the 3D reconstruction data of the solder creeping area on the PCB board or other structures. The component to be tested can be an electronic component on the PCB board or a soldering area on the PCB board.

[0144] Figure 2 FIG. 1 is a flow chart of a method for repairing height data provided in an embodiment of the present application. Figure 2 As shown, the specific steps include:

[0145] S201, acquiring height data of a component under test and image data of the component under test, wherein the image data includes an RGB image;

[0146] S202, performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data;

[0147] S203, converting the RGB image into a target space image; wherein the target space image includes an L channel, an A channel, and a B channel;

[0148] The target space image may refer to image data in the LAB space. Specifically, the LAB space is a color space used to describe color information in an image.

[0149] The L channel represents the brightness information of the image, and its value range is usually 0 to 100, where 0 represents black and 100 represents white; the A channel represents the change from green to red, and its value range is from -128 to 127; the b channel represents the change from blue to yellow, and its value range is also from -128 to 127.

[0150] In one embodiment, the step of converting an RGB image into a target space image includes: linearizing the RGB values ​​of the RGB image, that is, dividing the values ​​of the RGB channels by 255 and normalizing them to the range [0, 1]; converting the linearized RGB values ​​into XYZ values ​​using a conversion matrix multiplication; converting the XYZ values ​​into LAB values ​​using a nonlinear transformation function; and dividing the image into three images, namely, an L channel image, an A channel image, and a B channel image, according to the LAB value.

[0151] S204, for the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel;

[0152] The scale can refer to the size or dimension of the convolution kernel used to calculate the gradient. The convolution kernel, also known as a filter or convolution matrix, is a commonly used tool in image processing and computer vision. It is a small matrix or window used to perform convolution operations on the input image. The convolution operation is a basic mathematical operation. In image processing, the convolution operation is often used for tasks such as image filtering, feature extraction, and image enhancement.

[0153] The gradient information specifically refers to the intensity of the pixel intensity change.

[0154] The direction may refer to a direction in a pixel coordinate system, and may include a horizontal direction, a vertical direction, and two oblique directions with an angle of 45 degrees to the horizontal direction.

[0155] In one embodiment, the method for obtaining the gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel can be to construct a structure tuple, and determine the gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel based on the structure tuple.

[0156] In a feasible embodiment, optionally, for the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel includes:

[0157] Constructing a structure tuple; wherein the structure tuple includes a structure element in at least one direction;

[0158] Gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel is determined based on the structure tuple.

[0159] Among them, the structure element is a template or window used in mathematical morphology operations to define the shape and size of the neighborhood considered during the operation. The structure element can be a small binary matrix or kernel, in which the elements can be 0 or 1. In mathematical morphology, the structure element is used to perform morphological operations such as erosion, dilation, opening and closing operations, which are used in shape analysis, edge detection, and image enhancement in image processing.

[0160] The structure tuple can be an array with structure elements as elements, and each structure element corresponds to a direction. An array is a data structure used to store a collection of elements of the same type. An array provides a continuous, fixed-size storage space, in which each element can be accessed by an index.

[0161] In one embodiment, the structure tuple may be constructed by defining an array with the structure element as the data type and a size of at least 2, and assigning each structure element to the array.

[0162] In one embodiment, the method of determining the gradient information of the target space image in at least one direction on the L channel, the A channel and the B channel based on the structure tuple can be adopted by calculating the gradient information of each structure element in the structure tuple in at least one direction on the L channel, the A channel and the B channel. Wherein, the method of calculating the gradient information of a structure element in one direction on one channel can be adopted by extracting the channel image of the target space image, and performing morphological erosion and morphological dilation operations according to the structure element and the channel image, and calculating the difference between the morphological erosion operation result and the morphological dilation operation result as the gradient information of the structure element on the channel. Wherein, the morphological erosion operation refers to using the structure element to compare the channel image pixel by pixel, and retaining the pixels that are completely contained in the channel image area of ​​the structure element, otherwise, it is set to 0; the morphological dilation operation refers to comparing the structure element with the channel image pixel by pixel, and as long as the structure element and the image have overlapping parts, the pixels in the image are set to the maximum value.

[0163] This scheme, by constructing a structure tuple and determining the gradient information of the target space image in different directions on each channel based on the structure tuple, can more comprehensively describe the edge features, enhance the robustness to interference factors such as illumination changes and noise, and make the determined edge information more stable.

[0164] S205, determining gradient information of a current scale according to gradient information of at least one direction on the L channel, the A channel, and the B channel;

[0165] In one embodiment, the method for determining the gradient information of the current scale may be to use the average value of the gradient information of the current channel in at least one direction for any channel as the gradient information of the current channel, traverse each channel to obtain the gradient information of each channel of the current scale, and use the maximum value of the gradient information of the three channels of the current scale as the gradient information of the current scale.

[0166] In a feasible embodiment, optionally, determining the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel, and the B channel includes:

[0167] For any channel, taking the mean value of the gradient information of the current channel in the at least one direction as the gradient information of the current channel;

[0168] Traverse each channel to obtain the gradient information of each channel at the current scale;

[0169] The maximum value of the gradient information of the three channels at the current scale is used as the gradient information of the current scale.

[0170] In one embodiment, assuming that the structure element of the current scale includes structure elements in four directions, and the gradient information of the target space image in each direction on the L channel has been calculated according to the above steps, namely gs1, gs2, gs3 and gs4, then the calculation formula of the gradient information l of the L channel of the current scale is:

[0171] l = (gs1 + gs2 + gs3 + gs4) / 4;

[0172] In one embodiment, by calculating the mean of the gradient information of each channel in different directions as the gradient information, the gradient information l, gradient information a and gradient information b of the current scale can be obtained. The gradient information l is the gradient information of the L channel of the current scale, the gradient information a is the gradient information of the A channel of the current scale, and the gradient information b is the gradient information of the B channel of the current scale.

[0173] In one embodiment, the max() function can be used to obtain the maximum value of the three gradient information of the current scale gradient information l, gradient information a and gradient information b, and the maximum value is used as the gradient information esi of the current scale. The calculation formula of the gradient information esi of the current scale is:

[0174] esi = max(l, a, b);

[0175] Where esi represents the gradient information of the i-th scale.

[0176] This scheme can suppress noise or local changes in individual directions by taking the mean of the gradient information of each channel in different directions as the gradient information of the channel. It can suppress noise interference on other channels by taking the maximum value of the gradient information of the three channels at the current scale as the gradient information of the current scale, thus helping to retain the strongest edge information.

[0177] S206, traversing all scales, and determining the edge information between any two of the electronic components, the solder creeping area, and the PCB board included in the component to be tested according to the gradient information of each scale.

[0178] In one embodiment, the edge information of the component to be tested may be determined by traversing all scales and determining the edge information of the component to be tested according to the average value of the gradient information at each scale.

[0179] In a feasible embodiment, optionally, all scales are traversed, and the edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested is determined according to the gradient information of each scale, including:

[0180] All scales are traversed, and the edge information of the component to be tested is determined according to the mean of the gradient information at each scale.

[0181] In one embodiment, assuming that three scales are set, and the gradient information of the three scales calculated according to the above steps are es1, es2, and es3, respectively, then the calculation formula of the edge information edge of the device under test is:

[0182] edge=(es1+es2+es3) / 3;

[0183] Smaller scale gradient information can better detect subtle edge details, while larger scale gradient information can help capture a wider range of edge structures. This solution can improve the accuracy and robustness of the edge information extraction algorithm by determining the mean of the gradient information of each scale as the edge information of the component to be tested.

[0184] S207: Process the semantic segmentation data, the height data, and the RGB image using a pre-trained restoration model to obtain a restoration result of the height data.

[0185] This solution determines the edge information of the component to be tested based on the gradient information in different directions on each channel at each scale, which can improve the detection capability for weak edges and the edge detection capability in directions other than the horizontal and vertical directions, thereby better adapting the repair model of this solution.

[0186] Figure 3 FIG. 1 is a flow chart of a method for repairing height data provided in an embodiment of the present application. Figure 3 As shown, the specific steps include:

[0187] S301, acquiring height data of a component under test and image data of the component under test, wherein the image data includes an RGB image;

[0188] S302, performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data;

[0189] S303, using an edge information extraction algorithm in a pre-trained repair model to process the image data to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested;

[0190] S304, taking the height data and the RGB image as original input data, and taking the semantic segmentation data and the edge information as intermediate layer input data, and inputting them into a first encoding structure to obtain a first encoding result;

[0191] The first encoding structure may be a micro CNN (Convolutional Neural Network) structure. A convolutional neural network is a deep learning model commonly used in image processing and computer vision tasks. It can effectively learn features and patterns in images through multi-layer convolution and pooling operations, as well as components such as fully connected layers and activation functions.

[0192] The original input data may be data input into the first coding structure for processing and analysis through the input layer of the first coding structure. The input layer is the first layer of the neural network, which is responsible for receiving external data and passing it to the next layer of the network for processing.

[0193] The intermediate layer input data may be data input to the intermediate layer in the first coding structure for processing and analysis. The intermediate layer refers to a neural network layer located between the input layer and the output layer, and mainly includes a convolution layer and a deconvolution layer. The convolution layer performs a linear transformation on the input operation through a convolution operation and introduces a nonlinear transformation through an activation function; the deconvolution layer is used to restore the convolution result to the original input space, that is, to reconstruct the image.

[0194] The first coding result may be data outputted through the output layer of the first coding structure after being processed and analyzed by the first coding structure. The output layer is the last layer of the neural network and is responsible for outputting the final result to the user or other systems.

[0195] S305, taking the height data and the first encoding result as original input data, and taking the semantic segmentation data and the edge information as intermediate layer input data, and inputting them into a second encoding structure to obtain a second encoding result;

[0196] Among them, the second encoding structure can be a micro CNN structure.

[0197] The original input data may be data that is input into the second encoding structure through an input layer of the second encoding structure for processing and analysis.

[0198] The intermediate layer input data may be data input into the intermediate layer in the second coding structure for processing and analysis.

[0199] The second encoding result may be data that is processed and analyzed by the second encoding structure and then output through the output layer of the second encoding structure.

[0200] Figure 4 Schematic diagram of the first coding structure and the second coding structure provided in the embodiment of the present application. Figure 4As shown, the height data and the RGB image are used as the original input data, and the semantic segmentation data and the edge information are used as the intermediate layer input data, which are input into the first encoding structure to obtain the first encoding result; the height data and the first encoding result are used as the original input data, and the semantic segmentation data and the edge information are used as the intermediate layer input data, which are input into the second encoding structure to obtain the second encoding result; when the second encoding structure performs a convolution operation, the deconvolution layer data in the first encoding structure is input into the convolution layer of the second encoding structure.

[0201] S306: Perform a fusion calculation based on the first encoding result and the second encoding result to obtain a fusion result, and use the fusion result as a restoration result.

[0202] The fusion result may be height data obtained by fusion calculation based on the first encoding result and the second encoding result.

[0203] In one embodiment, the fusion calculation may be performed by performing a weighted average calculation on the first encoding result and the second encoding result to obtain a fusion result.

[0204] like Figure 4 As shown, a fusion result is obtained by performing a fusion calculation based on the first encoding result and the second encoding result, and the fusion result is used as the restoration result.

[0205] This solution, by using a restoration network with a dual encoding structure, can deeply combine semantic segmentation data, height data, RGB images, and edge information to improve the restoration efficiency and the accuracy of the restoration results.

[0206] In a feasible embodiment, optionally, after obtaining a fusion result based on the fusion of the first encoding result and the second encoding result, the method further includes:

[0207] The fusion result and the height data are input into a decoding and redefinition network, and an output result based on the decoding and redefinition network is used as a restoration result.

[0208] Among them, the decoding redefinition network can be a CSPN decoding redefinition network. CSPN (Contextual SpatialPyramid Network) is a decoding redefinition structure for image segmentation tasks. It improves the accuracy and detail retention ability of image segmentation by utilizing context information and spatial pyramid structure.

[0209] Figure 5 Schematic diagram of the decoding redefinition network input and output provided by the embodiment of the present application. Figure 5 As shown, the fusion result and the height data are input into the decoding and redefinition network, and the decoding and redefinition network outputs the repair result.

[0210] This solution can solve the problem of the overall height information of the fusion result being too large and improve the accuracy of the repair result by inputting the fusion result and the height data into the decoding and redefinition network, and the decoding and redefinition network outputs the repair result.

[0211] Figure 6 Schematic diagram of the structure of the height data repair device provided in the embodiment of the present application. Figure 6 As shown, specifically including the following:

[0212] A data acquisition module 610 is used to acquire height data of the component under test and image data of the component under test, wherein the image data includes an RGB image;

[0213] A semantic segmentation module 620 is used to perform semantic segmentation of the DUT on the RGB image to obtain semantic segmentation data;

[0214] The restoration result determination module 630 is used to process the semantic segmentation data, the height data and the RGB image using a pre-trained restoration model to obtain a restoration result of the height data.

[0215] This solution, by inputting semantic segmentation data, height data and RGB images into the repair model for processing, can obtain the repair result of the height data and realize fast, efficient and accurate abnormal repair of the height data.

[0216] Furthermore, the semantic segmentation module 620 is specifically used for:

[0217] The RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested as semantic segmentation data; wherein the semantic segmentation data includes the electronic component area, the tin creeping area, and the PCB board area;

[0218] Accordingly, the repair result determination module 630 is specifically used for:

[0219] The semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the tin creeping area in the height data.

[0220] The beneficial effect of this solution is that by using an edge information extraction algorithm to extract edge information of the component to be tested, the boundaries of the body area and the connection area of ​​the component to be tested can be maintained intact, and at the same time a strong prior knowledge is provided for the repair model, so that the repair model can keep the boundaries of the body area and the connection area of ​​the component to be tested in the height data consistent with the boundaries of the body area and the connection area of ​​the component to be tested in reality.

[0221] Furthermore, the semantic segmentation module 620 is specifically used for:

[0222] Convert the RGB image into a target space image; wherein the target space image includes an L channel, an A channel, and a B channel;

[0223] For the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel;

[0224] Determine the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel and the B channel;

[0225] All scales are traversed, and the electronic components, tin creeping areas, and edge information between any two of the electronic components, tin creeping areas, and PCB boards included in the component to be tested are determined according to the gradient information of each scale.

[0226] The beneficial effect of this solution is that by determining the edge information of the component to be tested based on the gradient information in different directions on each channel at each scale, the detection capability for weak edges can be improved, and the edge detection capability in directions other than the horizontal and vertical directions can be improved, thereby better adapting the repair model of this solution.

[0227] Furthermore, the semantic segmentation module 620 is specifically used for:

[0228] Constructing a structure tuple; wherein the structure tuple includes a structure element in at least one direction;

[0229] Gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel is determined based on the structure tuple.

[0230] The beneficial effect of this scheme is that by constructing a structure tuple and determining the gradient information of the target space image in different directions on each channel based on the structure tuple, the edge features can be described more comprehensively, the robustness to interference factors such as illumination changes and noise can be enhanced, and the determined edge information can be made more stable.

[0231] Furthermore, the semantic segmentation module 620 is specifically used for:

[0232] For any channel, taking the mean value of the gradient information of the current channel in the at least one direction as the gradient information of the current channel;

[0233] Traverse each channel to obtain the gradient information of each channel at the current scale;

[0234] The maximum value of the gradient information of the three channels at the current scale is used as the gradient information of the current scale.

[0235] The beneficial effect of this scheme is that by taking the mean of the gradient information of each channel in different directions as the gradient information of the channel, the noise or local changes in individual directions can be suppressed. By taking the maximum value of the gradient information of the three channels at the current scale as the gradient information of the current scale, the noise interference on other channels can be suppressed, helping to retain the strongest edge information.

[0236] Furthermore, the semantic segmentation module 620 is specifically used for:

[0237] All scales are traversed, and the edge information between the electronic components, the tin creeping area, and any two in the PCB board included in the component to be tested is determined according to the mean value of the gradient information of each scale.

[0238] The beneficial effect of this scheme is that smaller-scale gradient information can better detect subtle edge details, while larger-scale gradient information can help capture a larger range of edge structures. By determining the mean of the gradient information at each scale as the edge information of the component to be tested, the accuracy and robustness of the edge information extraction algorithm can be improved.

[0239] Furthermore, the semantic segmentation module 620 is specifically used for:

[0240] The RGB image is input into a pre-trained semantic segmentation model to obtain the electronic component area, the solder creeping area and the PCB board area; wherein the semantic segmentation model adopts a lightweight network FDDWNET architecture, uses Lovasz-Softmax Loss as the loss function, uses the distillation strategy of the Teacher-Student algorithm, and is obtained based on supervised training.

[0241] The beneficial effect of this solution is that by inputting image data into a pre-trained semantic segmentation model, the main body area and connection area of ​​the component to be tested can be obtained, providing clues for the subsequent repair model to repair the height data, thereby improving efficiency and accuracy.

[0242] Furthermore, the repair network adopts a dual encoding structure;

[0243] Accordingly, the repair result determination module 630 is specifically used for:

[0244] The height data and the RGB image are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a first encoding structure to obtain a first encoding result;

[0245] The height data and the first encoding result are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a second encoding structure to obtain a second encoding result;

[0246] A fusion result is obtained by performing a fusion calculation based on the first encoding result and the second encoding result, and the fusion result is used as a restoration result.

[0247] The beneficial effect of this solution is that by using a restoration network with a dual encoding structure, semantic segmentation data, height data, RGB images and edge information can be deeply combined to improve the restoration efficiency and the accuracy of the restoration results.

[0248] Furthermore, the repair result determination module 630 is specifically used to:

[0249] The fusion result and the height data are input into a decoding and redefinition network, and an output result based on the decoding and redefinition network is used as a restoration result.

[0250] The beneficial effect of this solution is that by inputting the fusion result and the height data into the decoding and redefinition network, and the decoding and redefinition network outputs the repair result, the problem of the overall height information of the fusion result being too large can be solved, and the accuracy of the repair result can be improved.

[0251] The technical solution provided in the embodiment of the present application includes a data acquisition module for acquiring height data of the component to be tested and image data of the component to be tested, wherein the image data includes an RGB image; a semantic segmentation module for performing semantic segmentation of the component to be tested on the RGB image to obtain semantic segmentation data; and a repair result determination module for processing the semantic segmentation data, the height data and the RGB image using a pre-trained repair model to obtain a repair result of the height data. The above-mentioned height data repair device can obtain a repair result of the height data by inputting the semantic segmentation data, the height data and the RGB image into the repair model for processing, thereby realizing fast, efficient and accurate abnormal repair of the height data.

[0252] The device for repairing the height data in the embodiment of the present application can be a device, or a component, integrated circuit or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0253] The height data repairing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0254] The height data repairing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0255] Figure 7 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a program or instruction stored in the memory 702 and executable on the processor 701. When the program or instruction is executed by the processor 701, each process of the above-mentioned height data repair method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0256] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0257] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned height data repair method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0258] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0259] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned height data repair method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0260] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0261] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0262] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0263] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0264] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for repairing height data, characterized in that: The method comprises: Acquire height data of the component to be tested and image data of the component to be tested, wherein the image data includes an RGB image; Performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data; The semantic segmentation data, the height data and the RGB image are processed using a pre-trained restoration model to obtain a restoration result of the height data.

2. The method according to claim 1, characterized in that Performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data includes: The RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested as semantic segmentation data; wherein the semantic segmentation data includes the electronic component area, the tin creeping area, and the PCB board area; Accordingly, the semantic segmentation data, the height data and the RGB image are processed using a pre-trained restoration model to obtain a restoration result of the height data, including: The semantic segmentation data, the height data and the RGB image are processed using a pre-trained repair model to obtain a repair result of the tin creeping area in the height data.

3. The method according to claim 2, characterized in that The RGB image is processed using an edge information extraction algorithm in a pre-trained repair model to obtain edge information between any two of the electronic components, the tin creeping area, and the PCB board included in the component to be tested, including: Convert the RGB image into a target space image; wherein the target space image includes an L channel, an A channel, and a B channel; For the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel; Determine the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel and the B channel; All scales are traversed, and the electronic components, tin creeping areas, and edge information between any two of the electronic components, tin creeping areas, and PCB boards included in the component to be tested are determined according to the gradient information of each scale.

4. The method according to claim 3, characterized in that For the current scale, obtaining gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel includes: Constructing a structure tuple; wherein the structure tuple includes a structure element in at least one direction; Gradient information of the target space image in at least one direction on the L channel, the A channel, and the B channel is determined based on the structure tuple.

5. The method according to claim 3, characterized in that: Determining the gradient information of the current scale according to the gradient information of at least one direction on the L channel, the A channel, and the B channel, including: For any channel, taking the mean value of the gradient information of the current channel in the at least one direction as the gradient information of the current channel; Traverse each channel to obtain the gradient information of each channel at the current scale; The maximum value of the gradient information of the three channels at the current scale is used as the gradient information of the current scale.

6. The method according to claim 3, characterized in that Traverse all scales and determine the electronic components, tin creeping areas and edge information between any two in the PCB board included in the component to be tested based on the gradient information of each scale, including: All scales are traversed, and the edge information between the electronic components, the tin creeping area, and any two in the PCB board included in the component to be tested is determined according to the mean value of the gradient information of each scale.

7. The method according to claim 1, characterized in that Performing semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data includes: The RGB image is input into a pre-trained semantic segmentation model to obtain the electronic component area, the solder creeping area and the PCB board area; wherein the semantic segmentation model adopts a lightweight network FDDWNET architecture, uses Lovasz-SoftmaxLoss as the loss function, uses the distillation strategy of the Teacher-Student algorithm, and is obtained based on supervised training.

8. The method according to claim 1, characterized in that The repair network adopts a dual encoding structure; Accordingly, the semantic segmentation data, the height data, the RGB image and the edge information are input as input data to the repair network of the repair model to obtain the repair result of the height data, including: The height data and the RGB image are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a first encoding structure to obtain a first encoding result; The height data and the first encoding result are used as original input data, and the semantic segmentation data and the edge information are used as intermediate layer input data, and are input into a second encoding structure to obtain a second encoding result; A fusion result is obtained by performing a fusion calculation based on the first encoding result and the second encoding result, and the fusion result is used as a restoration result.

9. The method according to claim 8, characterized in that After obtaining a fusion result based on the fusion of the first encoding result and the second encoding result, the method further includes: The fusion result and the height data are input into a decoding and redefinition network, and an output result based on the decoding and redefinition network is used as a restoration result.

10. A device for repairing height data, characterized in that: The device comprises: A data acquisition module, used to acquire height data of the component to be tested and image data of the component to be tested, wherein the image data includes an RGB image; A semantic segmentation module, used to perform semantic segmentation of the component under test on the RGB image to obtain semantic segmentation data; The restoration result determination module is used to process the semantic segmentation data, the height data and the RGB image using a pre-trained restoration model to obtain a restoration result of the height data.

11. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the height data repair method as described in any one of claims 1 to 9.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the height data repair method according to any one of claims 1 to 9 are implemented.

Images