Height image determination method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a height image determination method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first height image which is used for displaying the height of a to-be-measured board card obtained through measurement, and the to-be-measured board card comprises a plurality of components; obtaining a corresponding second height image according to the first height image, wherein the resolution of the second height image is higher than that of the first height image; a reference image of the to-be-tested board card is obtained, the resolution of the second height image is equal to the resolution of the reference image, and the reference image is an RGB image obtained after the to-be-tested board card is shot; and processing the second height image and the reference image through a neural network model, and determining the features of the second height image by using the features of the reference image in the processing process to obtain a third height image after feature restoration. By adopting the method, the technical problem that the imaging precision of the height image is low when the resolution of the height image is consistent with the resolution of the RGB image in the prior art can be solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, equipment and storage medium for determining a height image. Background Art

[0002] Automated Optical Inspection (AOI) is a device that detects common defects encountered in welding production based on optical principles. When using AOI equipment to scan welded printed circuit boards (PCBs), 3D imaging is an extremely important step. When 3D imaging is performed, a height image of the PCB is required, which contains the height information obtained after the PCB is inspected.

[0003] At present, the resolution of the height image obtained by the AOI device is generally lower than the resolution of the RGB image obtained by the AOI device when shooting the PCB. In the related art, in order to facilitate subsequent inspections, the height image needs to be interpolated so that the resolution of the height image is consistent with the resolution of the RGB image, that is, the PCB is aligned in the two types of images. However, the imaging accuracy of small-sized components on the PCB in the height image after interpolation is low, and they are prone to present a "valley bag" shape in the height image after interpolation, that is, the originally sharp places in the components (such as edges) present a smooth effect in the height image after interpolation, so that the height information contained in the height image after interpolation is no longer accurate, which in turn affects the accuracy of subsequent inspections.

[0004] In summary, how to ensure the imaging accuracy of components in the height image while keeping the resolution of the height image consistent with that of the RGB image has become an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and storage medium for determining a height image to solve the technical problem in the related art that the imaging accuracy of the height image is low when the resolution of the height image is consistent with the resolution of the RGB image.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a height image, comprising:

[0007] Acquire a first height image, where the first height image is used to show the measured height of a board to be tested, where the board to be tested includes a plurality of components;

[0008] Obtaining a corresponding second height image according to the first height image, wherein a resolution of the second height image is higher than a resolution of the first height image;

[0009] Acquire a reference image of the board to be tested, wherein the resolution of the second height image is equal to the resolution of the reference image, and the reference image is an RGB image obtained by photographing the board to be tested;

[0010] The second height image and the reference image are processed by a neural network model, and during the processing, features of the reference image are used to determine features of the second height image to obtain a feature-restored third height image.

[0011] In the above, by obtaining a low-resolution first height image including the height of the board to be tested, a high-resolution second height image is obtained according to the first height image, and the resolution of the second height image is equal to the resolution of the RGB image obtained by photographing the board to be tested, and then the RGB image and the second height image are processed by a neural network model, and in the process, the features (specifically structural features) of the RGB image are used as a reference to determine the features (specifically structural features) of the second height image, and then the technical means of obtaining the third height image can solve the technical problem of low imaging accuracy of the height image when the resolution of the height image is consistent with the resolution of the RGB image in the related art. When the neural network model is processed, the high-frequency structural features of the RGB image are used as semantic supervision signals for guidance, and the height value in the second height image is corrected using a deep learning method to obtain a third height image with a more accurate height value, and then when the board to be tested is three-dimensionally reconstructed based on the third height image, the edges of the components on the board to be tested can be kept sharp, with a certain sharp repair effect, especially for small-sized components, the "valley bag" shape can be eliminated, and the accuracy of three-dimensional reconstruction can be improved. In addition, the neural network model has good robustness and can adapt to the influence of size, color, shape, light changes, etc. of different non-polar components. It can obtain more accurate height images and has the characteristics of high versatility.

[0012] In one embodiment of the present application, the method further includes:

[0013] Performing median filtering on the second height image to obtain a fourth height image;

[0014] The fourth height image and the third height image are fused to obtain a fifth height image that meets a preset condition, wherein the preset condition includes that an evaluation index corresponding to accuracy is greater than a preset threshold.

[0015] As mentioned above, by fusing the fourth height image and the third height image, that is, through the two-level fusion strategy, the effect of taking advantage of each other's strengths and making up for each other's weaknesses can be achieved, so that the height value shown in the fifth height image can maintain the smoothness of the plane, for example, maintain the smoothness of the solder joints on the board, and also maintain the sharpness of the edges of components, especially for small-sized components, it can avoid the "valley bag" shape, and has a certain correction ability for the slight collapse (i.e. loss) of the edge of the component.

[0016] In one embodiment of the present application, the second height image and the reference image are processed by a neural network model, and features of the reference image are used to determine features of the second height image during the processing to obtain a feature-restored third height image, including:

[0017] Using a preset segmentation rule to segment the reference image and the second height image respectively to obtain a plurality of reference image blocks and a plurality of first image blocks, wherein the reference image blocks correspond to the first image blocks one by one;

[0018] Inputting the reference image block and the corresponding first image block into a neural network model together;

[0019] The neural network model processes the reference image block and the first image block, determines the second structural feature of the first image block using the first structural feature of the reference image block during the processing, and obtains the second image block based on the first structural feature of the reference image block and the second structural feature of the first image block, each of the first image blocks has a corresponding second image block;

[0020] After obtaining the second image blocks corresponding to the first image blocks, the second image blocks are spliced ​​to obtain a feature-restored third height image.

[0021] As described above, the high-resolution second height image and the RGB image are segmented, and the segmented image blocks are processed using a neural network model, which can improve the processing speed of the neural network model.

[0022] In one embodiment of the present application, the neural network model includes a first extraction module, a second extraction module and a decoding module;

[0023] The neural network model is used to process the reference image block and the first image block, and during the processing, the first structural feature of the reference image block is used to determine the second structural feature of the first image block, and the second image block is obtained based on the first structural feature of the reference image block and the second structural feature of the first image block, including:

[0024] Extracting a first structural feature of the reference image block by the first extraction module;

[0025] Extracting a second structural feature of the first image block by the second extraction module in combination with the first structural feature;

[0026] The decoding module decodes the first structural features finally output by the first extraction module and the second structural features finally output by the second extraction module, and modifies the first image block based on the decoding result to obtain the second image block.

[0027] In one embodiment of the present application, the second extraction module includes a plurality of convolution blocks connected in series, and the convolution block includes a first depth-separable convolution submodule, a second depth-separable convolution submodule, a third depth-separable convolution submodule, an addition unit and a ghostnet submodule; the input of the first depth-separable convolution submodule and the input of the second depth-separable convolution submodule are used as the input of the convolution block, the output of the first depth-separable convolution submodule and the output of the second depth-separable convolution submodule are used as the input of the addition unit, the output of the addition unit is used as the input of the third depth-separable convolution submodule, the output of the third depth-separable convolution submodule is used as the input of the ghostnet submodule, and the output of the ghostnet submodule is used as the output of the convolution block.

[0028] As mentioned above, the convolution block for extracting the second structural feature in the neural network model is optimized using depthwise separable convolution and ghostnet, so that the convolution block is small enough, thereby making the neural network model lightweight, which can reduce the demand for GPU video memory, and the quality of the obtained third height image is better than the quality of the second height image obtained by the interpolation algorithm.

[0029] In one embodiment of the present application, acquiring the first height image includes:

[0030] Acquire multiple original height images of the board to be tested taken by multiple side cameras, each of the original height images corresponding to a different shooting angle;

[0031] The multiple original height images are fused to obtain a first height image.

[0032] In one embodiment of the present application, obtaining a corresponding second height image according to the first height image includes:

[0033] An interpolation calculation is performed on the first height image to obtain a second height image.

[0034] In a second aspect, an embodiment of the present application further provides a height image determination device, comprising:

[0035] A first acquisition unit is used to acquire a first height image, where the first height image is used to show the measured height of a board to be tested, where the board to be tested includes a plurality of components;

[0036] A first processing unit, configured to obtain a corresponding second height image according to the first height image, wherein a resolution of the second height image is higher than a resolution of the first height image;

[0037] A second acquisition unit is used to acquire a reference image of the board to be tested, wherein the resolution of the second height image is equal to the resolution of the reference image, and the reference image is an RGB image obtained by photographing the board to be tested;

[0038] The second processing unit is used to process the second height image and the reference image through a neural network model, and use the features of the reference image to determine the features of the second height image during the processing to obtain a feature-restored third height image.

[0039] In a third aspect, an embodiment of the present application further provides a height image determination device, including: one or more processors and a memory;

[0040] The memory is used to store one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the height image determination method as described in the first aspect.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the height image determination method as described in the first aspect.

[0043] The beneficial effects of the above-mentioned height image determination device, equipment and storage medium can refer to the beneficial effects of the height image determination method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a part of the board area provided in the related art;

[0045] Figure 2 A height image of a small-sized component provided in the related art;

[0046] Figure 3 Based on Figure 2 A first perspective view of a three-dimensional schematic diagram constructed from the height image shown;

[0047] Figure 4 Based on Figure 2A second perspective view of the three-dimensional schematic diagram constructed from the height image shown;

[0048] Figure 5 It is a schematic diagram of components obtained based on height image reconstruction in the related art;

[0049] Figure 6 A flowchart of a height image determination method provided for one embodiment of the present application;

[0050] Figure 7 An example diagram of a neural network model processing flow provided for one embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a neural network model provided in one embodiment of the present application;

[0052] Fig. 9 for Figure 8 A schematic diagram of the structure of the middle convolution block 421;

[0053] Fig.10 A schematic diagram of a training process of a neural network model provided for one embodiment of the present application;

[0054] Fig.11 A first schematic diagram of a processing flow of a height image determination method provided by an embodiment of the present application;

[0055] Fig.12 A first three-dimensional schematic diagram provided for an embodiment of the present application;

[0056] Fig.13 A second three-dimensional schematic diagram provided for an embodiment of the present application

[0057] Fig.14 A schematic diagram of a component provided for one embodiment of the present application;

[0058] Fig.15 A flowchart of a height image determination method provided for one embodiment of the present application;

[0059] Fig.16 A schematic diagram of a shooting area provided for an embodiment of the present application;

[0060] Fig.17 A second schematic diagram of a processing flow of a height image determination method provided by an embodiment of the present application;

[0061] Fig.18 A two-dimensional image of a board to be tested provided in one embodiment of the present application;

[0062] Fig.19 A three-dimensional image of a board to be tested provided in one embodiment of the present application;

[0063] Fig. 20 A schematic diagram of the structure of a height image determination device provided in one embodiment of the present application;

[0064] Fig.21 A schematic structural diagram of a height image determination device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0066] It should be noted that due to space limitations, this application specification does not list all optional implementation methods. After reading this application specification, those skilled in the art should be able to understand that as long as the technical structural features do not contradict each other, any combination of the technical structural features can constitute an optional implementation method.

[0067] Each embodiment is described in detail below.

[0068] In the related art, before the AOI equipment detects, a standard PCB (hereinafter referred to as a board) is first photographed to obtain an image containing the standard board, and then a board inspection program is prepared as the inspection standard. After the inspection begins, the board to be tested flows into the AOI equipment, which takes a photo of the board to be tested, and compares the board to be tested with the standard board according to the prepared inspection program, and then determines whether the board to be tested is qualified.

[0069] During the inspection process, when the AOI equipment scans the board, 3D imaging is an extremely important link. 3D imaging refers to the three-dimensional reconstruction of the board. Through 3D imaging, the defects of components in the board can be detected and identified, such as multiple parts, tin creep, volume, and tin connection. A high-quality imaging system is the key to 3D imaging. At present, the 3D imaging system of AOI equipment usually uses structured light imaging technology for three-dimensional reconstruction. Specifically, the 3D imaging system projects a structured light source and uses a camera to capture the surface deformation information of the object (currently the board), thereby realizing the three-dimensional reconstruction of the object. Structured light imaging technology has the characteristics of high precision, real-time and relative simplicity, but it has the characteristics of poor robustness and sensitivity to optical noise signals.

[0070] In the related art, after the AOI device uses structured light imaging technology to capture the deformation information of the board surface, it can generate a height image of the board. The height image shows the measured board, but each point (i.e., pixel) represents the height value of the corresponding position in the board, and different height values ​​are distinguished by different presentations, for example, different height values ​​are presented by different grayscales or different colors to achieve distinction. That is, the height image can be considered to be the imaging of the board by displaying the height of the board. The height information of the board and the components installed on it can be known through the height image. When the AOI device detects the board, it can also use a camera to shoot the board to obtain an RGB image. The RGB image can be considered as a clear color image obtained by shooting the board. Generally speaking, the resolution of the obtained height image is less than the resolution of the captured RGB image. In order to facilitate subsequent detection, the height image needs to be interpolated, such as using the CUBIC interpolation (i.e., cubic spline interpolation) algorithm to upsample the height image so that the resolution of the height image is equal to the resolution of the RGB image, thereby aligning the height information of the board in the height image with the board image in the RGB image.

[0071] However, interpolation is not very friendly to small-sized components in the height image. Before interpolation, the area occupied by small-sized components in the height image (which can also be understood as the number of pixels) is small, that is, very few height values ​​are used in the height image to represent the height of small-sized components. After interpolation, small-sized components will appear in the height image as a "bag" shape. The "bag" shape can be understood as a small-sized component appearing only as a bulging bag, without showing the original structure. For example, Figure 1 It is a schematic diagram of a part of the board area provided in the related art, Figure 2 It is a height image of a small-sized component provided in the related art. Figure 1 A partial area of ​​the board is shown. Components 11 and 12 in the area can be considered as small-sized components. After the height image corresponding to the board is interpolated, the style of component 11 in the height image can be referred to. Figure 2 (It should be noted that for ease of viewing, Figure 2 The height image in is the enlarged height image). Figure 2 In the height image shown, the height value change represented by the edge area 13 of the component 11 is not obvious (that is, the height value change is relatively smooth), and the actual height change of the edge of the component is not reflected, that is, the sharpness of the edge is not reflected. For ease of understanding, a three-dimensional schematic diagram is constructed based on the height value of the component 11 represented in the height image. Figure 3 Based on Figure 2 The first perspective view of the three-dimensional schematic diagram constructed from the height image shown, Figure 4 Based on Figure 2The second perspective of the three-dimensional schematic diagram constructed from the height image shown, refer to Figure 3 and Figure 4 , which shows from different perspectives based on Figure 2 A three-dimensional height map is generated by the height value of the component 11 in the height image shown, wherein: Figure 3 and Figure 4 The edge of the component 11 is relatively smooth, which does not truly reflect the sharp edge of the component 11. Figure 2 The height value of the component 11 in the height image shown is used as a reference. In the image containing the component 11, the edge of the component 11 is relatively blurred and does not show sharp edges. For example, Figure 5 This is a schematic diagram of components obtained based on height image reconstruction in the related art, refer to Figure 5 When reconstructing small-sized components based on the height image after interpolation, the small-sized components are Figure 5 The edge region 14 in the image shown is relatively smooth. Figure 5 The image shown may affect the detection accuracy when detecting small-sized components. In addition, in the height image after interpolation processing, the edges of components other than small-sized components may also have the negative effect of over-smoothing, and the height image after interpolation processing may also show a blurred effect, that is, when the resolution of the height image is consistent with the resolution of the RGB image, the imaging accuracy of the height image is low, which is not conducive to subsequent detection.

[0072] Based on this, a height image determination method is provided in an embodiment of the present application. The height image determination method uses deep learning technology to construct a neural network model to process the height image, and uses an RGB image for guidance during the processing, that is, the structure of the components on the board in the RGB image is used to correct the height value in the height image, so that the processed height image has higher clarity and tries to maintain the original structure of the components, that is, the height value of the component in the height image can reflect the sharpness of the edge of the component to eliminate the "valley bag" shape of the component.

[0073] The height image determination method provided in the embodiment of the present application can be performed by a height image determination device, which can be a physical entity or composed of multiple physical entities. Currently, the height image determination device can be an electronic device with functions such as image processing and neural network operation. In one embodiment, the height image determination device is described as an AOI device.

[0074] The height image determination device includes one or more processors. The processor may be one or more of a microcontroller unit (MCU) or a central processing unit (CPU). The processor can implement various functions of the height image determination device by running a computer executable program (i.e., a computer program, including software programs, instructions, and modules, etc.).

[0075] When the processor has a storage function, the computer executable program required during the operation of the height image determination device can be stored in the processor, that is, the computer executable program for implementing the height image determination method is stored in the processor. Alternatively, the height image determination device also includes a memory, which, as a storage medium, can store various computer executable programs, operating systems, and other contents required during the operation of the height image determination device for the processor to read and run, so that the processor can implement the height image determination method.

[0076] In addition, the height image determination device may also include other devices. For example, when the AOI device is used as a height image determination device, it also includes a driving module, a shooting module, a transmission track, and a display screen. The driving module can drive the corresponding target to move or move according to the instructions of the processor. For example, a driving module can drive the transmission track to transmit according to the instructions of the processor to realize the board in and out of the board. Another driving module can control the shooting device to move in the vertical direction according to the instructions of the processor. Another driving module can control the shooting device to move in the horizontal direction according to the instructions of the processor. Generally speaking, the driving module at least includes components such as a motor for realizing operation or movement. The shooting device can realize the shooting function. For example, the shooting device shoots the board to obtain a height image, an RGB image, etc. for three-dimensional reconstruction. The number of shooting devices is not currently limited. The transmission track is used to transmit the board. The transmission track and the height image determination device can be a physical entity or different physical entities. The transmission track at least includes a conveyor belt. The number of display screens is at least one. The display screen realizes the display function, and its display content is determined by the processor. The size and type of the display screen are not currently limited. In an optional manner, the display screen may also have a touch function, and in this case, the touch function may be implemented by infrared touch, capacitive touch, electromagnetic touch and / or resistive touch, etc. The user interacts with the height image determination device through the content displayed on the display screen.

[0077] When the height image determination device executes the height image determination method, the implementation process can refer to Figure 6 . Figure 6 A flowchart of a method for determining a height image is provided in one embodiment of the present application, referring to Figure 6 , the height image determination method includes steps 210 to 240:

[0078] Step 210: Acquire a first height image, where the first height image is used to show the measured height of the board to be tested, where the board to be tested includes a plurality of components.

[0079] Exemplarily, the first height image refers to a height image obtained after the height image determination device scans (i.e., measures) the board to be tested using structured light imaging technology. The first height image can reflect the measured height of the board to be tested and the height of the components installed on the board to be tested. Currently, the board to be tested refers to the board to be tested (i.e., PCB), and multiple components are installed on the board to be tested, and the multiple components can be different types of components. In an embodiment, the components installed on the board to be tested include at least small-sized components, for example, 01005-type components and 0201-type components can both be considered small-sized components.

[0080] It can be understood that the first height image used in the embodiment includes the entire area of ​​the board to be tested. In actual applications, the first height image may also include only a partial area of ​​the board to be tested.

[0081] Exemplarily, the first height image is generated by the height image determination device and then acquired and used by the height image determination device. The generation rule of the first height image is not currently limited. For example, the height image determination device can project structured light onto the board to be tested and use a camera sensor (i.e., a shooting device) to capture deformation information on the surface of the board to be tested, thereby obtaining the first height image of the board to be tested. In one embodiment, the use of multiple camera sensors to capture deformation information on the surface of the board to be tested is used as an example for description. In this case, step 210 includes steps 211-212:

[0082] Step 211: Acquire multiple original height images of the board to be tested taken by multiple side cameras, each original height image corresponding to a different shooting angle.

[0083] The original height image refers to the height image obtained based on the captured spectral image after projecting structured light onto the board to be tested. The number of original height images is related to the number of camera sensors that capture structured light in the height image determination device. For example, there are 4 camera sensors in the height image determination device that capture structured light, so there are 4 original height images in total, and one original height image can be obtained based on the structured light captured by each camera sensor. It can be understood that the resolution of each original height image is the same. Each point on the board to be tested (including points on the components to be tested) has a corresponding pixel in each original height image, and the pixel can reflect the height value of the point.

[0084] The device capable of projecting structured light is installed in the height image determination device, and when the board to be tested enters the height image determination device, the device capable of projecting structured light is located directly above the board to be tested, so as to project structured light from the center to the board to be tested. After projecting the structured light, the board to be tested projecting the structured light is photographed by the camera sensor to obtain an image containing the structured light, that is, a spectral image. At present, the height image determination device is equipped with multiple camera sensors, and each camera sensor is installed at a different position in the height image determination device to achieve shooting from different angles (i.e., shooting angles). Generally, the camera sensor is located above the side of the board to be tested in different directions. For example, the height image determination device is equipped with 4 camera sensors, and the installation positions of the 4 camera sensors are respectively located in the front, back, left, right and four directions of the device capable of projecting structured light, so as to shoot from four directions respectively. At this time, the original height image obtained based on the spectral image shot by each camera sensor corresponds to different shooting angles. In the embodiment, the camera sensor currently used is recorded as a side camera, that is, it is used to shoot the board to be tested from the side of the board to be tested, and each side camera corresponds to a shooting angle.

[0085] At present, the original height image is obtained by the moiré fringe light three-dimensional reconstruction technology. At this time, the structured light projected by the device that can project structured light is a multi-frequency moiré fringe (also known as moiré fringe), and generally five moiré fringe of different frequencies can be projected. After that, the spectral image corresponding to the moiré fringe at each frequency is captured by the side camera. After the height image determination device obtains multiple spectral images captured by the side camera, it performs phase resolution based on the multiple spectral images, and can obtain the original height image of the board to be tested at the shooting angle corresponding to the side camera. It can be understood that the moiré fringe light three-dimensional reconstruction technology is already a relatively common technology, and the embodiment will not be repeated. Among them, the spectral formula of the moiré fringe is f=a+bcosx. a represents the offset, which can be understood as the basic brightness, and the general value is 128. b represents the amplitude of the moiré fringe, which can be understood as the degree of brightness and darkness, and the general value range is 0-127. The larger b is, the higher the brightness is, and the higher the confidence of the corresponding determined height value (located in the original height image). At this time, each point in the original height image obtained has a corresponding confidence, and the value of b can be used as the confidence. x represents the phase of the moiré fringe.

[0086] Step 212: fuse the multiple original height images to obtain a first height image.

[0087] Exemplarily, multiple original height images are fused to obtain a height image (i.e., the first height image). During fusion, the confidence corresponding to each point in each original height image (i.e., the value of b) can be considered. The fusion method is not currently limited. For example, taking a point in the board to be tested as an example, the height value and confidence of the corresponding pixel of the point in each original height image are obtained, and then, the height value with the highest confidence is selected as the height value of the point. After obtaining the height values ​​of all points in the board to be tested according to this method, the first height image can be obtained. For another example, the height value and confidence corresponding to the point in each original height image are obtained, and then, the height value is weighted averaged in combination with the confidence, and the height value obtained is used as the height value of the point. After obtaining the height values ​​of all points in the board to be tested according to this method, the first height image can be obtained. In practical applications, fusion can also be performed in other ways.

[0088] It is understandable that in practical applications, the first height image can also be obtained by other means. For example, only one camera sensor is used for shooting, and the obtained original height image is recorded as the first height image. Alternatively, the height image determination device can project other types of structured light, and obtain the first height image based on the other types of structured light.

[0089] Step 220: Obtain a corresponding second height image according to the first height image, wherein the resolution of the second height image is higher than the resolution of the first height image.

[0090] In one embodiment, the height image determination device is also equipped with a shooting device for obtaining an RGB image. After the board to be tested enters through the transmission track, the height image determination device not only uses the structured light imaging technology to obtain the first height image, but also shoots the board to be tested to obtain an RGB image. Among them, the resolution of the RGB image is relatively high, generally higher than the resolution of the first height image, to ensure that the board to be tested is clearly imaged in the RGB image. In order to facilitate the detection of the board to be tested, it is necessary to increase the resolution of the first height image to the same as the resolution of the RGB image, so that each point of the board to be tested in the first height image corresponds to each point of the board to be tested in the RGB image, that is, the area of ​​the pixel points occupied by the board to be tested in the first height image is exactly the same as the area of ​​the pixel points occupied in the RGB image, presenting an alignment effect. In the embodiment, the height image obtained after the resolution of the first height image is increased is recorded as the second height image. The resolution of the second height image is the same as the resolution of the RGB image.

[0091] At this time, it can be considered that the first height image belongs to a low-resolution image, and the second height image belongs to a high-resolution image, that is, the resolution of the second height image is higher than that of the first height image.

[0092] The process of improving the resolution of the first height image can be considered as a super-resolution process or an upsampling process, and its specific implementation process can refer to the existing super-resolution process or upsampling process. In one embodiment, the process of improving the resolution by an interpolation algorithm is described as an example. At this time, step 220 may specifically include: performing interpolation calculation on the first height image to obtain a second height image. At present, the interpolation algorithm used in the interpolation calculation can be selected in combination with actual conditions. For example, after obtaining the resolution of the RGB image (the resolution can be pre-stored in the height image determination device for use), the first height image is upsampled using the CUBIC interpolation algorithm, and the resolution of the second height image obtained after upsampling is the same as the resolution of the RGB image. Among them, the CUBIC interpolation algorithm is an existing technical means, and the embodiment will not be repeated here. It can be understood that since the resolution of the first height image is relatively small, the number of height values ​​that can be used in the first height image is limited (this number is related to the resolution). Therefore, the edge information (i.e., height values) of the components in the second height image after interpolation is not restored (i.e., the edge information is obtained after interpolation of a limited number of height values). At this time, the missing information is magnified and the clarity remains unchanged, especially the height values ​​of the edges of small components are over-smoothed.

[0093] Step 230: Acquire a reference image of the board to be tested. The resolution of the second height image is equal to the resolution of the reference image. The reference image is an RGB image obtained by photographing the board to be tested.

[0094] In the embodiment, the RGB image obtained by the above shooting is recorded as a reference image. After the height image determination device shoots the RGB image, the RGB image is obtained for use. It should be noted that in actual applications, the reference image can also be obtained in step 210, or before or after step 210, or in step 220. In the embodiment, there is no limitation on the implementation node for obtaining the reference image.

[0095] It can be understood that when the first height image is obtained through multiple original height images, the RGB image can also be referenced, that is, the RGB image can be used for constraints. For example, when structured light is projected onto the board to be tested, the structured light projected on a certain area on the board to be tested may be blocked, so that the height value obtained based on the structured light may be inaccurate, that is, the height value represented by the corresponding point in the original height image may be inaccurate. At this time, the RGB image can be combined to determine the component (or the board surface) where the point is located, and then the height value of the point can be repaired based on the height values ​​of other points on the same surface of the component to maximize the accuracy of the height value.

[0096] Step 240: Process the second height image and the reference image through a neural network model, and use the features of the reference image to determine the features of the second height image during the processing to obtain a feature-restored third height image.

[0097] In one embodiment, in order to avoid the appearance of a "valley bag" shape of components in the second height image and ensure that the height change of the edge of the component can be accurately reflected in the height image, after obtaining the second height image, artificial intelligence (AI) technology is also used to process the second height image, and during the processing, the structural information of the component presented in the RGB image (which can reflect the sharpness of the edge) is used for guidance to ensure that the height value represented by the edge area of ​​the component in the height image obtained after processing can reflect the real height change of the edge of the component. In the embodiment, the height image obtained after processing the second height image is recorded as the third height image.

[0098] In one embodiment, when the second height image is processed using AI technology, a neural network model is specifically constructed, and processing is performed based on the neural network model to obtain a third height image. Currently, the constructed neural network model is a deep learning network, which can be a convolutional neural network or other types of neural networks. The specific network structure is not currently limited. The neural network model is deployed in the height image determination device for use.

[0099] Figure 7 An example diagram of a neural network model processing flow provided for one embodiment of the present application. Figure 7 RGB refers to the RGB image, depth_L refers to the first height image, depth_U refers to the second height image, and depth_H refers to the third height image. Figure 7After obtaining the second height image through the first height image, the neural network model processes the second height image and the RGB image to extract the features of the RGB image and the features of the second height image. Among them, when the neural network model extracts the features of the RGB image, it specifically extracts the structural features of the RGB image. The structural features are high-frequency features, which can reflect the superficial / deep edge information, texture information and other structural information in the RGB image, that is, the structure of the edges of each component on the board to be tested in the RGB image can be reflected in the structural features. When the neural network model extracts the features of the second height image, it specifically extracts the structural features of the components in the second height image, which can reflect the superficial / deep edge information, texture information and other structural information in the height image, that is, the structural features can reflect the structural information related to the edges of the components in the second height image, such as the structural information related to the corners of the edge, the structural information related to the slope, etc. In one embodiment, when extracting the structural features of the second height image, the neural network model also uses the structural features of the RGB image as a guide, that is, the structural features of the RGB image extracted by the neural network model can be used by the network in the neural network model that extracts the structural features of the second height image, that is, the structural features of the second height image are obtained by combining the structural features of the RGB image, which can ensure that the extracted structural features of the second height image can more accurately reflect the structural information of the components, that is, more accurately reflect the sharpness of the edges of the components. In one embodiment, when the neural network model extracts the structural features of the RGB image, it will extract structural features at different depths. At this time, some or all of the structural features (at different depths) can be used by the network that extracts the structural features of the second height image.

[0100] Afterwards, the neural network model inputs the structural features of the RGB image, the structural features of the second height image, and the second height image into the decoder, so that the decoder decodes the structural features of the RGB image and the structural features of the second height image, and uses the second height image during decoding, that is, based on the extracted structural features, the height value in the second height image is corrected to obtain a third height image. The resolution of the third height image is equal to the resolution of the second height image. Based on Figure 7 The processing flow shown in the figure, the algorithm executed by the neural network model can be expressed as the following formula:

[0101] depth H =depth U +F(depth U , f(RGB); θ ) (1)

[0102] Among them, θ represents the learnable parameters of the neural network model, which are determined during the training process of the neural network model. H Indicates the third height image, depth Urepresents the second height image, RGB represents the RGB image, f(RGB) represents the structural features of the RGB image extracted by the neural network model, and F(depth U ,f(RGB);θ) represents the structural features of the second height image extracted by the neural network model. Based on formula (1), it can be seen that the neural network model is guided by f(RGB) when extracting the structural features of the second height image. Afterwards, the second height image is corrected based on the structural features of the second height image (specifically, the height value is corrected) to obtain the third height image. It can be understood that the components on the board to be tested are relatively clear in the RGB image. Therefore, the structural features of the RGB image extracted by the neural network model can more clearly reflect the structural information of the real edge of the components. The structural features of the second height image guided by the structural features of the RGB image can also more accurately reflect the structural information of the edge of the components. Therefore, the height value of the edge of the component in the third height image obtained based on the guided structural features is more accurate, and the "valley bag" shape can basically be avoided, so that the height change of the edge area of ​​the component in the third height image can meet the actual situation, that is, the structural information in the third height image is feature repaired.

[0103] It is understandable that in order to obtain clear images of the board under test, the resolution of the RGB image is generally larger. For example, the resolution of the RGB image is 4000×3000. In order to match the resolution of the RGB image, the resolution of the second height image also needs to be 4000×3000. When the neural network model is used directly to process the RGB image and the second height image, the demand for the graphics processing unit (GPU) video memory is very high, which is not conducive to the deployment and operation of the neural network model in the GPU. Based on this, in one embodiment, before using the neural network model for processing, the RGB image and the second height image are first segmented to obtain multiple image blocks. After that, the image blocks are processed by the neural network model, that is, the resolution of the image input to the neural network model is reduced to reduce the demand for GPU video memory. At this time, step 240 may include steps 241-244:

[0104] Step 241 : Use a preset segmentation rule to segment the reference image and the second height image respectively to obtain a plurality of reference image blocks and a plurality of first image blocks, wherein the reference image blocks correspond to the first image blocks one by one.

[0105] Exemplarily, the preset segmentation rule can be set in combination with the actual situation. Currently, the preset segmentation rule is that the image blocks obtained after segmentation are all of a preset size. For example, the preset size is a resolution of 1000×1000. At this time, after the RGB image is segmented according to the preset segmentation rule, the resolution of each image block (i.e., small image) obtained is 1000×1000. Currently, the image block obtained after segmenting the RGB image is recorded as a reference image block, and the image block obtained after segmenting the second height image is recorded as a first image block. For example, the resolutions of the RGB image and the second height image are both 4000×3000, and the resolution of the image block is 1000×1000. After segmenting the RGB image and the second height image, 12 reference image blocks and 12 first image blocks can be obtained respectively.

[0106] Since the resolution of the RGB image and the second height image is the same, after segmentation according to the preset segmentation rule, each reference image block has a corresponding first image block, that is, the reference image block and the corresponding first image block show the same area of ​​the board to be tested.

[0107] Step 242: Input the reference image block and the corresponding first image block into the neural network model together.

[0108] Exemplarily, after obtaining the reference image block and the first image block, the neural network model can be used to process the reference image block and the first image block. Each time, one reference image block and the corresponding first image block are input into the neural network model. After the neural network model is processed, another reference image block and the corresponding first image block are input into the neural network model. This process is repeated to enable the neural network model to process each reference image block and the corresponding first image block.

[0109] Step 243: The neural network model processes the reference image block and the first image block. During the processing, the first structural feature of the reference image block is used to determine the second structural feature of the first image block, and the second image block is obtained based on the first structural feature of the reference image block and the second structural feature of the first image block. Each first image block has a corresponding second image block.

[0110] After the neural network model processes the reference image block and the first image block, the output image has the same resolution as the first image block. Currently, the image output by the neural network model is recorded as the second image block, each first image block has a corresponding second image block, and the first image block and the second image block show the same board area. Compared with the first image block, the second image block is guided by the structural features of the reference image block, so the height value of the edge of the component it reflects is more accurate, and the height change of the edge of the component in the three-dimensional model reconstructed based on the height value of the edge is more in line with reality, that is, the edge remains sharp.

[0111] In one embodiment, in order to facilitate understanding of the processing process of the neural network model, the structure of the neural network model is first exemplified. Figure 8 This is a schematic diagram of the structure of a neural network model provided in one embodiment of the present application, refer to Figure 8 The neural network model 40 includes a first extraction module 41, a second extraction module 42 and a decoding module 43.

[0112] The first extraction module 41 can extract the structural features of the reference image block and provide them for use by the second extraction module 42. In the embodiment, the extracted structural features of the reference image block are recorded as the first structural features. The first extraction module 41 includes a plurality of convolution blocks 411 connected in series. Figure 8 In the description, a first extraction module 41 including four convolution blocks 411 connected in series is taken as an example. A convolution block can be considered as a feature extraction network. The convolution block 411 can perform a convolution operation to extract a first structural feature. The first structural feature extracted by each convolution block 411 is a high-frequency feature (also can be recorded as a high-frequency signal), and the depth of the high-frequency signal extracted by different convolution blocks 411 is different, that is, each convolution block 411 outputs a high-frequency signal at a different depth. The specific structure of the convolution block 411 can be set according to the situation. For example, the convolution block 411 can use Octave Convolution, and in order to make the convolution block 411 smaller, a depth-separable convolution (DWC) can be used instead of the ordinary convolution in OctaveConvolution to form the convolution block 411 that is finally used.

[0113] The second extraction module 42 is used to extract the structural features of the first image block. In the embodiment, the extracted structural features of the first image block are recorded as the second structural features. When the second extraction module 42 extracts the second structural features, the first structural features are used as a guide so that the second structural features can more accurately reflect the real structural information of the edge of the component in the first image block. The second extraction module 42 includes a plurality of convolution blocks 421 connected in series. Figure 8 In the description, the second extraction module 42 includes four serially connected convolution blocks 421 as an example. The convolution blocks 421 are used to perform convolution operations, and each convolution block 421 can be considered as a multi-scale reconstruction branch for extracting structural features of the second image block. Fig. 9 for Figure 8 The structural diagram of the convolution block 421 is shown in FIG. Fig. 9, each convolution block 421 includes a first depth-wise separable convolution submodule 4211, a second depth-wise separable convolution submodule 4212, a third depth-wise separable convolution submodule 4213, an adding unit 4214 and a ghostnet submodule 4215; the input of the first depth-wise separable convolution submodule 4211 and the input of the second depth-wise separable convolution submodule 4212 are used as the input of the convolution block 421, the output of the first depth-wise separable convolution submodule 4211 and the output of the second depth-wise separable convolution submodule 4212 are used as the input of the adding unit 4214, the output of the adding unit 4214 is used as the input of the third depth-wise separable convolution submodule 4213, the output of the third depth-wise separable convolution submodule 4213 is used as the input of the ghostnet submodule 4215, and the output of the ghostnet submodule 4215 is used as the output of the convolution block 421. Among them, the first depth-separable convolution submodule 4211, the second depth-separable convolution submodule 4212 and the third depth-separable convolution submodule 4213 all use depth-separable convolution (DWC) and the size of the convolution kernel is 3×3. Among them, the dilation of the first depth-separable convolution submodule 4211 and the third depth-separable convolution submodule 4213 is 1, and the dilation of the second depth-separable convolution submodule 4212 is 2, so that the receptive field of the convolution block 421 can be accurate to the scene size. The ghostnet submodule 4215 uses ghostnet (a lightweight network) and the size of the convolution kernel is 1×1. By using DWC and ghostnet, the efficiency of the convolution block 421 can be improved and the convolution block 421 can be made small enough. It can be understood that in actual applications, the convolution block 421 can also use other structures, as long as the relevant functions are completed.

[0114] The decoding module 43 is used to decode the first structural feature output by the first extraction module 41 and the second structural feature output by the second extraction module 42 and correct the height value in the first image block based on the decoding result to obtain the second image block. The decoding module 43 corresponds to Figure 7 The decoding module 43 can use a common decoder structure and only needs to implement the currently required decoding function. Figure 8 It is only exemplarily shown that the decoding module 43 includes two decoding units 431, and each decoding unit 431 can use a common decoder structure. In addition, when using a common decoder structure, a pixel shuffle operation plus a conventional convolution can be used instead of a deconvolution operation to avoid the deconvolution operation.

[0115] based on Figure 8 When the neural network model shown processes the reference image block and the first image block, step 243 may include steps 2431-2433:

[0116] Step 2431: A first extraction module extracts a first structural feature of a reference image block.

[0117] Exemplarily, when the first extraction module 41 extracts the first structural feature of the reference image block, each convolution block 411 can extract the structural feature and different convolution blocks 411 extract at different depths, and the first structural features output by some convolution blocks 411 can be used by the second extraction module 42. Figure 8 Taking the structure shown as an example, the processing process of the first extraction module 41 is: the reference image block is input to the first convolution block 411, the first convolution block 411 extracts the corresponding first structural features and inputs them to the second convolution block 411, the second convolution block 411 extracts the first structural features of different depths again and inputs them to the third convolution block 411 and the second extraction module 42 respectively (specifically for use by the third convolution block 421 in the second extraction module 42), the third convolution block 411 extracts the first structural features of different depths again and inputs them to the fourth convolution block 411 and the second extraction module 42 respectively (specifically for use by the fourth convolution block 421 in the second extraction module 42), the fourth convolution block 411 extracts the first structural features of different depths again and uses them as the first structural features finally output by the first extraction module 41, so as to be input to the decoding module 43 for use.

[0118] Step 2432: The second extraction module extracts the second structural feature of the first image block in combination with the first structural feature.

[0119] Exemplarily, when the second extraction module 42 extracts the second structural feature of the first image block, each convolution block 421 can perform multi-scale structural feature extraction to output different second structural features, and the convolution block 421 that receives the first structural feature can extract the second structural feature in combination with the first structural feature. Figure 8Taking the structure shown as an example, the processing process of the second extraction module 42 is as follows: the first image block is input to the first convolution block 421, the first convolution block 421 extracts the corresponding second structural feature and inputs it to the second convolution block 421, the second convolution block 411 extracts the corresponding second structural feature again and inputs it to the third convolution block 421, the third convolution block 421 fuses the input first structural feature and the second structural feature, extracts the corresponding second structural feature again based on the fused structural feature and inputs it to the fourth convolution block 421, the fourth convolution block 421 fuses the input first structural feature and the second structural feature, extracts the corresponding second structural feature again based on the fused structural feature and uses it as the second structural feature finally output by the second extraction module 42, so as to be input to the decoding module 43 for use. Among them, the torch.cat function can be used to fuse the first structural feature and the second structural feature. For example, the first structural feature is recorded as x, and the second structural feature is recorded as x'. The fusion of structural features can be realized through the torch.cat function (i.e., torch.cat(x, x')). Assume that the size of x is (h, w, c1) and the size of x' is (h, w, c2), where h and w represent the height and width of the first image block (which can also be the reference image block), respectively, and c1 and c2 represent the corresponding number of channels, respectively. Then, the size of the fused structural feature can be expressed as (h, w, c1+c2).

[0120] Step 2433: The decoding module decodes the first structural features finally output by the first extraction module and the second structural features finally output by the second extraction module, and modifies the first image block based on the decoding results to obtain the second image block.

[0121] The decoding module 43 receives the first structural feature (such as Figure 8 The first structural feature output by the fourth convolution block 411 in the first extraction module) and the second structural feature finally output by the second extraction module (such as Figure 8 After the first structural feature and the second structural feature are obtained, the first structural feature and the second structural feature are fused, for example, the fusion is realized by using the torch.cat function, and then the decoding module 43 decodes the fused structural feature to obtain a height value, and then the height value shown in the first image block is corrected based on the decoded height value to obtain the second image block. The correction process may be to replace the height value of the corresponding position in the first image block with the decoded height value.

[0122] After obtaining the second image block, it can be considered that the neural network model has completed one processing. Afterwards, the neural network model can continue to process other first image blocks and reference image blocks to continue to obtain second image blocks. Until all first image blocks and reference image blocks are processed. It can be understood that when using RGB images, there is no need to rely on the integrity of the image, but to use the structural information on the board in the RGB image to limit and repair the height value of the structural information at the corresponding position in the second height image. Therefore, after the RGB image and the second height image are segmented and then processed by the neural network model, the processing accuracy will not be affected.

[0123] Step 244: After obtaining the second image blocks corresponding to the first image blocks, the second image blocks are spliced ​​to obtain a feature-restored third height image.

[0124] For example, after obtaining each second image block, each second image block can be spliced ​​into a complete height image, that is, a third height image. When splicing, each second image block can be arranged according to the arrangement order between the corresponding first image blocks, and after the arrangement, the third height image can be spliced. The splicing means can adopt the currently existing splicing means, and the embodiment does not limit this.

[0125] Based on the above description, it can be known that the neural network model is a residual learning mapping. At present, in order to better understand the neural network model, the training process of the neural network model is described. It can be understood that when training the neural network model, the device used can be a height image determination device or other devices. After the training is completed by other devices, it is deployed on the height image determination device. Fig.10 A schematic diagram of the training process of a neural network model provided for one embodiment of the present application. Fig.10 In , RGB refers to the RGB image used in the training process, and Height refers to the height image used in the training process. Each RGB image has a corresponding height image, and the height image is an image with accurate height values. The resolution of the RGB image and the height image are equal, for example, the resolution of both is 4000×3000. At this time, each point of the board in the RGB image can get an accurate height value based on the height image. Currently, the components on the board include small-sized components. There is currently no limit on the number of RGB images and height images used in the training process, and generally it needs to meet the training requirements. There is currently no limit on the source of the image.

[0126] In the embodiment, the training process is described by taking a processing process of a neural network model as an example, specifically: an RGB image and a corresponding height image used in the training process are selected. Afterwards, a reference image block and a first image block used in the training process are obtained based on the RGB image and the height image. Currently, the resolution of the RGB image and the height image is 4000×3000, and the resolution of the reference image block and the first image block is 1000×1000. In one embodiment, the RGB image and the height image can be processed by one of three methods: random cropping, center cropping, and random scaling, wherein random cropping refers to randomly selecting a point in the RGB image (or height image), and using this point as the center or corner point (such as the upper left corner point), and cutting a region with a resolution of 1000×1000 in the RGB image (or height image) as a reference image block (or first image block). Cut a region with a resolution of 1000×1000 at the same position in the height image (or RGB image) as the first image block (or reference image block). Center cropping means selecting a point at the center of the RGB image and cropping a block with a resolution of 1000×1000 with this point as the center as the reference image block. Similarly, cropping a block with a resolution of 1000×1000 with the point at the center of the height image as the first image block. Random scaling means scaling the RGB image to a resolution of 1000×1000 as the reference image block. Similarly, scaling the height image to a resolution of 1000×1000 as the first image block.

[0127] After obtaining the reference image block and the first image block, the first image block is used as the Height-GT, that is, the accurate height value in the Height-GT is used to supervise the neural network model. In addition, the first image block is blurred to blur the accurate height value and obtain a less accurate height value. The means for implementing the blurring process can be set according to the actual situation. For example, the first image block is downsampled to reduce the resolution of the first image block. At this time, some points in the first image block will be discarded during the downsampling process (the height values ​​represented by these points are discarded). Then, the downsampled first image block is randomly mean filtered to change the height value therein, so that the height value becomes inaccurate. Then, the first image block after the mean filtering is upsampled to obtain a first image block with a resolution of 1000×1000. The first image block can be considered as the blurred first image block. After that, the reference image block and the blurred first image block are input into the neural network model ( Fig.10 The AI ​​module in the image is processed to output the processed image blocks by the neural network model. Fig.10The image block output by the neural network model is represented as Height Pred. After that, based on the height values ​​recorded by Height-GT and Height Pred, the loss function of the neural network model is determined, and the parameters of the neural network model are adjusted based on the loss function. After a large number of processing processes, the neural network model tends to be stable, that is, the height value in Height Pred output by the neural network model is close to or equal to the height value in Height-GT. It can be understood that since the first image block is blurred, the neural network model can learn to output a more accurate Height Pred after processing the blurred first image block, so as to ensure the processing accuracy of the neural network model.

[0128] After experimenting with the currently provided neural network model, it can be concluded that when the neural network model processes an image with a resolution of 1000×1000, the inference time is about 1-3ms, the size of the neural network model is 228.7kb, and the GPU occupies about 1G, which has better performance.

[0129] The height image determination method provided in the above embodiment is exemplarily described below. In this example, a part of the area of ​​the board to be tested is used as follows: Figure 1 In this example, the height image determination device is equipped with four camera sensors (ie, side cameras) that take spectral images from different angles to obtain four original height images. Fig.11 The first schematic diagram of the processing flow of a method for determining a height image provided by an embodiment of the present application. Fig.11 The height image determination device can obtain four original height images and RGB images of the board to be tested, where the four original height images are Fig.11 In the figure, they are represented as h1-h4 respectively, and the RGB image is Fig.11 It is represented as RGB. Afterwards, the four original height images are fused, and the RGB image is used as a constraint during the fusion to obtain a first height image. The first height image is interpolated (i.e., upsampled) to obtain a second height image. Afterwards, the RGB image and the second height image are cut into blocks, and the obtained reference image blocks and the first image blocks are processed by the neural network model. Afterwards, the second image blocks output by the neural network model are spliced ​​to obtain a third height image. It can be understood that the processing process of the neural network model can refer to the above description. Fig.11 The cutting and stitching process is not shown, only the processing of the RGB image and the second height image by the neural network model is shown.

[0130] Fig.12 A first three-dimensional schematic diagram provided for an embodiment of the present application, Fig.13A second three-dimensional schematic diagram is provided for one embodiment of the present application. Fig.12 and Fig.13 It is a three-dimensional height map generated based on the height values ​​of the components 11 represented in the third height image. Fig.12 and Fig.13 They correspond to the three-dimensional height maps under different viewing angles. Figure 3 and Figure 4 , Fig.12 and Fig.13 In the three-dimensional height map, the edge of the component 11 is relatively sharp and not overly smoothed, and the component does not present a "valley bag" shape, that is, it is closer to the actual edge of the component 11. Fig.14 A schematic diagram of a component provided by an embodiment of the present application is a schematic diagram of a component obtained by reconstructing the small-sized component based on the height value of the small-sized component represented in the third height image. Figure 5 , Fig.14 The edge area 14 of small and medium-sized components is sharper. Figure 5 The abnormal collapse of the middle edge area 14 has been repaired to a certain extent, which can better reflect the actual edge of the small-sized components.

[0131] The height image determination method provided in the above embodiment obtains a first height image with low resolution and including the height of the board to be tested, obtains a second height image with high resolution according to the first height image, and the resolution of the second height image is equal to the resolution of the RGB image obtained by photographing the board to be tested. After that, the RGB image and the second height image are processed by the neural network model, and in the process, the features (specifically structural features) of the RGB image are used as a reference to determine the features (specifically structural features) of the second height image, and then the technical means of obtaining the third height image can solve the technical problem of low imaging accuracy of the height image when the resolution of the height image is consistent with the resolution of the RGB image in the related art. When the neural network model is processed, the high-frequency structural features of the RGB image are used as semantic supervision signals for guidance, and the height value in the second height image is corrected using a deep learning method to obtain a third height image with a more accurate height value, and then when the board to be tested is three-dimensionally reconstructed based on the third height image, the edges of the components on the board to be tested can be kept sharp, with a certain sharp repair effect, especially for small-sized components, the "valley bag" shape can be eliminated, and the accuracy of three-dimensional reconstruction can be improved. In addition, the neural network model has good robustness, can adapt to the influence of the size, color, shape, light changes, etc. of different non-polar components, can obtain a more accurate height image, and has the characteristics of high versatility. In addition, the high-resolution second height image and RGB image are segmented, and the segmented image blocks are processed using the neural network model, which can improve the processing speed of the neural network model. In addition, the convolution block for extracting the second structural feature in the neural network model is optimized using DWC and ghostnet, so that the convolution block is small enough, thereby making the neural network model lightweight, which can reduce the demand for GPU video memory, and the quality of the third height image obtained is better than the quality of the second height image obtained by the interpolation algorithm.

[0132] Fig.15 A flowchart of a method for determining a height image is provided in one embodiment of the present application. Fig.15 The height image determination method shown is in Figure 6 Based on the height image determination method shown in FIG, a two-level fusion strategy is added, that is, in addition to using the neural network model to obtain the third height image, the second height image is also subjected to median filtering, and the third height image and the second height image after median filtering are fused to obtain a height image with better quality. Fig.15 , the height image determination method includes steps 310 to 360:

[0133] Step 310: Acquire a first height image, where the first height image is used to show the measured height of the board to be tested, where the board to be tested includes a plurality of components.

[0134] Step 320: Obtain a corresponding second height image according to the first height image, wherein the resolution of the second height image is higher than that of the first height image.

[0135] Step 330: Acquire a reference image of the board to be tested. The resolution of the second height image is equal to the resolution of the reference image. The reference image is an RGB image obtained by photographing the board to be tested.

[0136] Step 340: Process the second height image and the reference image through a neural network model, and use the features of the reference image to determine the features of the second height image during the processing to obtain a feature-restored third height image.

[0137] Step 350: Perform median filtering on the second height image to obtain a fourth height image.

[0138] Exemplarily, after obtaining the second height image, the second height image is also processed using a median filter. Among them, the implementation process of the median filter can refer to the existing median filter implementation process, the difference is that the existing median filter uses the gray value of the pixel, and the median filter in the embodiment uses the height value represented by the pixel. In the embodiment, the image obtained after the median filter is recorded as the fourth height image. The resolution of the fourth height image is equal to the resolution of the second height image. Compared with the second height image, the fourth height image is smoother, especially the surface of the same plane in the components and the surface of the board bottom plate are smoother.

[0139] It should be noted that in actual applications, in addition to using median filtering, other filtering methods or multiple filtering methods can also be used, mainly to ensure that the surface of the components and the surface of the board base in the fourth height image are smoother.

[0140] Step 340 and step 350 may be performed simultaneously or sequentially, and the embodiment does not limit this.

[0141] Step 360: Fusing the fourth height image and the third height image to obtain a fifth height image that meets a preset condition, wherein the preset condition includes that an evaluation index corresponding to the accuracy is greater than a preset threshold.

[0142] After obtaining the third height image and the fourth height image, the third height image and the fourth height image may be fused to obtain a new height image. In the embodiment, the height image obtained after fusion is recorded as the fifth height image.

[0143] The means of fusion are not currently limited. For example, the height value of the edge of the component in the fifth height image uses the height value in the third height image, and the height value of the non-edge (such as the non-edge surface) of the component uses the height value in the fourth height image. For another example, the fusion is performed in combination with the neighborhood information in the RBG image. Specifically, taking the 3×3 neighborhood as an example, for a certain pixel point in the RGB image (representing a certain position on the board to be tested), the corresponding point (i.e., the point representing the height value of the position on the board to be tested) can be found in both the third height image and the fourth height image, and the three points have a corresponding relationship. When fusion is performed, the pixel point in the RGB image is taken as the center, and each pixel point in the 3×3 neighborhood of the pixel point in the RGB image is found. At present, in order to distinguish, the pixel points in the 3×3 neighborhood are recorded as neighborhood pixels, and the pixel points at the center position are recorded as center pixels. Afterwards, the weight value is obtained based on the difference in grayscale value between each neighborhood pixel point and the center pixel point, and each neighborhood pixel point has a corresponding weight value. It is understandable that when the neighborhood pixel and the center pixel belong to the same object (such as being located on the same component or both being located on the board bottom plate), the difference in grayscale values ​​between the two is small, and the corresponding weight value should be larger. When the neighborhood pixel and the center pixel belong to different objects (i.e., being located on different components or respectively being located on components and board bottom plates), the difference in grayscale values ​​between the two is large, and the corresponding weight value should be small. The method for determining the weight value is not currently limited. Afterwards, the point corresponding to each neighborhood pixel is found on the third height image, and the corresponding height value is obtained. The height value and weight value corresponding to each neighborhood pixel are weighted and then averaged to obtain a weighted average height value. It is understandable that the larger the weight value, the greater the influence (i.e., contribution) of the corresponding height value on the weighted average height value. The weighted average height value mainly refers to the height value of the same object in the neighborhood (the difference in height values ​​of adjacent positions on the same object is small). And, calculate the square of the difference (i.e., variance) between the height value corresponding to each neighborhood pixel and the weighted average height value. At this time, each neighborhood pixel has a corresponding variance. After that, use the variance and weight value corresponding to each neighborhood pixel to weight and average, and obtain a weighted average variance. The larger the weighted average variance, the greater the deviation of the height value of the center pixel calculated based on the neighborhood from the true height value, that is, the smaller the number of neighborhood pixels belonging to the same object as the center pixel in each neighborhood pixel. And, according to the processing method of the third height image, the fourth height image is processed, and a weighted average height value and a weighted average variance are also obtained. After that, a height value is obtained by combining the two weighted average height values ​​obtained and the corresponding weighted average variance, and the obtained height value is used as the height value represented by the point corresponding to the center pixel in the fifth height image. For example, the weighted average height value with a small weighted average variance is selected as the final height value.For another example, a weight value is set for the corresponding weighted average height value based on the weighted average variance, and the smaller the weighted average variance, the larger the corresponding weight value. After that, a weighted average calculation is performed based on the set weight value and the weighted average height value to obtain the final height value. At this time, the larger the weighted average variance, the greater the contribution of the corresponding weighted average height value to the final height value, and vice versa, the smaller the contribution of the corresponding weighted average height value to the final height value. After each pixel in the RGB image is processed as a central pixel in this way, a fifth height image can be obtained. It can be understood that when calculating in the aforementioned manner, if a certain neighborhood pixel point is not the same object as the central pixel point, then the height value of the neighborhood pixel point in the third height image and the height value in the fourth height image contribute little to the height value in the fifth height image, and vice versa, the contribution is large, so that the height value in the fifth height image can be ensured to be closer to the actual height value, and the height value on the same surface (i.e., on the same object) can be made smoother, and the height value of the edge of the component can be more sharp.

[0144] After obtaining the fifth height image, the fifth height image can be used for subsequent inspections, such as tin creep inspection, volume inspection, etc.

[0145] It can be understood that the height values ​​of the surface of the board to be tested and the surface of the components in the fourth height image obtained by median filtering are smoother, and deviations will occur for the edges of the components, that is, they become less sharp, especially for small-sized components, which are prone to edge loss. The third height image obtained based on the neural network model can avoid the negative impact of smooth edges after median filtering as much as possible, so that the edges of the components remain sharp, but the surfaces (non-edges) of the components and the tinning places may not be smooth enough. Therefore, through the technical solution provided in the aforementioned embodiment, the fourth height image and the third height image are fused, that is, the effect of taking advantage of each other's strengths and making up for each other's weaknesses can be achieved through the secondary fusion strategy, so that the height value shown in the fifth height image can maintain the smoothness of the plane, for example, maintain the smoothness of the tinning places on the board, and also maintain the sharpness of the edges of the components, especially for small-sized components, it can avoid the "valley bag" shape, and has a certain correction ability for the slight collapse (i.e. loss) of the edges of the components.

[0146] In order to more concretely evaluate the height values ​​in the fifth height image, the aforementioned height image determination method is evaluated, and an evaluation index that can reflect the accuracy is calculated, and the evaluation index is used as an evaluation standard, that is, when the evaluation index is higher than the preset threshold, it means that the accuracy of the final result obtained by the height image determination method is high, and then it can be considered that the height value represented in the fifth height image finally obtained meets the actual measurement requirements, that is, the accuracy of the fifth height image meets the requirements. In other words, the fifth height image finally obtained meets the preset conditions, and the preset conditions include that the accuracy evaluation index is greater than the preset threshold. Generally, the preset threshold is 1.67, that is, the accuracy evaluation index of the fifth height image currently obtained is greater than 1.67.

[0147] In one embodiment, the accuracy can be calculated using a CMK algorithm, where CMK refers to Capability Index, which is an indicator used to measure the capability or stability of a process or system. In the embodiment, this indicator is used as an evaluation indicator of accuracy.

[0148] When the CMK algorithm is applied, a calibration object is used, and the height of the calibration object is known, for example, the height of the calibration object is 0.5 mm. Afterwards, the calibration object is transmitted to the height image determination device, and the height image determination device processes the calibration object according to the aforementioned height image determination method to obtain a fifth height image representing the height value of the calibration object. Afterwards, the CMK algorithm formula is used to calculate the CMK value. The CMK algorithm formula is:

[0149]

[0150] Among them, Cmk represents the calculated CMK value, which can be used to represent the accuracy evaluation index. USL represents the maximum height value of all height values ​​measured for the calibration object in the fifth height image, LSL represents the minimum height value of all height values ​​measured for the calibration object in the fifth height image, and S represents the standard deviation of all height values ​​of the calibration object in the fifth height image. Indicates the actual height value of the calibration object (such as 0.5mm). The CMK value calculated according to the above formula is greater than 1.67. That is, the fifth height image meets the preset conditions.

[0151] Currently, in order to make the accuracy evaluation more accurate, the position of the shooting device can be adjusted to shoot the calibration block from different angles to obtain the fifth height image at different angles. Fig.16 A schematic diagram of a shooting area provided by an embodiment of the present application. Fig.16, which shows 9 shooting angles (i.e., numbers 1 to 9 correspond to 9 shooting angles), and the calibration block is located directly below the shooting angle where number 5 is located (i.e., in the middle). Afterwards, the shooting device is controlled to shoot the calibration block from 9 shooting angles to obtain a first height image (the first height image can be an original height image obtained using a shooting device (a side camera located at the corresponding shooting angle)), and then, a second height image is obtained based on the first height image, and a third height image is obtained by combining the second height image and the reference image obtained at the current shooting angle, and a fourth height image is obtained based on the second height image, and then, a fifth height image is obtained by combining the fourth height image and the third height image. At this time, 9 fifth height images can be obtained, and 9 CMK values ​​can be obtained based on the aforementioned CMK algorithm formula, and all 9 CMK values ​​are greater than 1.67 (i.e., the smallest CMK value is greater than 1.67), that is, the fifth height image obtained by the height image determination method meets the preset conditions. It can be understood that each shooting device (ie, the side camera) can shoot from different shooting angles in the aforementioned manner, and 9 CMK values ​​that are all greater than 1.67 can be obtained.

[0152] It can be understood that if a CMK value is less than 1.67, the hardware parameters of the height image determination device can be adjusted until all CMK values ​​are greater than 1.67, and then the aforementioned height image determination method is executed in the height image determination device to obtain a fifth height image that meets the preset conditions when performing height detection on the board to be tested.

[0153] In addition, the preset conditions may also include that the evaluation index corresponding to the accuracy and / or clarity is less than the corresponding preset threshold. That is, the preset conditions include that the evaluation index corresponding to the accuracy is greater than the corresponding preset threshold, and the evaluation index corresponding to the accuracy and / or clarity is less than the corresponding preset threshold. Among them, the same evaluation index can be used for accuracy and clarity. Currently, the root mean square error (RMSE) is used as the evaluation index for evaluating accuracy and clarity. Generally, when RMSE is less than 6, it means that both accuracy and clarity meet the requirements.

[0154] Specifically, when using the calibration object to calculate the CMK value, the RMSE value can also be calculated. The calculation formula of the RMSE value is:

[0155]

[0156] Among them, RMSE represents the calculated RMSE value, which can be used to represent the evaluation index of accuracy and clarity. N represents the total number of points in the fifth height image representing the height value of the calibration object, that is, there are N points in the fifth height image representing the height value of the calibration object, x i Indicates the height value represented by the i-th point among N points. Represents the average value of the height values ​​represented by N points. It can be understood that the RMSE values ​​currently calculated should be less than 6. That is, the height image determination device executes the above-mentioned height image determination method to obtain a fifth height image that meets the preset conditions when performing height detection on the board to be tested.

[0157] The height image determination method provided in the above embodiment is exemplarily described below. Fig.17 A second schematic diagram of a processing flow of a method for determining a height image provided by an embodiment of the present application is shown in FIG. Fig.17 , the height image determination method includes steps 601 to 606:

[0158] Step 601: Acquire multiple original height images.

[0159] Step 602: Obtain a second height image based on the multiple original height images.

[0160] A plurality of original height images are fused and constrained by using the RGB image to obtain a first height image. Thereafter, the first height image is interpolated to obtain a second height image.

[0161] Step 603: Perform median filtering on the second height image to obtain a fourth height image.

[0162] Step 604: Obtain a third height image through a neural network model.

[0163] The second height image and the RGB image are cut into blocks to obtain reference image blocks and first image blocks, and the neural network model is used to process each reference image block and the corresponding first image block in sequence to obtain second image blocks, and each second image block is spliced ​​to obtain a third height image.

[0164] Step 605: fuse the third height image and the fourth height image to obtain a fifth height image.

[0165] Step 606: Output the fifth height image for use.

[0166] Fig.18 A two-dimensional image of a board to be tested provided in one embodiment of the present application, Fig.19 A three-dimensional imaging diagram of a board to be tested is provided in one embodiment of the present application. Fig.18 and Fig.19 The imaging images of partial areas of the board to be tested are shown respectively. Fig.18 It is a two-dimensional image obtained after shooting the board to be tested. Fig.19The three-dimensional image is obtained by three-dimensional reconstruction based on the fifth height image of the board to be tested, that is, after the fifth height image of the board to be tested is obtained by the aforementioned height image determination method, the three-dimensional reconstruction is performed based on the height value represented in the fifth height image to obtain the three-dimensional image. Fig.18 and Fig.19 It can be seen that the components in the three-dimensional imaging image are basically consistent with the components in the two-dimensional imaging image, and the three-dimensional reconstruction effect is good, which is convenient for the subsequent detection of the board to be tested.

[0167] An embodiment of the present application also provides a height image determination device. Fig. 20 A schematic diagram of a height image determination device provided in one embodiment of the present application, referring to Fig. 20 The height image determination device includes: a first acquisition unit 701, a first processing unit 702, a second acquisition unit 703 and a second processing unit 704.

[0168] Among them, the first acquisition unit 701 is used to acquire a first height image, the first height image is used to show the measured height of the board to be tested, and the board to be tested includes multiple components; the first processing unit 702 is used to obtain a corresponding second height image according to the first height image, and the resolution of the second height image is higher than the resolution of the first height image; the second acquisition unit 703 is used to acquire a reference image of the board to be tested, the resolution of the second height image is equal to the resolution of the reference image, and the reference image is an RGB image obtained by photographing the board to be tested; the second processing unit 704 is used to process the second height image and the reference image through a neural network model, and use the features of the reference image to determine the features of the second height image during the processing to obtain a feature-repaired third height image.

[0169] In one embodiment of the present application, the device also includes: a filtering unit, used to perform median filtering on the second height image to obtain a fourth height image; a first fusion unit, used to fuse the fourth height image and the third height image to obtain a fifth height image that meets preset conditions, and the preset conditions include an evaluation index corresponding to the accuracy being greater than a preset threshold.

[0170] In one embodiment of the present application, the second processing unit 704 includes: an image segmentation subunit, which is used to use a preset segmentation rule to segment the reference image and the second height image respectively to obtain multiple reference image blocks and multiple first image blocks, and the reference image blocks and the first image blocks correspond to each other one by one; an image block input subunit, which is used to input the reference image block and the corresponding first image block into the neural network model together; an image block processing subunit, which is used to process the reference image block and the first image block by the neural network model, and during the processing, the first structural feature of the reference image block is used to determine the second structural feature of the first image block, and obtain the second image block based on the first structural feature of the reference image block and the second structural feature of the first image block, and each first image block has a corresponding second image block; an image block splicing subunit, which is used to obtain the second image block corresponding to each first image block, and then splice each second image block to obtain a feature-restored third height image.

[0171] In one embodiment of the present application, the neural network model includes a first extraction module, a second extraction module and a decoding module; the image block processing subunit includes: a first feature extraction subunit, which is used to extract the first structural feature of the reference image block by the first extraction module; a second feature extraction subunit, which is used to extract the second structural feature of the first image block by the second extraction module in combination with the first structural feature; a feature decoding subunit, which is used to decode the first structural feature finally output by the first extraction module and the second structural feature finally output by the second extraction module by the decoding module and modify the first image block based on the decoding result to obtain a second image block.

[0172] In one embodiment of the present application, the second extraction module includes a plurality of convolution blocks connected in series, and the convolution block includes a first depth-separable convolution submodule, a second depth-separable convolution submodule, a third depth-separable convolution submodule, an addition unit and a ghostnet submodule; the input of the first depth-separable convolution submodule and the input of the second depth-separable convolution submodule are used as the input of the convolution block, the output of the first depth-separable convolution submodule and the output of the second depth-separable convolution submodule are used as the input of the addition unit, the output of the addition unit is used as the input of the third depth-separable convolution submodule, the output of the third depth-separable convolution submodule is used as the input of the ghostnet submodule, and the output of the ghostnet submodule is used as the output of the convolution block.

[0173] In one embodiment of the present application, the first acquisition unit 701 includes: a third acquisition sub-unit, used to acquire multiple original height images of the board to be tested taken by multiple side cameras, each of the original height images corresponding to a different shooting angle; a second fusion sub-unit, used to fuse the multiple original height images to obtain a first height image.

[0174] In one embodiment of the present application, the first processing unit 702 is specifically configured to perform interpolation calculation on the first height image to obtain a second height image.

[0175] The above-mentioned height image determination device can be used to execute the height image determination method provided by any of the above-mentioned embodiments, and has corresponding functions and beneficial effects.

[0176] It is worth noting that in the embodiment of the above-mentioned height image determination device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0177] An embodiment of the present application also provides a height image determination device, Fig.21 A schematic diagram of a height image determination device provided in one embodiment of the present application, referring to Fig.21 The height image determination device includes: a processor 81 and a memory 82. The processor 81 is one or more. Fig.21 The memory 82 is used to store one or more programs; when the one or more programs are executed by the one or more processors 81, the one or more processors 81 implement the height image determination method provided by any of the above embodiments. For the description of the height image determination device and the components included therein, reference can be made to the above content.

[0178] The above-mentioned height image determination device includes a height image determination device, which can be used to execute any height image determination method and has corresponding functions and beneficial effects. For specific details not currently described, please refer to the relevant description of the above-mentioned height image determination method.

[0179] One embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform relevant operations in the height image determination method provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0180] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0181] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing module of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processing module of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0182] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0183] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0184] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for determining a height image, characterized in that: include: Acquire a first height image, where the first height image is used to show the measured height of a board to be tested, where the board to be tested includes a plurality of components; Obtaining a corresponding second height image according to the first height image, wherein a resolution of the second height image is higher than a resolution of the first height image; Acquire a reference image of the board to be tested, wherein the resolution of the second height image is equal to the resolution of the reference image, and the reference image is an RGB image obtained by photographing the board to be tested; The second height image and the reference image are processed by a neural network model, and during the processing, features of the reference image are used to determine features of the second height image to obtain a feature-restored third height image.

2. The height image determination method according to claim 1, characterized in that: Also includes: Performing median filtering on the second height image to obtain a fourth height image; The fourth height image and the third height image are fused to obtain a fifth height image that meets a preset condition, wherein the preset condition includes that an evaluation index corresponding to accuracy is greater than a preset threshold.

3. The height image determination method according to claim 1, characterized in that: The processing of the second height image and the reference image by a neural network model, and determining the features of the second height image by using the features of the reference image during the processing to obtain a feature-restored third height image, includes: Using a preset segmentation rule to segment the reference image and the second height image respectively to obtain a plurality of reference image blocks and a plurality of first image blocks, wherein the reference image blocks correspond to the first image blocks one by one; Inputting the reference image block and the corresponding first image block into a neural network model together; The neural network model processes the reference image block and the first image block, determines the second structural feature of the first image block using the first structural feature of the reference image block during the processing, and obtains the second image block based on the first structural feature of the reference image block and the second structural feature of the first image block, each of the first image blocks has a corresponding second image block; After obtaining the second image blocks corresponding to the first image blocks, the second image blocks are spliced ​​to obtain a feature-restored third height image.

4. The method for determining height image according to claim 3, characterized in that: The neural network model includes a first extraction module, a second extraction module and a decoding module; The neural network model is used to process the reference image block and the first image block, and during the processing, the first structural feature of the reference image block is used to determine the second structural feature of the first image block, and the second image block is obtained based on the first structural feature of the reference image block and the second structural feature of the first image block, including: Extracting a first structural feature of the reference image block by the first extraction module; Extracting a second structural feature of the first image block by the second extraction module in combination with the first structural feature; The decoding module decodes the first structural features finally output by the first extraction module and the second structural features finally output by the second extraction module, and modifies the first image block based on the decoding result to obtain the second image block.

5. The height image determination method according to claim 4, characterized in that: The second extraction module includes multiple convolution blocks connected in series, and the convolution block includes a first depth-separable convolution submodule, a second depth-separable convolution submodule, a third depth-separable convolution submodule, an addition unit and a ghostnet submodule; the input of the first depth-separable convolution submodule and the input of the second depth-separable convolution submodule are used as the input of the convolution block, the output of the first depth-separable convolution submodule and the output of the second depth-separable convolution submodule are used as the input of the addition unit, the output of the addition unit is used as the input of the third depth-separable convolution submodule, the output of the third depth-separable convolution submodule is used as the input of the ghostnet submodule, and the output of the ghostnet submodule is used as the output of the convolution block.

6. The height image determination method according to claim 1, characterized in that: The acquiring of the first height image comprises: Acquire multiple original height images of the board to be tested taken by multiple side cameras, each of the original height images corresponding to a different shooting angle; The multiple original height images are fused to obtain a first height image.

7. The height image determination method according to claim 1, characterized in that: The step of obtaining a corresponding second height image according to the first height image comprises: An interpolation calculation is performed on the first height image to obtain a second height image.

8. A height image determination device, characterized in that: include: A first acquisition unit is used to acquire a first height image, where the first height image is used to show the measured height of a board to be tested, where the board to be tested includes a plurality of components; A first processing unit, configured to obtain a corresponding second height image according to the first height image, wherein a resolution of the second height image is higher than a resolution of the first height image; A second acquisition unit is used to acquire a reference image of the board to be tested, wherein the resolution of the second height image is equal to the resolution of the reference image, and the reference image is an RGB image obtained by photographing the board to be tested; The second processing unit is used to process the second height image and the reference image through a neural network model, and use the features of the reference image to determine the features of the second height image during the processing to obtain a feature-restored third height image.

9. A height image determination device, characterized in that: include: one or more processors and memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the height image determination method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the height image determination method as described in any one of claims 1 to 7 is implemented.

Images