Panoramic depth image acquisition system and method, defect detection method and device, and medium

Abstract

The invention provides a panoramic depth image acquisition system and method, a defect detection method and device and a medium. A plurality of cameras, a camera support and a carrying table for carrying a target object are configured; fixing the plurality of cameras at different shooting angles, and when at least one camera relatively moves to different distances between the camera bracket and the carrying table, respectively shooting the target object on the carrying table to obtain different depth-of-field images of the target object at the corresponding shooting angles, and generating a panoramic depth image of the target object at the corresponding shooting angle. According to the panoramic depth image acquisition system, the panoramic depth image of the target object can be acquired from different shooting angles, each part of the target object can be clearly imaged, more complete and reliable information is provided for subsequent visual analysis and decision making such as defect detection, and the accuracy of visual analysis and decision making is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to panoramic depth image acquisition systems and methods, defect detection methods, equipment and media. Background Technology

[0002] In many fields such as industrial inspection, precision manufacturing, and biomedicine, vision systems typically rely on optical lenses to acquire images for analysis and decision-making.

[0003] However, images captured by conventional optical lenses are only clear near the focal plane, while the foreground and / or background are blurred to varying degrees, resulting in information loss in the image and reducing the accuracy of visual analysis and decision-making. Summary of the Invention

[0004] This disclosure provides various aspects of a panoramic depth image acquisition system and method, a defect detection method, equipment, and medium to acquire panoramic depth images, providing more complete and reliable information for subsequent visual analysis and decision-making, and improving the accuracy of visual analysis and decision-making.

[0005] The first aspect of this disclosure provides a panoramic depth image acquisition system, the panoramic depth image acquisition system comprising: multiple cameras, camera brackets, and a platform for carrying target objects;

[0006] The camera bracket is used to fix multiple cameras at different shooting angles; and the camera bracket and the platform can move relative to each other.

[0007] At least one of the cameras is used to take pictures of the target object on the platform when the camera bracket and the platform move relative to each other to different distances, so as to obtain different depth images of the target object at the corresponding shooting angles, and to generate a full depth image of the target object at the corresponding shooting angles.

[0008] A second aspect of this disclosure provides a method for acquiring panoramic depth images, including:

[0009] The camera support is used to control the relative movement between the camera mount and the platform that carries the target object; wherein the camera mount is used to fix multiple cameras at different shooting angles.

[0010] By moving at least one of the cameras relative to the camera bracket and the platform to different distances, the target object is photographed to obtain different depth-of-field images of the target object at at least one shooting angle, so as to generate a full depth-of-field image of the target object at the corresponding shooting angle.

[0011] A third aspect of this disclosure provides a defect detection method, comprising:

[0012] Acquire a panoramic depth image of the target object from at least one shooting angle, or acquire a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object.

[0013] Based on the panoramic depth image of the target object from at least one shooting angle, or based on the panoramic depth image of the target object from at least one shooting angle and the true-color point cloud image of the target object, the defect type of the target object is identified by a preset model.

[0014] A fourth aspect of this disclosure provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the second aspect or the method of the third aspect.

[0015] The fifth aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of the second aspect or the method of the third aspect described above.

[0016] The sixth aspect of this disclosure provides a computer program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method of the second aspect or the method of the third aspect described above.

[0017] The panoramic depth image acquisition system, method, defect detection method, device, and medium of this disclosure are configured with multiple cameras, camera supports, and a platform supporting a target object. The multiple cameras are fixed at different shooting angles, and at least one camera moves relative to the camera support and platform to different distances, respectively capturing images of the target object on the platform. This yields different depth-of-field images of the target object at the corresponding shooting angles, generating a panoramic depth image of the target object at the corresponding shooting angle. The panoramic depth image acquisition system of this embodiment can acquire panoramic depth images of a target object from different shooting angles, providing clear imaging of various parts of the target object. This provides more complete and reliable information for subsequent visual analysis and decision-making, such as defect detection, improving the accuracy of visual analysis and decision-making. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0019] Figure 1A schematic diagram of a panoramic depth image acquisition system provided as an exemplary embodiment of this disclosure;

[0020] Figure 2 A comparative illustration of full depth-of-view images and non-full depth-of-view images provided for exemplary embodiments of this disclosure;

[0021] Figure 3 A schematic diagram of a true-color point cloud image provided for an exemplary embodiment of this disclosure;

[0022] Figure 4 A flowchart of a panoramic depth image acquisition method provided as an exemplary embodiment of this disclosure;

[0023] Figure 5 A flowchart of a defect detection method provided as an exemplary embodiment of this disclosure;

[0024] Figure 6 A schematic diagram of a defect detection method provided for an exemplary embodiment of this disclosure;

[0025] Figure 7 A schematic diagram of another defect detection method provided as an exemplary embodiment of this disclosure;

[0026] Figure 8 A schematic diagram of another defect detection method provided as an exemplary embodiment of this disclosure;

[0027] Figure 9 A schematic diagram of another defect detection method provided as an exemplary embodiment of this disclosure;

[0028] Figure 10 A schematic diagram of another defect detection method provided as an exemplary embodiment of this disclosure;

[0029] Figure 11 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0031] In many fields such as industrial inspection, precision manufacturing, and biomedicine, vision systems typically rely on optical lenses to acquire images for analysis and decision-making.

[0032] However, images captured by conventional optical lenses are only clear near the focal plane, while the foreground and / or background are blurred to varying degrees. In other words, the captured images are non-full depth-of-field images or images with limited depth of field, resulting in information loss and reducing the accuracy of visual analysis and decision-making.

[0033] To address the aforementioned issues, this disclosure provides a panoramic depth image acquisition system and method that can obtain panoramic depth images with full-focus clarity, providing more complete and reliable information for subsequent visual analysis and decision-making (such as defect detection), thereby improving the accuracy of visual analysis and decision-making.

[0034] The following will provide a detailed description of the panoramic depth image acquisition system and method, defect detection method, equipment and medium disclosed herein, with reference to specific embodiments.

[0035] Figure 1 A schematic diagram of a panoramic depth image acquisition system provided for an exemplary embodiment of this disclosure, such as... Figure 1 As shown, the panoramic depth image acquisition system includes: multiple cameras 101, camera brackets (not shown), and a platform (not shown) for carrying the target object 110.

[0036] The camera bracket is used to fix multiple cameras 101 at different shooting angles; and the camera bracket and the platform can move relative to each other; at least one camera is used to take pictures of the target object 110 on the platform when the camera bracket and the platform move relative to each other to different distances, so as to obtain different depth images of the target object 110 at the corresponding shooting angle, and to generate a full depth image of the target object at the corresponding shooting angle.

[0037] This embodiment does not limit the specific shooting angle, and may include vertical shooting angle and non-vertical shooting angle (such as shooting angle at 45° with the vertical direction). Multiple cameras 101 are set to shoot from different shooting angles. The purpose is to obtain full depth images of the target object from different shooting angles. In particular, for surface depressions, different angles can be used to present the internal situation of the surface depressions more comprehensively.

[0038] Furthermore, the camera bracket and the platform can move relative to each other. Specifically, the camera bracket can be fixed while the platform can move; the platform can be fixed while the camera bracket can move; or both the camera bracket and the platform can move. The specific method of movement is not limited in this embodiment.

[0039] At least one camera 101 can capture images of the target object 110 on the platform at different distances as the camera bracket and the platform move relative to each other. This results in different depth-of-field images of the target object at corresponding shooting angles. The different depth-of-field images are generated because the camera's focus shifts accordingly as the camera bracket and the platform move. The parts of the target object 110 (whose surface is not flat) that are at the focus of the camera 101 are captured more clearly, while the parts not at the focus of the camera 101 are captured less clearly. Therefore, when the camera bracket and the platform move relative to each other at different distances, a portion of the target object 110 will always be captured more clearly. By stitching these clear areas together, a full depth-of-field image at the corresponding shooting angle can be obtained. That is, in the full depth-of-field image, all parts (at different heights) of the target object 110 are clearly imaged. Figure 2 The diagram showing the comparison between full depth-of-view and non-full depth-of-view images illustrates that in the full depth-of-view image of the circuit board solder holes on the left, all parts (such as inside and outside the solder holes) are clear and sharp, while the non-full depth-of-view image of conventional imaging cannot simultaneously capture all parts clearly.

[0040] Furthermore, the panoramic depth image acquisition system may also include a processor, which is communicatively connected to multiple cameras to synthesize different depth images of the target object from any shooting angle to obtain a panoramic depth image of the target object from any shooting angle.

[0041] When synthesizing images of a target object at different depths of field from any shooting angle, the system can identify the sharp region of each image in the images of the target object at different depths of field from that shooting angle, and stitch together the sharp regions of each image to obtain a full depth-of-field image of the target object at that shooting angle.

[0042] The panoramic depth image acquisition system provided in this embodiment configures multiple cameras, camera supports, and a platform supporting the target object. The multiple cameras are fixed at different shooting angles. At least one camera moves relative to the target object on the platform at different distances, capturing images of the target object at the corresponding shooting angles to obtain panoramic depth images of the target object. This generates a panoramic depth image of the target object at the corresponding shooting angle. This panoramic depth image acquisition system can acquire panoramic depth images of the target object from different shooting angles, providing clear imaging of various parts of the target object. This provides more complete and reliable information for subsequent visual analysis and decision-making, such as defect detection, thereby improving the accuracy of visual analysis and decision-making.

[0043] Based on the above embodiments, the processor can also be used to generate an initial point cloud image by performing 3D reconstruction based on different depth images of the target object at each shooting angle, and to color the initial point cloud image based on different depth images of the target object at each shooting angle to generate a true-color point cloud image of the target object, such as... Figure 3 The image shows a true-color point cloud image of the solder joint area on the circuit board.

[0044] In generating the initial point cloud image for 3D reconstruction, the strong correlation between sharpness and distance under different focus states in images at different depths of field from the same shooting angle can be utilized. The principle is that a certain part of the target object is sharpest in a particular depth-of-field image at the same shooting angle. Therefore, the depth information of that part of the target object can be determined based on information such as the camera's focal length and the relative distance between the camera and the platform. By combining different depth-of-field images of the target object from each shooting angle, an initial point cloud image of the target object can be reconstructed comprehensively.

[0045] Coloring the initial point cloud image involves coloring each point in the initial point cloud image with the real color information (such as RGB values) of the corresponding position in the image at different depths of field, resulting in a true-color point cloud image. This image can reflect the surface color and texture of the target object and contains both geometric and visual information. It can be used for size detection, appearance detection (such as defect detection), material recognition, and so on.

[0046] Based on any of the above embodiments, the panoramic depth image acquisition system further includes a hemispherical light source for illuminating the target object on the platform from multiple angles. The hemispherical light source is provided with multiple openings for multiple cameras to capture images of the target object on the platform from the multiple openings respectively.

[0047] In this embodiment, a hemispherical light source illuminates the target object on the platform from multiple angles, ensuring that the surface of the target object is free of shadows, especially at different heights. This guarantees consistent brightness across images at different depths of field, thereby improving the quality of the synthesized panoramic depth-of-field images. The hemispherical light source has multiple openings, allowing multiple cameras to capture images of the target object on the platform from these openings. Optionally, multiple cameras can also extend into the openings, and the opening size typically needs to be larger than the camera's cross-sectional size to prevent interference between the cameras and the openings during relative movement between the camera support and the platform.

[0048] Of course, other shapes of light sources can also be used in this embodiment, as long as they can make the surface of the target object free of shadows. This embodiment does not impose any limitations on this.

[0049] Figure 4 A flowchart illustrating the steps of a panoramic depth image acquisition method provided as an exemplary embodiment of this disclosure. Figure 4The method for acquiring full depth-of-field images shown includes the following steps:

[0050] S401, Control the relative movement between the camera bracket and the platform carrying the target object; wherein the camera bracket is used to fix multiple cameras at different shooting angles;

[0051] S402. When at least one of the cameras moves relative to the camera bracket and the platform to different distances, the target object is photographed respectively to obtain different depth images of the target object at at least one shooting angle, so as to generate a full depth image of the target object at the corresponding shooting angle.

[0052] In this embodiment, by controlling the relative movement between the camera bracket and the platform carrying the target object, the camera can take pictures at different distances from the target object. Specifically, the camera bracket can be fixed while the platform moves; the platform can be fixed while the camera bracket moves; or the camera bracket and the platform can move simultaneously, etc. The specific movement method is not limited in this embodiment.

[0053] After a camera at at least one shooting angle captures images of the target object at different distances from the target object and obtains images of the target object at different depths of field at that shooting angle, a full depth image of the target object at that shooting angle can be synthesized based on the images of the target object at different depths of field at that shooting angle.

[0054] The panoramic depth image acquisition method provided in this embodiment involves configuring multiple cameras, camera supports, and a platform supporting the target object. The multiple cameras are fixed at different shooting angles, and at least one camera moves relative to the camera support and platform to different distances, capturing images of the target object on the platform to obtain different depth images of the target object at the corresponding shooting angles, thereby generating a panoramic depth image of the target object at the corresponding shooting angle. The panoramic depth image acquisition system of this embodiment can acquire panoramic depth images of the target object from different shooting angles, providing clear imaging of various parts of the target object. This provides more complete and reliable information for subsequent visual analysis and decision-making, such as defect detection, improving the accuracy of visual analysis and decision-making.

[0055] Optionally, the sharp regions of each image in different depth-of-field images of the target object at any shooting angle can be identified, and the sharp regions of each image can be stitched together to obtain a full depth-of-field image of the target object at that shooting angle, that is, all parts of the target object are clearly imaged in the full depth-of-field image.

[0056] Optionally, an initial point cloud image can be generated by 3D reconstruction based on the different depth images of the target object at each shooting angle, and the initial point cloud image can be colored based on the different depth images of the target object at each shooting angle to generate a true-color point cloud image of the target object.

[0057] In generating the initial point cloud image for 3D reconstruction, the strong correlation between sharpness and distance under different focus states in images at different depths of field from the same shooting angle can be utilized. The principle is that a certain part of the target object is sharpest in a particular depth-of-field image at the same shooting angle. Therefore, the depth information of that part of the target object can be determined based on information such as the camera's focal length and the relative distance between the camera and the platform. By combining different depth-of-field images of the target object from each shooting angle, an initial point cloud image of the target object can be reconstructed comprehensively.

[0058] Coloring the initial point cloud image involves coloring each point in the initial point cloud image with the real color information (such as RGB values) of the corresponding position in the image at different depths of field, resulting in a true-color point cloud image. This image can reflect the surface color and texture of the target object and contains both geometric and visual information. It can be used for size detection, appearance detection (such as defect detection), material recognition, and so on.

[0059] Figure 5 A flowchart illustrating the steps of a defect detection method provided as an exemplary embodiment of this disclosure. Figure 5 The defect detection method shown specifically includes the following steps:

[0060] S501. Obtain a panoramic depth image of the target object from at least one shooting angle, or obtain a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object.

[0061] S502. Based on the panoramic depth image of the target object from at least one shooting angle, or based on the panoramic depth image of the target object from at least one shooting angle and the true color point cloud image of the target object, identify the defect type of the target object using a preset model.

[0062] In this embodiment, for the target object, a panoramic depth image of the target object at at least one shooting angle can be acquired first, or a panoramic depth image of the target object at at least one shooting angle and a true-color point cloud image of the target object can be acquired. Then, the panoramic depth image of the target object at at least one shooting angle can be input into a preset model, and the preset model can identify the defect type of the target object based on the panoramic depth image of the target object at at least one shooting angle. Alternatively, the panoramic depth image of the target object at at least one shooting angle and the true-color point cloud image of the target object can be input into a preset model, and the preset model can identify the defect type of the target object based on the panoramic depth image of the target object at at least one shooting angle and the true-color point cloud image of the target object.

[0063] In this embodiment, the preset model can be any feasible machine learning model, such as a deep learning model, and this embodiment is not limited to any particular model. The preset model can be trained in advance using training data, which may include panoramic depth images labeled with defect types, and / or panoramic depth images labeled with defect types and true color point cloud images, etc. The specific training process will not be described in detail in this embodiment.

[0064] In this embodiment, the defect type of the target object can refer to the surface defect type of the target object. Different types of defect types have different requirements for panoramic depth images and true color point cloud images. For example, defect types involving different heights require panoramic depth images of the target object from multiple shooting angles, or panoramic depth images of the target object from a single shooting angle combined with true color point cloud images, in order to be identified; while defect types in planar areas only require panoramic depth images of the target object from a single shooting angle to be identified, and so on.

[0065] Optionally, the panoramic depth image and / or true-color point cloud image of the target object in this embodiment are obtained based on the panoramic depth image acquisition system as described in the above embodiments, or based on the panoramic depth image acquisition method as described in the above embodiments. Of course, other methods can also be used to obtain the image, and this embodiment is not limited to any particular method.

[0066] The defect detection method provided in this embodiment uses a preset model based on a panoramic depth image of the target object from at least one shooting angle, or based on a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object, to identify the defect type of the target object. This allows the model to obtain more complete and reliable information, improving the accuracy, comprehensiveness and efficiency of defect detection. It also eliminates the need for manual intervention and reduces detection costs.

[0067] In the above embodiments, identifying the defect type of the target object using a preset model may specifically include:

[0068] For defects present in the planar portion of the target object, the type of defect is identified using the preset model based on the panoramic depth image of the target object from at least one shooting angle; or

[0069] For defects present in the surface depressions of the target object, the type of defect present in the surface depressions of the target object is identified by a preset model based on the panoramic depth image of the target object from a non-perpendicular shooting angle, or based on the panoramic depth image of the target object from a vertical shooting angle and the true color point cloud image of the target object.

[0070] In this embodiment, for defects in the planar parts of the target object, since depth information is not involved, it is only necessary to use panoramic depth images in two-dimensional space for defect identification. Panoramic depth images from any one or more shooting angles can be used, with panoramic depth images from a vertical shooting angle being the most effective. Specifically, panoramic depth images from any one or more shooting angles can be input into a preset model, which can directly identify the type of defects in the planar parts of the target object, or the type of defects in the planar parts of the target object can be identified by combining the preset model with measurement methods.

[0071] For defects in the surface depressions of a target object, since depth information is involved, defect identification needs to be performed in three-dimensional space. However, the panoramic depth image taken at a vertical shooting angle cannot reflect the depth information of the surface depressions. Therefore, a panoramic depth image of the target object taken at a non-perpendicular shooting angle is required. Defects in the surface depressions can be identified from a side view using a preset model; alternatively, the panoramic depth image of the target object taken at a vertical shooting angle and a true-color point cloud image of the target object can be combined to reflect the morphology of the surface depressions, and the defects can be identified using a preset model. Specifically, the panoramic depth image of the target object taken at a non-perpendicular shooting angle can be input into the preset model, or the panoramic depth image of the target object taken at a vertical shooting angle and a true-color point cloud image of the target object can be input into the preset model. The preset model can then directly identify the type of defect in the surface depressions of the target object, or the type of defect in the surface depressions of the target object can be identified by combining the preset model with measurement methods.

[0072] Based on any of the above embodiments, as a specific application scenario, the target object can be a circuit board, including but not limited to a flexible printed circuit (FPC) or a rigid circuit board (PCB). The panoramic depth image and / or true color point cloud image of the circuit board can be obtained based on the panoramic depth image acquisition system as described in the above embodiments, or based on the panoramic depth image acquisition method as described in the above embodiments. Then, based on the panoramic depth image of the circuit board at at least one shooting angle, or based on the panoramic depth image of the circuit board at at least one shooting angle and the true color point cloud image of the circuit board, the defect type of the circuit board is identified by a preset model.

[0073] The types of defects on the circuit board include, but are not limited to, unsoldered or insufficient solder in the solder holes, misaligned solder joints, solder overflow or bridging, foreign objects, scratches or cracks, etc.

[0074] In one optional embodiment, for defect detection of open solder joints or insufficient solder in solder holes on a circuit board, since the solder holes are located in recessed areas on the circuit board surface, the defect detection is performed on these recessed areas. (See [reference needed]). Figure 6 The specific process can be as follows:

[0075] For the solder holes of the circuit board, the solder hole area image is identified from the panoramic depth image of the circuit board at a non-vertical shooting angle using the preset model, and the solder hole sidewall image is segmented from the solder hole area image.

[0076] The solder crawling height inside the solder hole is determined based on the solder hole sidewall image, and the defect type of the solder hole on the circuit board is determined based on the solder crawling height, namely, whether it is an empty solder or insufficient solder.

[0077] In this embodiment, the solder hole sidewall position can be captured in the panoramic depth image of the circuit board at a non-perpendicular shooting angle. Therefore, the solder hole area can be located from the panoramic depth image of the circuit board at a non-perpendicular shooting angle using a preset model. Then, the solder hole sidewall image can be segmented from the solder hole area. Based on the solder hole sidewall image, the solder creep height inside the solder hole can be determined by a measurement method. The solder creep height determines whether the defect type of the solder hole on the circuit board is empty solder or insufficient solder. For example, the lower the solder creep height, the lower it is than a first preset height, the solder hole defect type of the circuit board can be determined to be insufficient solder. The lower it is than a second preset height, the solder hole defect type of the circuit board can be determined to be empty solder. The first preset height is higher than the second preset height.

[0078] Optionally, the solder crawling height within the solder hole can be determined based on the image of the solder hole sidewall. Specifically, the pixel width occupied by the exposed solder hole sidewall in the image can be measured. Given a known shooting angle, the pixel width can effectively characterize the solder crawling height within the solder hole; that is, a wider pixel width indicates more exposed solder hole sidewall, thus indicating a lower solder crawling height, and vice versa. Optionally, a mapping relationship can be pre-established between the shooting angle, the pixel width of the exposed solder hole sidewall, and the solder crawling height within the solder hole, and then the solder crawling height within the solder hole can be quantified based on this mapping relationship.

[0079] Optionally, post-processing such as color conversion can be performed on the weld hole sidewall image, for example, converting it to a black and white image. Figure 6 The processed solder hole sidewall image only shows the crescent-shaped solder hole sidewall, which makes it easier to make a more intuitive qualitative and quantitative comparison of solder climb height.

[0080] In another alternative embodiment, for defect detection of open solder joints or insufficient solder in circuit boards, since the solder joints are located in recessed areas on the circuit board surface, defect detection of these recessed areas can be performed by combining a full-depth image and a true-color point cloud image of the circuit board. (See [link to relevant documentation]). Figure 7 The specific steps can be as follows:

[0081] For the solder holes of the circuit board, the solder hole area image is identified from the panoramic depth image of the circuit board at the vertical shooting angle using the preset model, and the contour image of the solder crawling position inside the solder hole is segmented from the solder hole area image.

[0082] Based on the solder crawling location contour image, the solder crawling height inside the solder hole is determined from the true color point cloud image of the circuit board, and the defect type of the solder hole of the circuit board is determined based on the solder crawling height, whether it is an empty solder or insufficient solder.

[0083] In this embodiment, the outline of the solder crawling position inside the solder hole can be captured in the panoramic depth image of the circuit board at a vertical shooting angle. That is, the outline formed by the junction of the solder and the sidewall of the solder hole. The solder hole area can be located from the panoramic depth image of the circuit board at a vertical shooting angle using a preset model. Then, the outline image of the solder crawling position inside the solder hole is segmented from the solder hole area. Then, the solder crawling height inside the solder hole is determined by combining the mapping relationship between the solder crawling position outline image and the true color point cloud image of the circuit board. Specifically, the point cloud of the solder crawling position outline in the true color point cloud image of the circuit board can be determined based on the solder crawling position outline image. The solder crawling height inside the solder hole can be determined by measurement method based on the point cloud of the solder crawling position outline and the point cloud of the solder hole. Then, the solder crawling height can be used to determine whether the defect type of the solder hole of the circuit board is empty solder or insufficient solder. For example, the lower the solder crawling height, the lower it is than a first preset height, the solder hole defect type of the circuit board can be determined to be insufficient solder. The lower it is than a second preset height, the solder hole defect type of the circuit board can be determined to be empty solder. The first preset height is higher than the second preset height.

[0084] In another alternative embodiment, for defect detection of solder joint misalignment on a circuit board, since solder joint misalignment does not involve depth, it can be considered as defect detection of a planar portion of the circuit board. Defect detection can be performed using a full-depth image of the circuit board at any shooting angle, preferably a full-depth image of the circuit board at a vertical shooting angle. See [link to relevant documentation]. Figure 8 The specific process can be as follows:

[0085] The reference point image of the circuit board is segmented from the panoramic depth image of the circuit board at a vertical shooting angle using the preset model.

[0086] The intersection area between the reference point image and the preset position of the reference point is determined, and the area of ​​the intersection area is measured. Based on the comparison result between the area of ​​the intersection area and the preset area threshold, it is determined whether the defect type of the circuit board is solder joint misalignment.

[0087] In this embodiment, to detect whether solder joint misalignment has occurred on the circuit board, the reference point of the circuit board is typically soldered simultaneously with the solder joints (e.g., Figure 8If the solder at the circuit board solder joint shifts, the solder at the reference point also shifts synchronously, causing the solder at the reference point to deviate from the preset position of the reference point. Therefore, in this embodiment, the reference point image of the circuit board, i.e. the image of the reference point solder, can be segmented from the panoramic depth image of the circuit board at the vertical shooting angle using a preset model. Then, the reference point image is compared with the preset position of the reference point to determine whether the circuit board solder joint has shifted. Specifically, the intersection area of ​​the reference point image and the preset position of the reference point can be determined, and the area of ​​the intersection area can be measured. If the area of ​​the intersection area is less than a preset area threshold, it is determined that the circuit board solder joint has shifted. If the area of ​​the intersection area is not less than the preset area threshold, it is determined that the circuit board solder joint has not shifted.

[0088] In another alternative embodiment, for defect detection of solder overflow or bridging at circuit boards, since solder overflow or bridging does not involve depth, it can be considered as defect detection of a planar portion of the circuit board. Defect detection can be performed using a full-depth image of the circuit board from any shooting angle, preferably a full-depth image of the circuit board from a vertical shooting angle. See [link to relevant documentation]. Figure 9 The specific process can be as follows:

[0089] The solder area image is segmented from the panoramic depth image of the circuit board at a vertical shooting angle using the preset model;

[0090] Construct the minimum bounding rectangle of the solder area image, and determine whether the defect type of the circuit board is solder overflow or solder bridging based on the comparison result between the size of the minimum bounding rectangle and the preset size threshold.

[0091] In this embodiment, a preset model can be used to segment the solder area image of each solder joint from the panoramic depth image of the circuit board at a vertical shooting angle, and the minimum bounding rectangle of the solder area image of each solder joint can be constructed. Figure 9 The image shows only the minimum bounding rectangle of a solder joint solder area. In this embodiment, a preset size threshold, such as a length threshold and a width threshold, can be set for the minimum bounding rectangle of the solder joint solder area image. If the size of the minimum bounding rectangle of at least one solder joint solder area image on the circuit board exceeds the preset size threshold, it is determined that there is solder overflow on the circuit board. Solder bridging is caused by more severe solder overflow. Similarly, it can be determined whether there is bridging by comparing the size of the minimum bounding rectangle with the preset size threshold corresponding to the bridging defect. Alternatively, it can be directly identified by a preset model whether at least two solder joint solder area images are connected together.

[0092] In another alternative embodiment, for defect detection of foreign objects, scratches, or cracks on the solder joints of a circuit board, since these defects do not involve depth, they can be considered as defects in the planar portion of the circuit board. A full-view depth-of-field image of the circuit board at any shooting angle can be used for defect detection, preferably a full-view depth-of-field image of the circuit board at a vertical shooting angle. See [link to relevant documentation]. Figure 10 The specific process can be as follows:

[0093] The solder joint area image is segmented from the panoramic depth image of the circuit board from the vertical shooting angle using the preset model, and the defect type is classified according to the solder joint area image to determine whether the defect type of the solder joint of the circuit board is foreign object, scratch or crack.

[0094] In this embodiment, since foreign objects, scratches, or cracks on the solder joints of the circuit board are visually obvious defect types, the solder joint area can be directly identified from the panoramic depth image of the circuit board at a vertical shooting angle by a preset model. Based on the solder joint area image, image recognition and defect type classification are performed, and the classification result of the defect type of the solder joints of the circuit board is output.

[0095] Based on any of the above embodiments, different solder joint defect type detection methods can be executed in a preset order or in parallel, so as to ultimately determine whether there are defects in the solder joints of the circuit board, and if there are defects, to determine the specific defect type.

[0096] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0097] Furthermore, this disclosure also provides a panoramic depth image acquisition device for performing the panoramic depth image acquisition method in the above embodiments. The panoramic depth image acquisition device may include:

[0098] A control unit is used to control the relative movement between the camera bracket and the platform that carries the target object; wherein the camera bracket is used to fix multiple cameras at different shooting angles;

[0099] The acquisition unit is configured to take pictures of the target object by means of at least one camera when the camera bracket and the platform move to different distances relative to each other, and obtain different depth images of the target object at at least one shooting angle, so as to generate a full depth image of the target object at the corresponding shooting angle.

[0100] Optionally, after obtaining images of the target object at at least one shooting angle with different depths of field, the acquisition unit is further configured to:

[0101] By synthesizing different depth-of-field images of the target object from any shooting angle, a full depth-of-field image of the target object from any shooting angle is obtained.

[0102] Optionally, when the acquisition unit synthesizes different depth-of-field images of the target object from any shooting angle to obtain a full depth-of-field image of the target object from any shooting angle, it is used to:

[0103] Identify the sharp region in each image of the target object at different depth-of-field images under any shooting angle, and stitch the sharp regions of each image together to obtain a full depth-of-field image of the target object under any shooting angle.

[0104] Optionally, the acquisition unit is further configured to:

[0105] An initial point cloud image is generated by 3D reconstruction based on the different depth images of the target object at each shooting angle, and the initial point cloud image is colored based on the different depth images of the target object at each shooting angle to generate a true color point cloud image of the target object.

[0106] The apparatus provided in this embodiment can be used to execute the technical solution of the above-described panoramic depth image acquisition method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0107] Furthermore, this disclosure also provides a defect detection apparatus for performing the defect detection method in the above embodiments, the defect detection apparatus including:

[0108] The acquisition unit is used to acquire a panoramic depth image of the target object from at least one shooting angle, or to acquire a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object.

[0109] The detection unit is used to identify the defect type of the target object based on a panoramic depth image of the target object from at least one shooting angle, or based on a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object, using a preset model.

[0110] Optionally, the panoramic depth image and / or true-color point cloud image of the target object are obtained based on the panoramic depth image acquisition system as described in the above embodiments, or based on the panoramic depth image acquisition method as described in the above embodiments.

[0111] Optionally, when the detection unit identifies the defect type of the target object based on a panoramic depth image of the target object from at least one shooting angle, or based on a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object, using a preset model, it is used to:

[0112] For defects present in the planar portion of the target object, the type of defect is identified using the preset model based on the panoramic depth image of the target object from at least one shooting angle; or

[0113] For defects present in the surface depressions of the target object, the type of defect present in the surface depressions of the target object is identified by a preset model based on the panoramic depth image of the target object from a non-perpendicular shooting angle, or based on the panoramic depth image of the target object from a vertical shooting angle and the true color point cloud image of the target object.

[0114] Optionally, the target object is a circuit board;

[0115] Accordingly, when the detection unit identifies the type of defect in the surface depression of the target object based on the panoramic depth image of the target object taken from a non-perpendicular shooting angle and using a preset model, it is used to:

[0116] For the solder holes of the circuit board, the solder hole area image is identified from the panoramic depth image of the circuit board at a non-vertical shooting angle using the preset model, and the solder hole sidewall image is segmented from the solder hole area image.

[0117] The solder crawling height inside the solder hole is determined based on the solder hole sidewall image, and the defect type of the solder hole on the circuit board is determined based on the solder crawling height, namely, whether it is an empty solder or insufficient solder.

[0118] Optionally, the target object is a circuit board;

[0119] Accordingly, when the detection unit identifies the type of defect in the surface depression of the target object based on the panoramic depth image of the target object at the vertical shooting angle and the true-color point cloud image of the target object, and through a preset model, it is used to:

[0120] For the solder holes of the circuit board, the solder hole area image is identified from the panoramic depth image of the circuit board at the vertical shooting angle using the preset model, and the contour image of the solder crawling position inside the solder hole is segmented from the solder hole area image.

[0121] Based on the solder crawling location contour image, the solder crawling height inside the solder hole is determined from the true color point cloud image of the circuit board, and the defect type of the solder hole of the circuit board is determined based on the solder crawling height, whether it is an empty solder or insufficient solder.

[0122] Optionally, the target object is a circuit board;

[0123] Accordingly, when the detection unit identifies the type of defect in the planar portion of the target object based on the panoramic depth image of the target object at at least one shooting angle and using the preset model, it is used to:

[0124] The reference point image of the circuit board is segmented from the panoramic depth image of the circuit board at a vertical shooting angle using the preset model.

[0125] The intersection area between the reference point image and the preset position of the reference point is determined, and the area of ​​the intersection area is measured. Based on the comparison result between the area of ​​the intersection area and the preset area threshold, it is determined whether the defect type of the circuit board is solder joint misalignment.

[0126] Optionally, the target object is a circuit board;

[0127] Accordingly, when the detection unit identifies the type of defect in the planar portion of the target object based on the panoramic depth image of the target object at at least one shooting angle and using the preset model, it is used to:

[0128] The solder area image is segmented from the panoramic depth image of the circuit board at a vertical shooting angle using the preset model;

[0129] Construct the minimum bounding rectangle of the solder area image, and determine whether the defect type of the circuit board is solder overflow or solder bridging based on the comparison result between the size of the minimum bounding rectangle and the preset size threshold.

[0130] Optionally, the target object is a circuit board;

[0131] Accordingly, when the detection unit identifies the type of defect in the planar portion of the target object based on the panoramic depth image of the target object at at least one shooting angle and using the preset model, it is used to:

[0132] The solder joint area image is segmented from the panoramic depth image of the circuit board from the vertical shooting angle using the preset model, and the defect type is classified according to the solder joint area image to determine whether the defect type of the solder joint of the circuit board is foreign object, scratch or crack.

[0133] The apparatus provided in this embodiment can be used to execute the technical solution of the above-described defect detection method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0134] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an example embodiment of this disclosure. For example... Figure 11 As shown, the electronic device 1100 includes a processor 1101 and a memory 1102 communicatively connected to the processor 1101, the memory 1102 storing computer execution instructions.

[0135] The processor 1101 executes the computer execution instructions stored in the memory 1102 to implement the panoramic depth image acquisition method or defect detection method provided in any of the above method embodiments. The specific functions and technical effects that can be achieved will not be elaborated here.

[0136] This disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the panoramic depth image acquisition method or defect detection method provided in any of the above method embodiments.

[0137] This disclosure also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the panoramic depth image acquisition method or defect detection method provided in any of the above method embodiments.

[0138] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0141] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0143] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0144] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A panoramic depth image acquisition system, characterized in that, The panoramic depth image acquisition system includes: multiple cameras, camera brackets, and a platform for carrying the target object; The camera bracket is used to fix multiple cameras at different shooting angles; and the camera bracket and the platform can move relative to each other. At least one of the cameras is used to take pictures of the target object on the platform when the camera bracket and the platform move relative to each other to different distances, so as to obtain different depth images of the target object at the corresponding shooting angles, and to generate a full depth image of the target object at the corresponding shooting angles.

2. The panoramic depth image acquisition system according to claim 1, characterized in that, The panoramic depth image acquisition system also includes a processor; The processor is communicatively connected to multiple cameras and is used to synthesize different depth-of-field images of the target object from any shooting angle to obtain a full depth-of-field image of the target object from any shooting angle.

3. The panoramic depth image acquisition system according to claim 2, characterized in that, The processor is further configured to perform three-dimensional reconstruction based on different depth images of the target object at each shooting angle to generate an initial point cloud image, and to color the initial point cloud image based on different depth images of the target object at each shooting angle to generate a true-color point cloud image of the target object.

4. The panoramic depth image acquisition system according to any one of claims 1-3, characterized in that, The panoramic depth image acquisition system also includes a hemispherical light source for illuminating the target object on the platform from multiple angles. The hemispherical light source has multiple openings for multiple cameras to capture images of the target object on the platform from the multiple openings.

5. A method for acquiring panoramic depth images, characterized in that, include: Control the relative movement between the camera mount and the platform that carries the target object; The camera bracket is used to fix multiple cameras at different shooting angles; By moving at least one of the cameras relative to the camera bracket and the platform to different distances, the target object is photographed to obtain different depth-of-field images of the target object at at least one shooting angle, so as to generate a full depth-of-field image of the target object at the corresponding shooting angle.

6. The method according to claim 5, characterized in that, After obtaining images of the target object at different depths of field from at least one shooting angle, the method further includes: By synthesizing different depth-of-field images of the target object from any shooting angle, a full depth-of-field image of the target object from any shooting angle is obtained.

7. The method according to claim 6, characterized in that, The step of synthesizing different depth-of-field images of the target object from any shooting angle to obtain a full depth-of-field image of the target object from any shooting angle includes: Identify the sharp region in each image of the target object at different depth-of-field images under any shooting angle, and stitch the sharp regions of each image together to obtain a full depth-of-field image of the target object under any shooting angle.

8. The method according to any one of claims 5-7, characterized in that, The method further includes: An initial point cloud image is generated by 3D reconstruction based on the different depth images of the target object at each shooting angle, and the initial point cloud image is colored based on the different depth images of the target object at each shooting angle to generate a true color point cloud image of the target object.

9. A defect detection method, characterized in that, include: Acquire a panoramic depth image of the target object from at least one shooting angle, or acquire a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object. Based on the panoramic depth image of the target object from at least one shooting angle, or based on the panoramic depth image of the target object from at least one shooting angle and the true-color point cloud image of the target object, the defect type of the target object is identified by a preset model.

10. The method according to claim 9, characterized in that, The panoramic depth image and / or true-color point cloud image of the target object are obtained based on the panoramic depth image acquisition system as described in any one of claims 1-4, or based on the panoramic depth image acquisition method as described in any one of claims 5-8.

11. The method according to claim 9, characterized in that, The step of identifying the defect type of the target object using a preset model based on a panoramic depth image of the target object from at least one shooting angle, or based on a panoramic depth image of the target object from at least one shooting angle and a true-color point cloud image of the target object, includes: For defects present in the planar portion of the target object, the type of defect is identified using the preset model based on the panoramic depth image of the target object from at least one shooting angle; or For defects present in the surface depressions of the target object, the type of defect present in the surface depressions of the target object is identified by a preset model based on the panoramic depth image of the target object from a non-perpendicular shooting angle, or based on the panoramic depth image of the target object from a vertical shooting angle and the true color point cloud image of the target object.

12. The method according to claim 11, characterized in that, The target object is a circuit board; Accordingly, regarding the defects existing in the surface depressions of the target object, based on the panoramic depth image of the target object taken from a non-perpendicular shooting angle, the type of defect existing in the surface depressions of the target object is identified by a preset model, including: For the solder holes of the circuit board, the solder hole area image is identified from the panoramic depth image of the circuit board at a non-perpendicular shooting angle using the preset model, and the solder hole sidewall image is segmented from the solder hole area image. The solder crawling height inside the solder hole is determined based on the solder hole sidewall image, and the defect type of the solder hole on the circuit board is determined based on the solder crawling height, namely, whether it is an empty solder or insufficient solder.

13. The method according to claim 11, characterized in that, The target object is a circuit board; Accordingly, regarding the defects present in the surface depressions of the target object, based on the panoramic depth image of the target object at the vertical shooting angle and the true-color point cloud image of the target object, the type of defect present in the surface depressions of the target object is identified through a preset model, including: For the solder holes of the circuit board, the solder hole area image is identified from the panoramic depth image of the circuit board at the vertical shooting angle using the preset model, and the contour image of the solder crawling position inside the solder hole is segmented from the solder hole area image. Based on the solder crawling location contour image, the solder crawling height inside the solder hole is determined from the true color point cloud image of the circuit board, and the defect type of the solder hole of the circuit board is determined based on the solder crawling height, whether it is an empty solder or insufficient solder.

14. The method according to claim 11, characterized in that, The target object is a circuit board; Accordingly, regarding defects existing in the planar portion of the target object, based on the panoramic depth image of the target object at at least one shooting angle, the type of defect existing in the planar portion of the target object is identified through the preset model, including: The reference point image of the circuit board is segmented from the panoramic depth image of the circuit board at a vertical shooting angle using the preset model. The intersection area between the reference point image and the preset position of the reference point is determined, and the area of ​​the intersection area is measured. Based on the comparison result between the area of ​​the intersection area and the preset area threshold, it is determined whether the defect type of the circuit board is solder joint misalignment.

15. The method according to claim 11, characterized in that, The target object is a circuit board; Accordingly, regarding defects existing in the planar portion of the target object, based on the panoramic depth image of the target object at at least one shooting angle, the type of defect existing in the planar portion of the target object is identified through the preset model, including: The solder area image is segmented from the panoramic depth image of the circuit board at a vertical shooting angle using the preset model; Construct the minimum bounding rectangle of the solder area image, and determine whether the defect type of the circuit board is solder overflow or solder bridging based on the comparison result between the size of the minimum bounding rectangle and the preset size threshold.

16. The method according to claim 11, characterized in that, The target object is a circuit board; Accordingly, regarding defects existing in the planar portion of the target object, based on the panoramic depth image of the target object at at least one shooting angle, the type of defect existing in the planar portion of the target object is identified through the preset model, including: The solder joint area image is segmented from the panoramic depth image of the circuit board from the vertical shooting angle using the preset model, and the defect type is classified according to the solder joint area image to determine whether the defect type of the solder joint of the circuit board is foreign object, scratch or crack.

17. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 5-16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of claims 5-16.

Images