3D-AOI detection equipment and three-dimensional reconstruction method

Abstract

The invention relates to 3D-AOI detection equipment and a three-dimensional reconstruction method. According to the method, an optical machine, a first industrial camera, a plurality of second industrial cameras and a coaxial telecentric lens are included; the optical machine is connected with the coaxial telecentric lens and is used for emitting structured light, and the structured light enters from the side view of the coaxial telecentric lens and is vertically projected to the measured object after being reflected by the inclined plane of an internal reflective mirror; the imaging end of the first industrial camera is in view connection with the top surface of the coaxial telecentric lens, and a light path of the first industrial camera penetrates through the center of the coaxial telecentric lens and extends to a measured object; the plurality of second industrial cameras are arranged around the first industrial camera, and the lens of each second industrial camera is aligned with a measured object at a set inclination angle. By adopting the method, the scanning efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional automatic optical inspection technology, and in particular to a 3D-AOI inspection device and a three-dimensional reconstruction method. Background Art

[0002] In today's intelligent manufacturing field, three-dimensional automatic optical inspection (3D-AOI) equipment plays a vital role in ensuring product quality, and the layout architecture and hardware design of 3D-AOI equipment are the core links.

[0003] Currently, the commonly used 3D-AOI equipment layout architecture is one camera and four optical machines, or one camera and eight optical machines. These systems employ four or eight monocular vision systems, with only one monocular vision system performing each scan. During the scan, the optical machine projects coded structured light onto the object being inspected, and one camera completes the acquisition, representing a single scan. This traditional 3D-AOI equipment layout requires four or eight scans to complete a single inspection, resulting in low scanning efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a 3D-AOI inspection device and a three-dimensional reconstruction method that can improve scanning efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a 3D-AOI inspection device, comprising:

[0006] Optical machine, first industrial camera, multiple second industrial cameras, and coaxial telecentric lens;

[0007] The optical engine is connected to the coaxial telecentric lens and is used to emit structured light. The structured light is incident from the side field of view of the coaxial telecentric lens, reflected by the internal reflective mirror oblique surface, and then projected vertically onto the object to be measured;

[0008] The imaging end of the first industrial camera is connected to the top surface field of view of the coaxial telecentric lens, and the optical path of the first industrial camera passes through the center of the coaxial telecentric lens and extends to the object to be measured;

[0009] The plurality of second industrial cameras are arranged around a coaxial telecentric lens, and a lens of each second industrial camera is aimed at the object to be measured at a set inclination angle.

[0010] Among them, the lens of the second industrial camera adopts a Sham structure to increase the depth of field of the measurement space.

[0011] In one embodiment, the first industrial camera is a high-resolution industrial camera;

[0012] The plurality of second industrial cameras are four high-speed industrial cameras, and the four high-speed industrial cameras are symmetrically arranged around the high-resolution industrial camera.

[0013] In one embodiment, the 3D-AOI inspection equipment further includes:

[0014] An annular LED fill light is arranged around the end of the coaxial telecentric lens and is used to provide lighting for imaging of the first industrial camera.

[0015] In one embodiment, the coaxial telecentric lens has a built-in beam splitter;

[0016] The reflectivity of the spectroscope for the target visible light band is greater than the preset reflectivity, and the transmittance of the spectroscope for other visible light bands is greater than the preset transmittance;

[0017] The other visible light bands are bands other than the target visible light band.

[0018] In one embodiment, the target visible light band is a blue light band with a central wavelength of 460±5 nm.

[0019] In one embodiment, the sum of the preset reflectivity and the preset transmittance is greater than or equal to 100%.

[0020] In one embodiment, the spectrometer comprises alternating stacks of high-refractive-index layers and low-refractive-index layers, wherein the refractive index of the high-refractive-index layers for the target light wavelength band is greater than a first preset refractive index, and the refractive index of the low-refractive-index layers for the target light wavelength band is less than a second preset refractive index.

[0021] In one embodiment, the spectrometer is deposited and formed by electron beam evaporation, and the spectrometer is deposited and formed by ion beam assisted deposition during the deposition process, and is annealed after deposition.

[0022] In one embodiment, the four high-speed industrial cameras are arranged at equal angular intervals of 90°, the angles between the optical axes of the four high-speed industrial cameras and the optical axis of the high-resolution industrial camera are within a preset angle range, and the distances between the center points of the lenses of the four high-speed industrial cameras and the surface of the object to be measured are equal.

[0023] Among them, the lenses of the four high-speed industrial cameras all adopt a Sham structure to increase the depth of field of the measurement space.

[0024] In a second aspect, the present application further provides a three-dimensional reconstruction method, which is implemented based on the 3D-AOI inspection device described in the first aspect, and includes:

[0025] Controlling the optical machine to project structured light onto the object to be measured, wherein the structured light is incident from the side field of view of the coaxial telecentric lens, is reflected by the internal reflective mirror oblique surface, and is projected vertically onto the object to be measured;

[0026] Controlling a plurality of second industrial cameras to synchronously capture multi-angle structured light images of the surface of the object to be measured;

[0027] Based on the multi-angle structured light image, calculating absolute phase data in the images captured by the plurality of second industrial cameras;

[0028] Based on the absolute phase data and the mapping relationship between the phase and the three-dimensional points, a three-dimensional reconstruction model of the object under test is generated.

[0029] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the second aspect when executing the computer program.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the second aspect when the computer program is executed by a processor.

[0031] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which implements the method described in the second aspect when executed by a processor.

[0032] The above-mentioned 3D-AOI inspection equipment includes: an optical machine, a first industrial camera, multiple second industrial cameras, and a coaxial telecentric lens; the output end of the optical machine is coaxially connected to the first port of the coaxial telecentric lens, and the emitted structured light is reflected to the surface of the object through the inclined surface of the spectrometer; the imaging end of the first industrial camera is coaxially connected to the second port of the coaxial telecentric lens, and the optical axis passes through the center of the coaxial telecentric lens and extends to the object to be measured; multiple second industrial cameras are arranged around the first industrial camera, and the lens of each second industrial camera is aligned with the object to be measured at a set inclination angle. Through this solution, the above-mentioned 3D-AOI inspection equipment emits structured light through the coaxial docking of the optical machine and the coaxial telecentric lens. At the same time, multiple second industrial cameras surround the first camera and aim at the object to be measured at a set angle, forming a layout architecture for multi-perspective synchronous acquisition. The first industrial camera then captures a two-dimensional high-definition image through the coaxial telecentric lens. Compared with the traditional solution of 1 camera and 4 or 8 optical machines, only one set of monocular vision works at a time and multiple scans are required. This solution uses multiple second industrial cameras to collect structured light information of the object to be measured from different angles at the same time, and can complete multi-dimensional data acquisition in a single scan, without the need for four or eight scans, thereby significantly improving scanning efficiency and effectively solving the technical problem of low detection efficiency due to the large number of scans in the traditional layout architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 A schematic diagram of the architecture of a 3D-AOI inspection device in one embodiment;

[0035] Figure 2 1 is a flow chart of a three-dimensional reconstruction method according to an embodiment;

[0036] Figure 3 1 is a flow chart of a method for calibrating a light model in one embodiment;

[0037] Figure 4 1 is a flow chart of a 3D reconstruction method based on phase-3D mapping technology in one embodiment;

[0038] Figure 5 FIG. 4 is a structural block diagram of a three-dimensional reconstruction device in an embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] Industrial cameras are key components of machine vision systems. They can convert optical signals into ordered electrical signals and have high image stability, high transmission and anti-interference capabilities. Based on CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) chips, they are classified in various ways (such as by chip and sensor structure). These directly determine the resolution and quality of the captured images and are also related to the system operating mode. They are widely used in scenarios such as industrial inspection.

[0041] High-speed industrial cameras have high acquisition rates and frame rates, and can quickly capture large amounts of image data. Combined with high-performance processors and high-speed transmission interfaces, they can clearly record subtle movements and transient changes of high-speed moving objects. They are suitable for quality inspection in industrial production and manufacturing, high-speed assembly line monitoring, and observing rapid processes in scientific research, military industry and other fields.

[0042] High-resolution industrial cameras are equipped with high-resolution image sensors with small pixel sizes (some can reach 3.5um×3.5um, etc.), which can achieve large-format, high-definition imaging, ensure that images are rich in details, and meet the needs of high-precision measurement and inspection. They are used to capture subtle features and assist in accurate interpretation in fields such as printed products, LCD (Liquid Crystal Display) panel inspection, and PCB (Printed Circuit Board) inspection.

[0043] The light engine is one of the core components of 3D-AOI inspection equipment. Its function is to generate and emit structured light in a specific pattern for three-dimensional measurement and defect detection.

[0044] A coaxial telecentric lens is a specialized optical imaging system characterized by a coaxial, collinear optical path with two fields of view, and the lens's object-side principal ray (the key imaging ray) parallel to the optical axis. This design, through optical structures such as dual or single telecentric optical paths, eliminates the perspective distortion and magnification differences associated with traditional lenses, ensuring highly consistent object size and shape at varying object distances.

[0045] In an exemplary embodiment, Figure 1 As shown, a schematic diagram of the architecture of a 3D-AOI inspection device is provided, which includes: an optical machine 11, a first industrial camera 12, a plurality of second industrial cameras 13, and a coaxial telecentric lens 14;

[0046] The optical engine 11 is connected to the coaxial telecentric lens 14 and is used to emit structured light. The structured light is incident from the side field of view of the coaxial telecentric lens 14, and is reflected by the internal reflective mirror oblique surface and then projected vertically to the object to be measured;

[0047] The imaging end of the first industrial camera 12 is connected to the top surface field of view of the coaxial telecentric lens 14, and the optical path of the first industrial camera 12 extends through the center of the coaxial telecentric lens 14 to the object to be measured;

[0048] A plurality of second industrial cameras 13 are arranged around the first industrial camera 12 , and a lens of each second industrial camera 13 is aimed at the object to be measured at a set inclination angle.

[0049] The coaxial telecentric lens 14 has a dual field of view, with two viewing ports. The side port is the input port for the optical engine 11. The optical engine 11 directs structured light incident on the beam splitter within the coaxial telecentric lens 14, reflecting it onto the object below. Multiple secondary industrial cameras 13 then simultaneously capture the image for 3D reconstruction. Ambient light then illuminates the object, and the primary industrial camera 12, with a separate field of view on the main axis, captures a high-definition 2D image.

[0050] The above-mentioned multiple second industrial cameras 13 are aimed at the object in a surrounding posture and at a set inclination angle, forming a multi-directional visual network centered on the first industrial camera 12. This layout allows each camera to synchronously capture the structured light deformation information of the object at different angles; compared with the traditional solution in which a single camera cooperates with multiple optical machines to work in time-sharing and requires multiple scans to piece together data, this solution uses multiple cameras to simultaneously collect multi-dimensional images, and completes three-dimensional reconstruction in one go after algorithm fusion, simplifying the scanning process that originally required 4 or 8 times, greatly reducing the number of scans, and thus improving scanning efficiency.

[0051] In the above-mentioned embodiment, the 3D-AOI inspection equipment emits structured light through the coaxial connection between an optical engine and a coaxial telecentric lens. Multiple second industrial cameras initiate scanning to capture images of the object under test, projecting structured light. Then, without projecting structured light, the first industrial camera uses the coaxial telecentric lens to capture images of the object under test under ambient light. The inspection process is then completed based on the images of the object under test, projected structured light, and under ambient light, captured by the multiple second industrial cameras. Compared to traditional solutions that utilize a single camera and four or eight optical engines, where only one monocular vision system operates at a time and multiple scans are required, this solution utilizes multiple second industrial cameras to simultaneously capture structured light information from different angles. Combined with the central perspective imaging of the first industrial camera, this solution allows for multi-dimensional data acquisition in a single scan, eliminating the need for four or eight separate scans. This significantly improves scanning efficiency and effectively addresses the technical issue of low inspection efficiency associated with traditional layouts due to the high number of scans.

[0052] Among them, the lens of the second industrial camera adopts a Sham structure to increase the depth of field of the measurement space.

[0053] In some embodiments, in the 3D-AOI inspection device, the first industrial camera may be a high-resolution industrial camera, and the plurality of second industrial cameras may be four high-speed industrial cameras, which are symmetrically arranged around the high-resolution industrial camera.

[0054] Among them, the lenses of the four high-speed industrial cameras all adopt a Sham structure to increase the depth of field of the measurement space.

[0055] Among them, the high-resolution industrial camera uses high imaging accuracy to capture subtle image data from the central perspective of the object being measured, while four high-speed industrial cameras are arranged in a surrounding manner, synchronously recording the object's multi-directional three-dimensional shape data with a millisecond response speed. This hardware combination of high-resolution industrial cameras and high-speed industrial cameras realizes the rapid processing of detail resolution and three-dimensional measurement in three-dimensional inspection.

[0056] The four high-speed cameras are arranged symmetrically around a central high-resolution camera, providing 360° uniform viewing angle coverage. The inclination angles of each camera's optical axis relative to the central optical axis are optimized using a Sham structure to reduce the projection distortion of the structured light pattern on the object surface, expand the measurement depth of field, and achieve full-view consistency of 3D point cloud data in conjunction with the distortion-free optical path of the coaxial telecentric lens.

[0057] In some embodiments, four high-speed industrial cameras are arranged around at equal 90° intervals, the angle between the optical axes of the four high-speed industrial cameras and the optical axis of the high-resolution industrial camera is within a preset angle range, and the distance between the center points of the lenses of the four high-speed industrial cameras and the surface of the object to be measured is equal.

[0058] Four high-speed industrial cameras are arranged at 90° intervals around the perimeter, providing uniform 360° coverage. This eliminates blind spots compared to non-equidistant layouts. For example, when inspecting curved components, each camera simultaneously captures the structured light fringes in different orientations, preventing distortion in 3D contour reconstruction caused by loss of perspective. This symmetrical distribution ensures consistent lighting and imaging conditions in all directions of the object under inspection, making it particularly suitable for comprehensive defect inspection of symmetrical structures such as solder joints and connector pins in semiconductor packages.

[0059] In the embodiment of the present application, the above-mentioned preset angle range can be set according to actual conditions and is not limited in the embodiment of the present application.

[0060] For example, the preset angle range may be 20°-30° to optimize the structured light projection and reflection paths.

[0061] Among them, if the angle between the optical axis of the high-speed industrial camera and the vertical optical axis is less than 20°, the measurement accuracy will be affected. If the angle between the optical axis of the high-speed industrial camera and the vertical optical axis is greater than 30°, the 3D reconstruction data will be lost due to excessive occlusion.

[0062] The center point of the lens is equidistant from the object being measured, which ensures that the object distances of each camera are consistent, avoids inconsistent imaging ratios due to differences in object distances, and simplifies the subsequent calibration and registration process of multi-view images.

[0063] In some embodiments, the 3D-AOI inspection device may further include a ring-shaped fill light disposed around the end of the coaxial telecentric lens to provide illumination for the first industrial camera. The ring-shaped fill light may be a ring-shaped LED fill light.

[0064] The annular LED fill lights are symmetrically distributed in an annular pattern with the coaxial telecentric lens as the center.

[0065] The above-mentioned method of setting the fill light can eliminate shadows caused by height differences or material reflections on the surface of the object being measured. At the same time, combined with the optical path characteristics of the coaxial telecentric lens, it reduces ambient light interference and improves image contrast, ensuring that the accuracy of the recognition of subtle defects is improved during high-resolution imaging by the First Industrial Camera. It can effectively solve the problem of uneven lighting in the inspection of complex contours or reflective surfaces, and enhance the adaptability of the equipment in various industrial scenarios.

[0066] In some embodiments, in the above-mentioned 3D-AOI inspection equipment, the coaxial telecentric lens has a built-in spectrometer, the reflectivity of the spectrometer to the target light band is greater than the preset reflectivity, and the transmittance of the spectrometer to other light bands is greater than the preset transmittance.

[0067] Among them, other optical bands are bands other than the target optical band.

[0068] Exemplarily, the target visible light band may be a blue light band with a central wavelength of 460±5 nm.

[0069] Exemplarily, the target visible light band may also be an infrared light band.

[0070] In some embodiments, the sum of the preset reflectivity and the preset transmittance is greater than or equal to 100%.

[0071] The semi-transparent, semi-reflective design of the beamsplitter used in existing technologies faces the challenge of a 50% reduction in light efficiency. In the above embodiment, however, the coaxial telecentric lens used in the beamsplitter design is fully reflective for blue light with a central wavelength of 460±5nm, while fully transparent for light of other wavelengths. This increases light efficiency to nearly 100%, a 2-fold improvement.

[0072] In some embodiments, the beam splitter includes alternating stacks of high refractive index layers and low refractive index layers, wherein the refractive index of the high refractive index layer for the target light wavelength band is greater than a first preset refractive index, and the refractive index of the low refractive index layer for the target light wavelength band is less than a second preset refractive index.

[0073] Among them, the above-mentioned first preset refractive index and second preset refractive index are not limited in this application and can be set according to actual needs.

[0074] In some embodiments, when the target visible light band is a blue light band with a central wavelength of 460±5 nm, the beam splitter includes alternately stacked tantalum pentoxide high refractive index layers and silicon dioxide low refractive index layers.

[0075] It should be noted that, for different visible light bands, different high refractive index layers and low refractive index layers may be provided in the above-mentioned spectroscope, which is not specifically limited in the embodiment of the present application.

[0076] In some embodiments, the preparation method of the spectrometer includes but is not limited to: the spectrometer is deposited by electron beam evaporation, the spectrometer is deposited by ion beam assisted deposition during the deposition process, and annealing is performed after deposition.

[0077] It should be noted that the above-mentioned preparation method of the spectrometer is only an exemplary description. The spectrometer in the embodiment of the present application can also be prepared by other methods. Any preparation method that can achieve the characteristics of the above-mentioned spectrometer can be used to prepare the above-mentioned spectrometer.

[0078] In the 460nm blue light band, the refractive index of tantalum pentoxide films is typically 2.08–2.18 (typically around 2.15). This high refractive index stems from the inherent optical properties of the material. The dense films produced through ion beam sputtering technology can further approach the theoretical upper limit of the refractive index, enhancing light reflection and interference effects. In the 460nm blue light band, the refractive index of tantalum pentoxide films is typically 2.20–2.28 (typically around 2.25). When stacked alternately with silicon dioxide, the refractive index difference creates strong optical interference, achieving high reflection in the target band and high transmission in other bands.

[0079] The beam splitter uses a multilayer dielectric film structure consisting of alternating high-refractive-index layers of tantalum pentoxide and low-refractive-index layers of silicon dioxide. The difference between high and low refractive indices creates an interference filtering effect. The reflectivity in the target wavelength band (such as 460nm blue light) is greater than the preset reflectivity, while the transmittance in other wavelength bands is greater than the preset transmittance. The light energy utilization rate is doubled compared to traditional single-layer coatings.

[0080] In the fabrication process of the aforementioned spectrometer, multiple technologies collaborate to ensure both optical performance and reliability. Electron beam evaporation technology precisely controls the thickness error of individual layers with precise power and timing. Ion beam-assisted deposition increases film density and reduces surface roughness. Annealing reduces internal stress and improves refractive index stability. The resulting film has low water vapor permeability, minimal reflectivity degradation in hot and humid environments, strong bonding strength, and no shedding during vibration testing. This significantly improves yield and reduces costs during large-scale production, meeting the long-term reliable operation and mass production requirements of industrial scenarios.

[0081] In an exemplary embodiment, Figure 2As shown, a flow chart of a three-dimensional reconstruction method is provided. The method is implemented based on the 3D-AOI inspection device in the above embodiment. The method flow can be executed under the control of an FPGA timing control circuit and may include the following steps:

[0082] 201. Control the optical machine to project structured light onto the object to be measured. The structured light is reflected by the coaxial telecentric lens and then projected vertically onto the object to be measured.

[0083] In some embodiments, the structured light may be blue light with a central wavelength of 460 nm.

[0084] The optical machine projects structured light under the triggering of the Field-Programmable Gate Array (FPGA) timing control circuit.

[0085] The upper data coaxial telecentric lens has a built-in beam splitter. The multi-layer dielectric film of the beam splitter is an alternating stack of titanium dioxide high-refractive index layers and silicon dioxide low-refractive index layers. It can highly reflect 460nm blue light and highly transmit other light. The structured light transmitted by the optical machine is reflected vertically along the optical axis of the lens to the surface of the object being measured, forming a regular distortion pattern.

[0086] 202. Control multiple second industrial cameras to synchronously capture multi-angle structured light images of the surface of the object to be measured.

[0087] Four secondary industrial cameras (such as high-speed cameras) can be arranged at 90° angles around a primary industrial camera (such as a high-resolution camera), with the lens centers equidistant from the surface of the object being measured. An FPGA timing circuit synchronously triggers the four cameras to capture structured light distortion caused by the three-dimensional contours of the surface, providing a 360-degree spatial perspective.

[0088] 203. Based on the multi-angle structured light image, calculate the absolute phase data in the images captured by the multiple second industrial cameras.

[0089] Among them, the phase solution algorithm can be used to analyze the offset and distortion of the stripes in the multi-angle structured light image. Combined with the pre-calibrated phase-3D mapping relationship (determined by ray reprojection calibration and sampling mapping calibration), the absolute phase value of each pixel point is calculated. This absolute phase value is used to characterize the structured light phase change caused by the height modulation of the object surface.

[0090] Exemplarily, the phase calculation algorithm may be Gray code decoding or a phase shift method.

[0091] 204. Based on the absolute phase data and the mapping relationship between the phase and the three-dimensional points, a three-dimensional reconstruction model of the object under test is generated.

[0092] Among them, the absolute phase data of four high-speed industrial cameras are converted into three-dimensional coordinates through pre-calibrated phase-depth mapping coefficients, and the phase values ​​are fused to generate a three-dimensional reconstruction model.

[0093] The 3D reconstruction method in the above embodiment leverages the innovative architecture of 3D-AOI inspection equipment, achieving multiple breakthroughs in inspection efficiency, accuracy, and resolution. FPGA-controlled optical engines project 460nm blue structured light, which is then projected vertically through a coaxial telecentric lens and a highly reflective beam splitter. This doubles the optical efficiency of traditional solutions. Combined with four high-speed industrial cameras for simultaneous acquisition, the method reduces the number of scans from the traditional 4-8 to 1, significantly improving scanning efficiency.

[0094] In an embodiment of the present application, the inspection process using the above-mentioned 3D-AOI inspection equipment may further include: controlling a first industrial camera to capture a vertical two-dimensional image of the object under inspection, when the optical machine does not project structured light onto the object under inspection. Subsequently, the inspection process of the object under inspection is completed based on the images of the object under inspection projected with structured light, captured by multiple second industrial cameras, and the vertical two-dimensional images of the object under inspection under ambient light.

[0095] The situation where the optical machine does not project the structured light onto the object under test may refer to the situation where the optical machine stops projecting the structured light or starts transmitting the structured light.

[0096] The aforementioned first-generation industrial camera (e.g., a high-resolution industrial camera) can capture a two-dimensional color image of the object under test from a vertical perspective under the illumination of a ring LED fill light. In this case, the coaxial telecentric lens beam splitter has high visible light transmittance (e.g., >90% for white light), ensuring a clear image free of structured light interference.

[0097] In some embodiments, during the three-dimensional reconstruction process, calibration can be performed first using a light model calibration method, and then the three-dimensional reconstruction of the object to be measured can be achieved using a phase-three-dimensional mapping technology.

[0098] In the calibration method based on the light model, since the light model calibrates the light corresponding to each pixel of the optical machine, it will not be affected by lens distortion. In addition, it is reconstructed through phase-three-dimensional mapping technology, which also has a better control effect on the nonlinear response error in the system, and therefore can achieve higher measurement accuracy and repeatability.

[0099] The idea of ​​the above light model is that any imaging system will collect incident light from a scene onto a photosensitive element.

[0100] For example, the photosensitive element may be a complementary metal oxide semiconductor (CMOS).

[0101] Among them, each pixel of the photosensitive element corresponds to a specific beam of light. Within the focal area of ​​the imaging system, a beam of light can be considered to be well-focused, especially when studying the geometric characteristics of the imaging system. Therefore, each pixel on the imaging surface corresponds to a specific beam of incident light. In this sense, the imaging system can be regarded as a black box, so the correspondence between the pixel point and the incident light can be established; the calibration idea of ​​the light model is to try to track the trajectory of the light in the measurement space. The specific principle can be understood as obtaining the three-dimensional coordinates of several spatial points corresponding to the pixels in the space. Within the scope of geometric optics and without considering errors, it is believed that these spatial points should be on a straight line, and this straight line can be considered as the light corresponding to the pixel. The spatial equation of the line can be solved by obtaining the coordinates of the spatial point.

[0102] In an exemplary embodiment, Figure 3 As shown, Figure 3 FIG. 1 is a flow chart of a method for calibrating a light model in one embodiment. The method for calibrating a light model may include the following steps:

[0103] 301. Prepare the target.

[0104] Among them, a plane target with n reference points printed on the surface can be prepared, and different numbers of reference points can be selected according to the requirements for accuracy.

[0105] For example, a planar target with n fiducials printed on its surface can be prepared. The number of fiducials is determined by the required detection accuracy. The target material must meet the requirements of uniform reflectivity and high flatness to ensure that the fiducials are easily identifiable in the image and do not deform.

[0106] The planar target with n reference points printed on its surface can be in the form of a checkerboard or dot matrix. The higher the detection accuracy required, the larger the value of n. The target material can be glass or a metal substrate.

[0107] In this embodiment of the present application, the target can be pre-calibrated using the optical machine and coaxial telecentric lens of the 3D-AOI equipment to ensure that the target plane is perpendicular to the optical axis. 302. Place the target.

[0108] Among them, m different positions can be placed arbitrarily in space to meet the calibration requirements.

[0109] For example, the target can be placed at m different locations in the measurement space and high-precision positioning can be achieved using mechanical fixtures or a translation stage. The principle of placement can be to randomly distribute the angle between the target plane normal and the camera optical axis within the range of 0°-20° to ensure multi-view coverage and avoid calibration blind spots.

[0110] Exemplarily, the above m is greater than or equal to 6, so that different pitch angles and yaw angles can be covered.

[0111] In this embodiment, the target is secured by the 3D-AOI equipment's mechanical fixture (e.g., a ring bracket). An FPGA timing control circuit synchronously triggers the translation stage, positioning the target at a preset angle within the measurement space. This ensures that the field of view of four high-speed industrial cameras (arranged in a 90° circle) covers the entire target area. 303. Acquire a uniformly illuminated image.

[0112] In each placement position, a uniformly illuminated target image is collected to locate and identify the image coordinates of the reference point.

[0113] For example, at each target placement, use the first industrial camera (i.e., a high-resolution industrial camera) to capture target images under uniform illumination from a ring LED fill light. Maintain consistent illumination intensity to avoid fiducial recognition errors caused by glare or shadows. Set the image resolution to the camera's highest resolution for subsequent sub-pixel positioning of the fiducials.

[0114] In the embodiment of the present application, the ring LED fill light of the device is enabled and the image is captured by a high-resolution industrial camera. At this time, the optical machine stops projecting structured light, and the coaxial telecentric lens spectrometer has a white light transmittance of >90%, avoiding structured light interference.

[0115] 304. Control the optical machine projection and collect the fringe image.

[0116] The orthogonal stripe structured light is projected onto the target, and the stripe image of the target is collected to calculate the orthogonal absolute phase map.

[0117] For example, the optical engine projects orthogonal stripe structured light onto the target surface. The structured light has a central wavelength of 460nm blue light and is projected vertically through a coaxial telecentric lens. Four secondary industrial cameras, arranged in a 90° surround configuration, synchronously capture the stripe images, ensuring that the phase changes of the stripe pattern are accurately recorded.

[0118] In the embodiment of the present application, the optical machine projects orthogonal stripe structured light under the timing control of the FPGA, and projects it vertically onto the target after high reflection through the coaxial telecentric lens spectrometer; four high-speed industrial cameras synchronously capture stripe distortion images, and the center points of their lenses are equidistant from the target surface to ensure consistent proportions of multi-view images.

[0119] 305. Calculate the absolute phase map.

[0120] Wherein, an orthogonal absolute phase map is calculated based on the collected fringe image.

[0121] For example, the phase of the captured fringe image can be solved using a phase shifting method or a Gray code decoding algorithm. Taking the four-step phase shifting method as an example, the wrapped phase is calculated using four phase-shifted fringe images. Phase unwrapping is then performed using the encoded information of the Gray code fringe. This yields an orthogonal absolute phase map of the target surface, with phase values ​​ranging from [0 to 2π]. This map represents the distortion of the structured light due to the height of the target plane. 306. Construct a cost function.

[0122] Among them, similar to binocular calibration, a cost function for ray reprojection calibration is constructed.

[0123] For example, based on the principle of ray reprojection, a cost function centered on spatial distance error is constructed. For each fiducial point, its image coordinates are back-projected into a spatial ray using the camera's intrinsic parameters, and the spatial distance between this ray and the fiducial point's actual 3D coordinates is calculated. The cost function is the sum of the squares of the spatial distances to all fiducial points.

[0124] The spatial distance between the ray and the actual three-dimensional coordinates of the reference point is the distance from the point to the line. 307. Optimization solution.

[0125] Among them, a suitable optimization algorithm can be selected to complete the ray reprojection calibration by minimizing the reprojection error (that is, minimizing the spatial distance from the object point to the back-projected spatial ray) and determine the system parameters of the back-projected stereo vision model, including the camera's intrinsic parameters, lens distortion, and rotation and translation parameters.

[0126] For example, the Levenberg-Marquardt (LM) algorithm can be used to iteratively optimize the cost function and adjust the system parameters of the back-projection stereo vision model. These parameters include: camera intrinsic parameters (focal length, principal point coordinates, and radial or tangential distortion coefficients); and extrinsic parameters (rotation matrix and translation vector, which describe the camera's position in the world coordinate system). The optimization goal is to minimize spatial distance error, ultimately obtaining high-precision camera parameters and eliminating the effects of lens distortion.

[0127] Among them, the LM algorithm is a nonlinear least squares optimization algorithm that combines the advantages of the gradient descent method and the Gauss-Newton method. It is used to solve the optimal solution for camera parameters in the light model calibration of 3D-AOI equipment.

[0128] In the embodiment of the present application, the FPGA timing circuit synchronizes the trigger signals of each camera and the optical machine to ensure the timestamp consistency of multi-view data during the back-projection ray calculation, thereby improving the optimization efficiency of the LM algorithm.

[0129] 308. Obtain calibration data.

[0130] Among them, the calibrated stereo vision system parameters can be used to calculate the spatial posture of the target, and the three-dimensional coordinates and corresponding absolute phase values ​​of sufficient spatial points can be obtained by using the intersection of the light corresponding to each pixel and the target plane.

[0131] For example, the camera parameters calibrated in step 307 may be used to project structured light onto the object to be measured and capture an image, and the absolute phase value may be obtained through phase calculation.

[0132] In this embodiment, the absolute phase is calculated using multi-angle structured light images from the device's four high-speed industrial cameras, combined with the distortion-free optical path of a coaxial telecentric lens to reduce three-dimensional coordinate measurement errors. 309. Fitting mapping coefficients.

[0133] Based on the acquired three-dimensional coordinates of the spatial point and the absolute phase value, coefficient fitting is performed to determine the phase-to-three-dimensional mapping coefficients.

[0134] For example, a polynomial fitting method may be used to model the mapping relationship between the phase values ​​in the calibration data set and the three-dimensional coordinates.

[0135] The light model calibration method embodiment shown in Figure 3 above achieves multi-dimensional technological breakthroughs based on the architecture of 3D-AOI inspection equipment. The coaxial design of the optical engine and coaxial telecentric lens ensures precise structured light projection, and the use of multiple industrial cameras in a surround configuration enables simultaneous multi-view acquisition of the target, significantly improving the integrity and consistency of image acquisition during the calibration process. The timing control mechanism based on the device hardware enables the coordinated operation of the optical engine and camera, significantly shortening the time for target placement and data acquisition, thereby improving calibration efficiency. Leveraging the optical properties of the device's beam splitter and the camera's high-precision imaging capabilities, combined with the direct calibration of the pixel-to-light correspondence using the light model, the effects of lens distortion are effectively eliminated, providing a high-precision parameter foundation for subsequent 3D reconstruction. This enhances the device's adaptability in complex industrial scenarios and achieves collaborative optimization from hardware to algorithm.

[0136] For 3D reconstruction based on phase-3D mapping technology, a two-step calibration method, ray reprojection calibration and sampling mapping calibration, is proposed to determine the phase-3D mapping coefficients for reconstructing the 3D shape of an object from the modulated phase. Ray reprojection calibration is used to determine the system parameters of the back-projection stereo vision model, including camera intrinsics, lens distortion, and rotational and translational parameters. Because phase-3D mapping utilizes a back-projection camera model, unlike the traditional forward projection camera model, which calibrates by minimizing the image reprojection error (i.e., the image distance between the projected image point and the measured image point), ray reprojection is employed to back-project image points through the projection center onto spatial rays. Ideally, these rays should pass through the corresponding object points, so calibration is achieved by minimizing the spatial distance from the object point to the back-projected spatial ray. A planar target with n fiducials printed on its surface is placed at m different locations in space. Uniformly illuminated images are collected at each location to locate the fiducial image coordinates. Orthogonal stripe structured light is then projected and the stripe images are collected to calculate the orthogonal absolute phase map. Calibration is achieved by constructing a cost function and optimizing the algorithm. Sampling mapping calibration uses the calibrated stereo vision system parameters to calculate the spatial posture of the target. The intersection of the light corresponding to each pixel and the target plane is used to obtain the three-dimensional coordinates and corresponding absolute phase values ​​of sufficient spatial points and perform coefficient fitting to determine the phase-to-three-dimensional mapping coefficients. This process does not require a precision translation stage or gauge block to obtain calibration data.

[0137] In an exemplary embodiment, Figure 4 As shown, Figure 4 FIG. 1 is a flow chart of a 3D reconstruction method based on phase-3D mapping technology in one embodiment. The 3D reconstruction method based on phase-3D mapping technology may include the following steps:

[0138] 401. Structured light projection and multi-view image acquisition.

[0139] Among them, under the triggering of the FPGA timing control circuit, the optical machine projects 460nm blue light structured light with phase-shifted stripes, which is highly reflected by the coaxial telecentric lens beam splitter and projected vertically onto the surface of the object being measured. Four high-speed industrial cameras are arranged in a 90° surround layout to synchronously capture the structured light distortion image generated by the three-dimensional contour of the object, covering a 360° spatial perspective.

[0140] 402. Absolute phase data calculation.

[0141] The multi-angle structured light image is decoded using Gray code or phase shift method for phase solution. Combined with the phase-3D mapping coefficient calibrated in step 309, the absolute phase value of each pixel is calculated to characterize the structured light phase change caused by the height modulation of the object surface.

[0142] 403. Phase data extraction.

[0143] Among them, the phase-shifted stripe structured light collected by four high-speed industrial cameras is subjected to phase solution and phase unwrapping algorithms to obtain absolute phase data 404 and three-dimensional coordinate solution respectively.

[0144] The phase-depth mapping coefficients fitted in step 309 are used to calculate the 3D coordinates based on the phase data. For each pixel, the phase-depth mapping coefficients are directly retrieved through a lookup table. Substituting the phase value into the corresponding 3D value yields the corresponding 3D value, completing the phase-to-3D point cloud solution.

[0145] 405. Point cloud model generation and optimization.

[0146] Among them, the four-view three-dimensional point cloud data is fused into a high-resolution whole, and finally outputs a high-precision three-dimensional point cloud containing the geometric features of the object surface.

[0147] In some embodiments of the present application, the inspection process using the aforementioned 3D-AOI inspection equipment may also include extracting high-definition two-dimensional image data. After the optical engine stops projecting structured light, a high-resolution industrial camera captures a vertical two-dimensional color image under the illumination of a ring-shaped LED fill light. After pre-processing such as denoising and enhancement, a high-definition two-dimensional image is obtained. The inspection process of the object is then completed based on the images of the object projected with structured light captured by multiple second industrial cameras and the high-definition two-dimensional image of the object under ambient light.

[0148] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0149] Based on the same inventive concept, embodiments of the present application also provide a 3D reconstruction device for implementing the aforementioned 3D reconstruction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more 3D reconstruction device embodiments provided below can be found in the above-described limitations of the 3D reconstruction method and will not be further elaborated here.

[0150] In an exemplary embodiment, Figure 5 As shown, a structural block diagram of a three-dimensional reconstruction device is provided, which includes the 3D-AOI inspection device in the above embodiment, and further includes:

[0151] The optical machine control module 501 is used to control the optical machine to project structured light onto the object to be measured. The structured light is incident from the side field of view of the coaxial telecentric lens, reflected by the internal reflective mirror and then projected vertically onto the object to be measured.

[0152] The camera control module 502 controls the plurality of second industrial cameras to synchronously capture multi-angle structured light images of the surface of the object being measured;

[0153] A calculation module 503 is configured to calculate absolute phase data in images captured by multiple second industrial cameras based on the multi-angle structured light images;

[0154] The calculation module 503 is used to generate a three-dimensional reconstruction model of the object under test based on the absolute phase data and the mapping relationship between the phase and the three-dimensional points.

[0155] It should be noted that the above-mentioned three-dimensional reconstruction device may further include more or fewer modules than the above-mentioned modules, and the three-dimensional reconstruction device may implement each process in the above-mentioned method embodiment based on these modules.

[0156] Each module in the above-mentioned 3D reconstruction apparatus may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0157] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements each process in the above method embodiment when executing the computer program.

[0158] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, each process in the above method embodiment is implemented.

[0159] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the various processes in the above method embodiments.

[0160] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0161] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A 3D-AOI inspection device, characterized in that: include: Optical machine, first industrial camera, multiple second industrial cameras, and coaxial telecentric lens; The optical engine is connected to the coaxial telecentric lens and is used to emit structured light. The structured light is incident from the side field of view of the coaxial telecentric lens, reflected by the internal reflective mirror oblique surface, and then projected vertically to the object to be measured; The imaging end of the first industrial camera is connected to the top surface field of view of the coaxial telecentric lens, and the optical path of the first industrial camera passes through the center of the coaxial telecentric lens and extends to the object to be measured; The plurality of second industrial cameras are arranged around the first industrial camera, and the lens of each second industrial camera is aimed at the object to be measured at a set inclination angle.

2. The 3D-AOI inspection equipment according to claim 1, characterized in that: The first industrial camera is a high-resolution industrial camera; The plurality of second industrial cameras are four high-speed industrial cameras, and the four high-speed industrial cameras are symmetrically arranged around the high-resolution industrial camera.

3. The 3D-AOI inspection equipment according to claim 1, characterized in that: The device further comprises: An annular LED fill light is arranged around the end of the coaxial telecentric lens and is used to provide lighting for imaging of the first industrial camera.

4. The 3D-AOI inspection equipment according to claim 1, characterized in that: The coaxial telecentric lens has a built-in beam splitter; The reflectivity of the spectroscope for the target light band is greater than the preset reflectivity, and the transmittance of the spectroscope for other light bands is greater than the preset transmittance; The other optical bands are bands other than the target optical band.

5. The 3D-AOI inspection equipment according to claim 4, characterized in that: The target visible light band is a blue light band with a central wavelength of 460±5nm.

6. The 3D-AOI inspection equipment according to claim 4, characterized in that: The sum of the preset reflectivity and the preset transmittance is greater than or equal to 100%.

7. The 3D-AOI inspection equipment according to claim 4, characterized in that: The spectroscope comprises alternately stacked high refractive index layers and low refractive index layers, wherein the refractive index of the high refractive index layers for the target light band is greater than a first preset refractive index, and the refractive index of the low refractive index layers for the target light band is less than a second preset refractive index.

8. The 3D-AOI inspection equipment according to claim 7, characterized in that: The spectroscope is deposited and formed by electron beam evaporation. The spectroscope is deposited and formed by ion beam assisted deposition during the deposition process. After deposition, the spectroscope is annealed.

9. The 3D-AOI inspection equipment according to claim 2, characterized in that: The four high-speed industrial cameras are arranged around at equal angles of 90°, the angles between the optical axes of the four high-speed industrial cameras and the optical axis of the high-resolution industrial camera are within a preset angle range, and the distances between the center points of the lenses of the four high-speed industrial cameras and the surface of the object to be measured are equal.

10. A three-dimensional reconstruction method, characterized in that: The method is implemented based on the 3D-AOI inspection device according to any one of claims 1 to 9, and the method includes: Controlling the optical machine to project structured light onto the object to be measured, wherein the structured light is incident from the side field of view of the coaxial telecentric lens, is reflected by the internal reflective mirror oblique surface, and is projected vertically onto the object to be measured; Controlling a plurality of second industrial cameras to synchronously capture multi-angle structured light images of the surface of the object to be measured; Based on the multi-angle structured light image, calculating absolute phase data in the images captured by the plurality of second industrial cameras; Based on the absolute phase data and the mapping relationship between the phase and the three-dimensional points, a three-dimensional reconstruction model of the object under test is generated.

Images