Sine fringe combination method and device, height map generation method and device and 3D AOI equipment

Abstract

The invention discloses a sine fringe combination, a height map generation method, a height map generation device and 3D AOI equipment, the sine fringe combination provided by the invention comprises a plurality of high-frequency sine fringes, and through the combination of the plurality of high-frequency sine fringes, more dense and detailed spatial information can be provided. And enough information points are ensured to capture tiny changes of the surface of the object to be measured, so that the measurement precision and resolution are improved. Besides, the redundancy of information is provided by using a plurality of high-frequency sine fringes, the robustness and the reliability of the system are enhanced, and errors and ambiguity can be reduced in the phase unwrapping process, especially in the process of processing objects with complex morphologies. And secondly, the number of projection pixels covered in each high-frequency sine fringe period is set to be at least eight, so that the projection quality of the sine fringes can be effectively ensured. The sine fringe combination provided by the embodiment of the invention provides a simple and efficient mode for determining the projection pattern of the 3D AOI equipment.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of optical detection, and in particular to a sinusoidal fringe combination, a height map generation method, a device and a 3D AOI device. Background Art

[0002] In the field of optical inspection, traditional 3D AOI (Automated Optical Inspection) equipment generally includes a central camera and four side projections. Each side projection and a central camera form a subsystem. The height map generated by each subsystem and the texture image used are acquired in the central camera pixel coordinate system. During the use of 3D AOI, regarding the frequency of the pattern projected by the side projection, the higher the frequency, the denser the stripes, the more sensitive to small device deformation, and the higher the imaging accuracy. However, the frequency cannot be too high. If the frequency is too high, the black and white contrast of the stripes will deteriorate. The selection of frequency has a great impact on the detection accuracy and efficiency of the 3D AOI equipment, and there is currently no simple and efficient way to determine the projection pattern of the 3D AOI equipment.

[0003] In summary, in the prior art, when using 3D AOI equipment, there is a technical problem that there is no simple and efficient way to determine the projection pattern of the 3D AOI equipment. Summary of the invention

[0004] The embodiments of the present invention provide a sinusoidal fringe combination, a height map generation method, a device and a 3D AOI device, which solve the technical problem in the prior art that the accuracy of the height map generated by the 3D AOI device during the three-dimensional reconstruction process is poor.

[0005] In a first aspect, an embodiment of the present invention provides a sinusoidal fringe combination, including:

[0006] A plurality of high-frequency sinusoidal stripes, and a plurality of sinusoidal stripes of the same frequency form a sinusoidal stripe pattern according to a preset period. The number of projection pixels covered by the high-frequency sinusoidal stripes in one period is at least 8. The horizontal sinusoidal stripe frequency and the vertical sinusoidal stripe frequency are determined according to the projection resolution of the projector. The horizontal or vertical sinusoidal stripe pattern at a single frequency is projected in sequence by the projector until the horizontal sinusoidal stripe pattern and the vertical sinusoidal stripe pattern of all frequencies are projected.

[0007] As described above, the embodiment of the present invention provides a sinusoidal fringe combination. The sinusoidal fringe combination provided by the embodiment of the present invention includes a plurality of high-frequency sinusoidal fringes. Through the combination of a plurality of high-frequency sinusoidal fringes, more dense and detailed spatial information can be provided. The use of a plurality of high-frequency sinusoidal fringes can ensure that there are enough information points to capture the slight changes on the surface of the object to be measured, thereby improving the accuracy and resolution of the measurement. In addition, the use of a plurality of high-frequency sinusoidal fringes provides information redundancy. In practical applications, due to various factors (such as lighting conditions, reflective characteristics of the surface of the object, noise, etc.), some fringe information may be lost or distorted. By using at least a plurality of high-frequency sinusoidal fringes, even if some fringes are affected, other fringes can still provide effective information, thereby enhancing the robustness and reliability of the system. In addition, the use of a plurality of high-frequency sinusoidal fringes also helps to reduce errors and ambiguities in the phase unwrapping process, especially in processing objects with complex morphologies. Secondly, the embodiment of the present invention can effectively ensure the projection quality of the sinusoidal fringes by setting the number of projection pixels covered in each high-frequency sinusoidal fringe period to at least 8. The sinusoidal fringe combination provided in the embodiment of the present invention provides a simple and efficient way to determine the projection pattern of the 3D AOI device, solving the technical problem that there is no simple and efficient way to determine the projection pattern of the 3D AOI device in the prior art when using the 3D AOI device.

[0008] Preferably, it also includes:

[0009] At least two low-frequency sinusoidal stripes, multiple sinusoidal stripes of the same frequency form a sinusoidal stripe pattern according to a preset period, the number of projection pixels covered by the low-frequency sinusoidal stripes in one period ranges from 40 to 50, and the horizontal sinusoidal stripe frequency and the vertical sinusoidal stripe frequency are determined according to the projection resolution.

[0010] As mentioned above, the sinusoidal fringe combination provided by the embodiment of the present invention also includes low-frequency sinusoidal fringes. The period of low-frequency sinusoidal fringes is relatively long, which can cover a larger spatial range, effectively capture the large-scale morphological changes of the object, and expand the measurement range. In addition, compared with high-frequency sinusoidal fringes, low-frequency sinusoidal fringes are not sensitive to the detailed changes on the surface of the object, can better reflect the overall shape and structure of the object, and better provide global information. At the same time, due to their longer period and more stable characteristics, low-frequency sinusoidal fringes are less sensitive to interference, which can enhance the robustness of the entire measurement system and reduce the influence of errors and noise.

[0011] Preferably, the number of steps of the sinusoidal fringe combination is at least three.

[0012] As mentioned above, the number of steps of the sinusoidal fringe combination in the embodiment of the present invention is at least three steps. The sinusoidal fringe combination of at least three steps can provide more phase information points, which helps to more accurately determine the phase jump point, thereby reducing errors and ambiguities in the phase unwrapping process. In addition, through the sinusoidal fringe combination of at least three steps, richer spatial frequency information can also be obtained, thereby improving the accuracy and resolution of the measurement, and at the same time, more data redundancy can be provided, thereby suppressing errors to a certain extent and improving the robustness and stability of the measurement. Secondly, the sinusoidal fringe combination of at least three steps can better adapt to the changes on the surface of the object, thereby better capturing the details and features of the surface of the object and improving the integrity and accuracy of the measurement.

[0013] In a second aspect, an embodiment of the present invention provides a height map generation method, which is applicable to a 3D AOI device, wherein the 3D AOI device includes a central camera and a central projection that share a lens, and includes a plurality of side cameras that are respectively arranged around the central camera, and the method includes:

[0014] Projecting the sinusoidal fringe patterns of the first aspect onto the surface of the object to be measured sequentially through central projection, wherein the sinusoidal fringe patterns include a horizontal sinusoidal fringe pattern and a vertical sinusoidal fringe pattern;

[0015] Each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured, a target image of the object to be measured is collected by each side camera;

[0016] A height map of the object to be measured in the pixel coordinate system of the central camera is determined according to the target images captured by at least two side cameras and a preset coordinate transformation matrix.

[0017] As mentioned above, the embodiment of the present invention can project the sinusoidal fringe pattern onto the surface of the object to be measured through central projection, and at the same time, open at least two side cameras to collect the target image of the surface of the object to be measured, and then generate a height map of the object to be measured in the central camera pixel coordinate system based on the target image and the preset coordinate transformation matrix. In the process of generating the height map, the embodiment of the present invention can use the preset coordinate transformation matrix, and there is no need to introduce the rotation matrix and translation vector between the central projection and the central camera, thereby avoiding the influence of the calibration error of the rotation matrix and the translation vector, reducing the loss of imaging accuracy, and improving the accuracy of the generated height map. The technical problem of poor accuracy of the generated height map in the process of three-dimensional reconstruction of 3D AOI equipment in the prior art is solved.

[0018] Preferably, determining a height map of the object to be measured in the pixel coordinate system of the central camera according to the target images captured by at least two side cameras and a preset coordinate transformation matrix includes:

[0019] According to the target image captured by each side camera and the preset coordinate conversion matrix, determine the second coordinate and height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera, and one side camera pixel coordinate system corresponds to one height value;

[0020] Determine a target height value corresponding to the second coordinate based on a plurality of height values ​​corresponding to a plurality of side cameras;

[0021] A height map is generated according to the target height values ​​corresponding to the plurality of second coordinates.

[0022] As described above, the embodiment of the present invention can project the sinusoidal fringe pattern onto the surface of the object to be measured through central projection, and at the same time turn on the side camera to collect the target image of the surface of the object to be measured, and then determine the second coordinate corresponding to the first coordinate in each side camera pixel coordinate system and the target height value in the central camera pixel coordinate system based on the target image and the preset coordinate conversion matrix, and generate a height map according to the target height value. In the process of generating the height map, when converting the data in the side camera pixel coordinate system into the data in the central camera pixel coordinate system, the embodiment of the present invention does not need to introduce the rotation matrix and translation vector between the central projection and the central camera, thereby avoiding the influence of the calibration error of the rotation matrix and the translation vector, reducing the loss of imaging accuracy, and improving the accuracy of the generated height map. This solves the technical problem of poor accuracy of the generated height map in the process of three-dimensional reconstruction of 3D AOI equipment in the prior art.

[0023] Preferably, according to the target image captured by each side camera and a preset coordinate conversion matrix, determining the second coordinate and height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera includes:

[0024] Determine the unfolding phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle according to the target image;

[0025] According to the first coordinates of the pixel points in each target image in the corresponding side camera pixel coordinate system, the preset coordinate transformation matrix and the unfolding phase, the second coordinates and the height value corresponding to the first coordinate in each side camera pixel coordinate system in the central camera pixel coordinate system are determined.

[0026] After the embodiment of the present invention determines the unfolding phase of the sinusoidal stripes based on the target image, the corresponding second coordinate and the target height value are determined in the center camera pixel coordinate system according to the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the preset coordinate conversion matrix and the unfolding phase, and a height map is generated according to the target height value. Therefore, in the process of generating the height map, when converting the data in the side camera pixel coordinate system into the data in the center camera pixel coordinate system, it is not necessary to introduce the rotation matrix and translation vector between the center projection and the center camera, thereby avoiding the influence of the calibration error of the rotation matrix and the translation vector, reducing the loss of imaging accuracy, and improving the accuracy of the generated height map.

[0027] Preferably, determining the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one period according to the target image comprises:

[0028] Determining a wrapping phase corresponding to each frequency based on a target image corresponding to the sinusoidal fringes of each frequency of the sinusoidal fringe pattern;

[0029] The wrapped phase corresponding to the target frequency is unfolded to obtain the unfolded phase of the sinusoidal fringe within one period, and the target frequency is the frequency with the largest value.

[0030] The embodiment of the present invention calculates the wrapped phase of each frequency of the sinusoidal stripes, thereby limiting the phase value within a specific range, eliminating the phase discontinuity, and making the phase value easier to process and analyze, so that the unfolded phase of the sinusoidal stripes within one period can be accurately and effectively determined subsequently.

[0031] Preferably, according to the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the preset coordinate conversion matrix and the unfolding phase, the second coordinate and the height value corresponding to the first coordinate in each side camera pixel coordinate system in the center camera pixel coordinate system are determined, including:

[0032] Determine, according to the unwrapped phase and the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the projection coordinate corresponding to the first coordinate in the central projection pixel coordinate system;

[0033] According to the projection coordinates and the preset coordinate conversion matrix, the second coordinates and the height value of the projection coordinates corresponding to each side camera pixel coordinate system in the central camera pixel coordinate system are determined.

[0034] Preferably, determining the projection coordinates corresponding to the first coordinates in the central projection pixel coordinate system according to the unwrapped phase and the first coordinates of the pixel points in each target image in the corresponding side camera pixel coordinate system includes:

[0035] Determine a horizontal unfolded phase corresponding to the horizontal sinusoidal stripes of the sinusoidal stripe pattern and a vertical unfolded phase corresponding to the vertical sinusoidal stripes of the sinusoidal stripe pattern in the unfolded phase;

[0036] Determine the projection resolution of the central projection;

[0037] According to the lateral unwrapping phase, the vertical unwrapping phase, the projection resolution and the first coordinate of each side camera pixel coordinate system, a projection coordinate corresponding to the first coordinate of each side camera pixel coordinate system in the central projection pixel coordinate system is determined.

[0038] The embodiment of the present invention can convert the first coordinate of the side camera pixel coordinate system into the central projection pixel coordinate system according to the lateral unfolding phase, vertical unfolding phase, projection resolution and the first coordinate of each side camera pixel coordinate system of the sinusoidal stripes, so as to facilitate the subsequent further conversion of the coordinates of the central projection pixel coordinate system into the central camera pixel coordinate system.

[0039] Preferably, the preset coordinate conversion matrix includes a plurality of first conversion matrices for converting coordinates of different side camera pixel coordinate systems into coordinates in a world coordinate system, a second conversion matrix for converting coordinates in a central projection pixel coordinate system into coordinates in a world coordinate system, and a third conversion matrix for converting coordinates in a central projection pixel coordinate system into coordinates in a central camera pixel coordinate system;

[0040] According to the projection coordinates and the preset coordinate conversion matrix, the second coordinates and the height value of the projection coordinates corresponding to each side camera pixel coordinate system in the center camera pixel coordinate system are determined, including:

[0041] According to the third transformation matrix, the projection coordinates in the central projection pixel coordinate system are transformed into the second coordinates in the central camera pixel coordinate system;

[0042] According to the second coordinate, the projection coordinate, the first transformation matrix corresponding to the plurality of side camera pixel coordinate systems, and the second transformation matrix, a plurality of height values ​​corresponding to the second coordinate are determined, wherein one side camera pixel coordinate system corresponds to one height value.

[0043] In the embodiment of the present invention, since the central projection and the central camera use the same lens, the first transformation matrix introduced makes the central projection and the central camera coaxial constraint, and the error will be much smaller than the rotation matrix and translation vector of the central projection and the central camera in two systems, which improves the accuracy of solving the height value. In addition, the height value corresponding to each side camera pixel coordinate system is also solved in this embodiment, so that the height value corresponding to each side camera pixel coordinate system can be merged later, further improving the accuracy of calculating the height value.

[0044] Preferably, after determining the second coordinates and the height value of the projection coordinates corresponding to each side camera pixel coordinate system in the central camera pixel coordinate system, the method further includes:

[0045] Determine a zoom factor, where the zoom factor is the ratio of the pixel size of the center camera pixel coordinate system to the pixel size of the side camera pixel coordinate system;

[0046] Scale the first coordinate according to the zoom factor.

[0047] As described above, after solving the second coordinate, the embodiment of the present invention further scales the first coordinate according to the ratio of the pixel size of the central camera pixel coordinate system to the pixel size of the side camera pixel coordinate system, thereby aligning the pixel size of the side camera pixel coordinate system with the pixel size of the central camera pixel coordinate system, so as to subsequently align the height map with the texture map of the central camera.

[0048] Preferably, determining a target height value corresponding to the second coordinate based on a plurality of height values ​​corresponding to a plurality of side cameras includes:

[0049] determining a valid height value from among a plurality of height values ​​corresponding to the second coordinate;

[0050] A target height value corresponding to the second coordinate is determined according to the effective height value corresponding to the second coordinate.

[0051] After solving the height value, the embodiment of the present invention further screens out valid height values ​​from the height values, thereby eliminating invalid data, and finally determines the target height value corresponding to the second coordinate based on the valid height value, thereby improving the accuracy of the subsequently generated target height value.

[0052] Preferably, determining the target height value corresponding to the second coordinate according to the effective height value corresponding to the second coordinate includes:

[0053] Determine a maximum height value with the largest value and a minimum height value with the smallest value among the valid height values ​​corresponding to each second coordinate;

[0054] When the difference between the maximum height value and the minimum height value is within a preset range, the effective height values ​​corresponding to the second coordinates are merged to obtain a target height value corresponding to the second coordinates;

[0055] When the difference between the maximum height value and the minimum height value is not within the preset range, the target height value corresponding to the second coordinate is set to an invalid height value.

[0056] After screening out the valid height values, the embodiment of the present invention further determines whether there is erroneous data in the valid height values ​​corresponding to each second coordinate. If there is no erroneous data, the valid height values ​​corresponding to the second coordinates are fused to determine the final target height value. If there is erroneous data, the target height value corresponding to the second coordinate is set to an invalid height value, and the wireless height value is filled in during the subsequent filtering process, thereby further improving the accuracy of the target height value.

[0057] In a third aspect, an embodiment of the present invention provides a height map generating device, which is applicable to a 3D AOI device, wherein the 3D AOI device includes a central camera and a central projection that share a lens, and includes a plurality of side cameras that are respectively arranged around the central camera, and the device includes:

[0058] A fringe projection module, used for sequentially projecting the sinusoidal fringe patterns of the first aspect onto the surface of the object to be measured through central projection, wherein the sinusoidal fringe patterns include a horizontal sinusoidal fringe pattern and a vertical sinusoidal fringe pattern;

[0059] An image acquisition module, used for acquiring a target image of the object to be measured through each side camera each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured;

[0060] The height map generation module is used to determine the height map of the object to be measured in the central camera pixel coordinate system according to the target images captured by at least two side cameras and a preset coordinate transformation matrix.

[0061] In a fourth aspect, an embodiment of the present invention provides a 3D AOI device, the 3D AOI device comprising a processor, a memory, a central camera and a central projection sharing a lens, and a plurality of side cameras respectively arranged around the central camera;

[0062] The memory is used to store the computer program and transmit the computer program to the processor;

[0063] The processor is used to execute a height map generating method as described in the second aspect according to instructions in the computer program.

[0064] In a fifth aspect, an embodiment of the present invention provides a storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute a height map generation method as in the second aspect.

[0065] As mentioned above, the embodiments of the present invention provide a sinusoidal fringe combination, a height map generation method, an apparatus and a 3D AOI device. The sinusoidal fringe combination provided by the present invention includes a plurality of high-frequency sinusoidal fringes. Through the combination of a plurality of high-frequency sinusoidal fringes, more dense and detailed spatial information can be provided. The use of a plurality of high-frequency sinusoidal fringes can ensure that there are enough information points to capture the slight changes on the surface of the object to be measured, thereby improving the accuracy and resolution of the measurement. In addition, the use of a plurality of high-frequency sinusoidal fringes provides information redundancy, enhances the robustness and reliability of the system, and also helps to reduce errors and ambiguities in the phase unwrapping process, especially when processing objects with complex morphologies. Secondly, the present invention can effectively ensure the projection quality of the sinusoidal fringes by setting the number of projection pixels covered in each high-frequency sinusoidal fringe period to at least 8. The sinusoidal fringe combination provided by the embodiments of the present invention provides a simple and efficient way to determine the projection pattern of a 3D AOI device. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic flow chart of a method for generating a height map provided in an embodiment of the present invention.

[0067] Figure 2 A schematic diagram of the viewing angle coverage relationship among the side cameras, center camera and center projection provided in an embodiment of the present invention.

[0068] Figure 3 A schematic diagram of the principle of aligning a height map and a texture map provided by an embodiment of the present invention.

[0069] Figure 4 A schematic flow chart of another method for generating a height map provided in an embodiment of the present invention.

[0070] Figure 5 Another schematic diagram of the principle of aligning a height map and a texture map provided by an embodiment of the present invention.

[0071] Figure 6 A schematic structural diagram of a height map generating device provided by an embodiment of the present invention.

[0072] Figure 7 A schematic structural diagram of a 3D AOI device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following description and accompanying drawings fully illustrate the specific embodiments of the present application so that those skilled in the art can practice them. The examples represent possible variations only. Unless explicitly required, separate components and functions are optional, and the order of operation can vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. The scope of the embodiments of the present application includes the entire scope of the claims, and all available equivalents of the claims. In this article, each embodiment may be represented individually or generally by the term "invention", which is only for convenience, and if more than one invention is disclosed in fact, it is not intended to automatically limit the scope of the application to any single invention or inventive concept. In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. The various embodiments are described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. As for the structures, products, etc. disclosed in the embodiments, since they correspond to the parts disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0074] At present, in the process of using 3D AOI, the frequency of the pattern projected by the side projection mainly depends on the specific application scenario, the detection requirements and the technical specifications of the equipment. In general, the pattern frequency may be adjusted according to the characteristics of the inspection object, the defect type and the required detection accuracy. For example, for applications that require high-precision detection, a higher frequency pattern may be required to obtain more detailed image information. Conversely, for general inspection tasks, a lower pattern frequency is sufficient. In addition, different 3D AOI devices may have different pattern projection and capture methods, which will also affect the choice of pattern frequency. Therefore, there is currently no simple and efficient way to determine the projection pattern of the 3D AOI device when using 3D AOI for side projection.

[0075] In order to solve the technical problem, the embodiment of the present invention provides a sinusoidal fringe combination, which can provide more dense and detailed spatial information through the combination of multiple high-frequency sinusoidal fringes, improve the measurement accuracy and resolution, and enhance the robustness and reliability of 3D AOI equipment. It also helps to reduce errors and ambiguities in the phase unwrapping process, especially in the processing of objects with complex morphology. The embodiment of the present invention provides a sinusoidal fringe combination including:

[0076] A plurality of high-frequency sinusoidal stripes, and a plurality of sinusoidal stripes of the same frequency form a sinusoidal stripe pattern according to a preset period. The number of projection pixels covered by the high-frequency sinusoidal stripes in one period is at least 8. The horizontal sinusoidal stripe frequency and the vertical sinusoidal stripe frequency are determined according to the projection resolution of the projector. The horizontal or vertical sinusoidal stripe pattern at a single frequency is projected in sequence by the projector until the horizontal sinusoidal stripe pattern and the vertical sinusoidal stripe pattern of all frequencies are projected.

[0077] In this embodiment, the sinusoidal fringe combination includes a plurality of high-frequency sinusoidal fringes, wherein the high-frequency sinusoidal fringes are fringes with a higher frequency, and the frequency of the fringes refers to the number of times the fringes appear in a unit time. In this embodiment, the sinusoidal fringe diagram is composed of a plurality of sinusoidal fringes of the same frequency according to a preset period, that is, for a plurality of sinusoidal fringes of the same frequency, after the plurality of sinusoidal fringes are projected according to a preset period, a sinusoidal fringe diagram can be obtained. The preset period is a pre-set period, and the interval of each period can be set according to actual needs, which is not specifically limited in this embodiment. In this embodiment, by periodically projecting sinusoidal fringes of the same frequency, it can be ensured that each sinusoidal fringe has the same spatial period and phase change during the entire measurement process, thereby improving the accuracy and consistency of the measurement. Secondly, since the frequency of all sinusoidal fringes is the same, the data processing algorithm can be more unified and simplified, and the sinusoidal fringe diagram of the same frequency is easier to analyze and interpret through the algorithm after acquisition, reducing the complexity and amount of calculation of data processing. In addition, the sinusoidal fringes of the same frequency projected using the periodicity can form specific patterns or features, which can be used to distinguish different objects or surfaces, thereby achieving accurate object recognition and positioning.

[0078] In addition, in this embodiment, the number of projection pixels covered by each high-frequency sinusoidal stripe in each cycle is at least 8, where the number of projection pixels refers to the number of pixels used to display the projected stripes, that is, when the 3D AOI device in this embodiment performs projection, at least 8 pixels are used to express the details and shape of the stripes in each cycle of projecting a high-frequency sinusoidal stripe. The reason why the number of projection pixels is at least 8 is that the more the number of projection pixels, the more accurately the details and shapes of the high-frequency sinusoidal stripes can be presented, thereby improving the clarity and quality of the overall picture; on the contrary, if the number of projection pixels is insufficient, the high-frequency sinusoidal stripes may appear blurred or distorted, affecting the viewing experience. Therefore, in this embodiment, the number of projection pixels covered in a high-frequency sinusoidal stripe cycle is determined to be at least 8 to ensure the projection quality of the sinusoidal stripes.

[0079] In addition, in this embodiment, when the projector projects high-frequency sinusoidal stripes, it is necessary to project a horizontal sinusoidal stripe pattern and a vertical sinusoidal stripe pattern. The high-frequency sinusoidal stripes include horizontal sinusoidal stripes and vertical sinusoidal stripes. When the sinusoidal stripe pattern is projected, the horizontal sinusoidal stripe pattern and the vertical sinusoidal stripe pattern refer to the images of sinusoidal stripes in different arrangement directions on the projection screen. The horizontal sinusoidal stripe pattern refers to the arrangement of sinusoidal stripes in the horizontal direction on the projection screen, that is, the extension direction of the sinusoidal stripes is parallel to the short side (width) of the projection screen. When the projector projects the horizontal sinusoidal stripe pattern, each sinusoidal stripe spans the width of the projection screen and extends along the length direction. The vertical sinusoidal stripe pattern refers to the arrangement of sinusoidal stripes in the vertical direction on the projection screen, that is, the extension direction of the sinusoidal stripes is parallel to the long side (height) of the projection screen. When the projector projects the vertical sinusoidal stripe pattern, each sinusoidal stripe extends along the height direction of the projection screen and spans its width. The frequency of the horizontal sinusoidal stripes and the frequency of the vertical sinusoidal stripes need to be determined according to the projection resolution of the projector. The width resolution in the projection resolution / the number of projection pixels covered by each horizontal sinusoidal stripe in each cycle = the maximum frequency of the horizontal sinusoidal stripes, and the height resolution in the projection resolution / the number of projection pixels covered by each vertical sinusoidal stripe in each cycle = the maximum frequency of the vertical sinusoidal stripes. For example, assuming that the projection resolution of the projector is 1080*1920, and the number of projection pixels covered in a high-frequency sinusoidal stripe cycle is 8, the frequency of the horizontal sinusoidal stripes is up to 135HZ, the frequency range is 108-135HZ, and the frequency of the vertical sinusoidal stripes is up to 240HZ, the frequency range is 192-240HZ. For example, when the sinusoidal stripes are combined into three-frequency four-step sinusoidal stripes, the frequencies of the horizontal sinusoidal stripes can be 135HZ, 130HZ and 108HZ respectively, and the frequencies of the vertical sinusoidal stripes can be 240HZ, 235HZ and 192HZ respectively. When the sinusoidal stripes are combined into two-frequency three-step sinusoidal stripes, the horizontal sinusoidal stripe frequencies can be 135 Hz and 120 Hz, respectively, and the vertical sinusoidal stripe frequencies can be 240 Hz and 213 Hz, respectively.

[0080] When measuring an object to be measured, the embodiment of the present invention can project a sinusoidal fringe pattern onto the surface of the object to be measured. Specifically, a horizontal or vertical sinusoidal fringe pattern at a single frequency can be projected sequentially by a projector until the horizontal sinusoidal fringe patterns and vertical sinusoidal fringe patterns of all frequencies are projected. By projecting the horizontal sinusoidal fringe pattern and the vertical sinusoidal fringe pattern respectively, the height change information of the object in different directions can be obtained respectively, so as to restore the three-dimensional information of the object more comprehensively and accurately. In addition, sequentially projecting the horizontal sinusoidal fringe pattern and the vertical sinusoidal fringe pattern at a single frequency can also help reduce information interference and confusion. If the sinusoidal fringe patterns of multiple frequencies or directions are projected at the same time, information superposition and confusion may occur, making subsequent data processing and analysis complicated. By sequentially projecting the horizontal or vertical sinusoidal fringe pattern at a single frequency, it can be ensured that only one frequency and direction of fringe information is processed each time, simplifying the data processing process and improving the accuracy and stability of the measurement. It can be understood that the projection order of the horizontal sinusoidal fringe pattern and the vertical sinusoidal fringe pattern can be set according to actual needs, and is not specifically limited in this embodiment.

[0081] As described above, the embodiment of the present invention provides a sinusoidal fringe combination. The sinusoidal fringe combination provided by the embodiment of the present invention includes a plurality of high-frequency sinusoidal fringes. Through the combination of a plurality of high-frequency sinusoidal fringes, more dense and detailed spatial information can be provided. The use of a plurality of high-frequency sinusoidal fringes can ensure that there are enough information points to capture the slight changes on the surface of the object to be measured, thereby improving the accuracy and resolution of the measurement. In addition, the use of a plurality of high-frequency sinusoidal fringes provides information redundancy. In practical applications, due to various factors (such as lighting conditions, reflective characteristics of the surface of the object, noise, etc.), some fringe information may be lost or distorted. By using a plurality of high-frequency sinusoidal fringes, even if some fringes are affected, other fringes can still provide effective information, thereby enhancing the robustness and reliability of the 3D AOI equipment. Secondly, the use of a plurality of high-frequency sinusoidal fringes also helps to reduce errors and ambiguities in the phase unwrapping process, especially in the processing of objects with complex morphologies. In addition, the embodiment of the present invention can effectively ensure the projection quality of the sinusoidal fringes by setting the number of projection pixels covered in each high-frequency sinusoidal fringe period to at least 8. The sinusoidal fringe combination provided in the embodiment of the present invention provides a simple and efficient way to determine the projection pattern of the 3D AOI device, solving the technical problem that there is no simple and efficient way to determine the projection pattern of the 3D AOI device in the prior art when using the 3D AOI device.

[0082] On the basis of the above embodiment, it also includes:

[0083] At least two low-frequency sinusoidal stripes, multiple sinusoidal stripes of the same frequency form a sinusoidal stripe pattern according to a preset period, the number of projection pixels covered by the low-frequency sinusoidal stripes in one period ranges from 40 to 50, and the horizontal sinusoidal stripe frequency and the vertical sinusoidal stripe frequency are determined according to the projection resolution.

[0084] In another embodiment, the sinusoidal stripe combination also includes at least two low-frequency sinusoidal stripes, wherein the low-frequency sinusoidal stripes are stripes with a lower frequency, and the low-frequency sinusoidal stripes also include horizontal sinusoidal stripes and vertical sinusoidal stripes. The number of projection pixels covered in a low-frequency sinusoidal stripe cycle ranges from 40 to 50, and its principle is the same as that of high-frequency sinusoidal stripes, both of which are to ensure the projection quality of the sinusoidal stripes. Similarly, in this embodiment, the frequency of the horizontal sinusoidal stripes and the frequency of the vertical sinusoidal stripes also need to be determined according to the projection resolution. For the horizontal sinusoidal stripes, the range of its low-frequency sinusoidal stripes is generally 24-25HZ; and for the vertical sinusoidal stripes, the range of its low-frequency sinusoidal stripes is generally 44-45HZ. In addition, it can be understood that a plurality of low-frequency sinusoidal stripes of the same frequency can form a sinusoidal stripe diagram by projecting according to a preset period, and its principle is the same as that of the sinusoidal stripe diagram composed of periodically projected high-frequency sinusoidal stripes, which will not be repeated in this embodiment.

[0085] As mentioned above, the sinusoidal fringe combination provided by the embodiment of the present invention also includes low-frequency sinusoidal fringes. The period of low-frequency sinusoidal fringes is relatively long, which can cover a larger spatial range, effectively capture the large-scale morphological changes of the object, and expand the measurement range. In addition, compared with high-frequency sinusoidal fringes, low-frequency sinusoidal fringes are insensitive to the detailed changes on the surface of the object, can better reflect the overall shape and structure of the object, and better provide global information. At the same time, due to their longer period and more stable characteristics, low-frequency sinusoidal fringes are less sensitive to interference, which can enhance the robustness of 3D AOI equipment and reduce the impact of errors and noise.

[0086] Based on the above embodiment, the number of steps of the sinusoidal fringe combination is at least three.

[0087] In one embodiment, the number of steps of the sinusoidal fringe combination is at least three steps. The number of steps of the sinusoidal fringe refers to the number of different phase states used when projecting or capturing the sinusoidal fringe pattern. Each sinusoidal fringe with a specific phase is called a "step". For example, if three sinusoidal fringes with different phases are projected (usually 0 degree, 120 degree and 240 degree phases), the number of steps of the sinusoidal fringes is three steps. Each step provides information about the surface of the object at different phases, which is then used to calculate the phase value, thereby obtaining three-dimensional information of the surface of the object. Increasing the number of steps of the sinusoidal fringes can provide more phase sampling points, which helps to more accurately recover the phase information, especially when processing objects with complex morphology or surface characteristics. However, increasing the number of steps will also increase the complexity and time of data processing. Therefore, in practical applications, it is necessary to select an appropriate number of steps according to specific needs and scenarios. In one embodiment, the number of steps of the sinusoidal stripe combination can be four steps, that is, the sinusoidal stripe combination is five-frequency four-step sinusoidal stripes. Assuming that the projection resolution of the projector is 1080*1920, the frequencies of the five vertical sinusoidal stripes expanded vertically are 240hz, 235hz, 192hz, 45hz and 44hz respectively, and the frequencies of the five horizontal sinusoidal stripes expanded horizontally are 135hz, 130hz, 108hz, 25hz and 24hz respectively.

[0088] As mentioned above, the number of steps of the sinusoidal fringe combination in the embodiment of the present invention is at least three steps. The sinusoidal fringe combination of at least three steps can provide more phase information points, which helps to more accurately determine the phase jump point, thereby reducing errors and ambiguities in the phase unwrapping process. In addition, through the sinusoidal fringe combination of at least three steps, richer spatial frequency information can also be obtained, thereby improving the accuracy and resolution of the measurement, and at the same time, more data redundancy can be provided, which can suppress errors to a certain extent and improve the robustness and stability of the measurement. Secondly, the sinusoidal fringe combination of at least three steps can better adapt to the changes on the surface of the object, better capture the details and features of the surface of the object, and improve the integrity and accuracy of the measurement.

[0089] In addition, in the process of using 3D AOI equipment to perform three-dimensional reconstruction of the object to be measured, after obtaining the height map including the height information of the object to be measured, in order to make the height map correspond to the object to be measured, it is necessary to align the texture map of the object to be measured with the height map, so that the height information corresponding to the texture map can be intuitively obtained. For 3D AOI equipment, in the process of generating the height map, when converting the point cloud data of the subsystem to the central camera pixel coordinate system, it is generally necessary to obtain the homography matrix relationship between the side camera and the central projection, as well as the rotation matrix and translation vector between the central projection and the central camera in the calibration stage, among which the rotation matrix is ​​mainly used to describe the rotation operation of a point or vector in space, and the vector or point can be rotated around a certain axis by a certain angle through the rotation matrix. The translation vector is mainly used to describe the translation operation of a point or vector in space. It is a three-dimensional vector that represents the offset from one point to another. Then, the point cloud data generated by the subsystem can be converted to the central camera pixel coordinate system using the homography matrix, rotation matrix and translation vector. Therefore, the conversion process will inevitably be affected by the calibration errors of the rotation matrix and translation vector, resulting in a certain degree of loss in the imaging accuracy of the height map, reducing the accuracy of the height map.

[0090] Based on this, in order to solve the above technical problems, an embodiment of the present invention provides a height map generation method. In the process of generating the height map, the embodiment of the present invention uses a preset coordinate transformation matrix, and there is no need to introduce a rotation matrix and a translation vector between the central projection and the central camera, thereby avoiding the influence of the calibration error of the rotation matrix and the translation vector, reducing the loss of imaging accuracy, and improving the accuracy of the generated height map. The technical problem of poor accuracy of the generated height map in the process of three-dimensional reconstruction of 3D AOI equipment in the prior art is solved.

[0091] like Figure 1 As shown, Figure 1A flow chart of a height map generation method provided in an embodiment of the present invention. The height map generation method provided in an embodiment of the present invention is applicable to a 3D AOI device, wherein the 3D AOI device includes a central camera and a central projection that share a lens, and includes a plurality of side cameras that are respectively arranged around the central camera. Among them, the central projection is a device for projecting to the center of the area to be measured, and the central projection can be realized by a projector, and the central camera is a camera whose optical axis is perpendicular to the area to be measured. The central projection and the central camera use the same lens, which can be realized in the hardware assembly process. For example, a 45° semi-transparent and semi-reflective lens can be used as a lens, and the central projection and the central camera share a telecentric lens. In addition, the side camera is a camera used to capture images from the side of the central camera. The number of side cameras can be set according to actual needs. In this embodiment, the number of side cameras is 4 as an example for explanation, and the side cameras can use wide-angle lenses. It can be understood that the difference between the side camera and the central camera is that the angles of capturing images are different, and the shooting objects are all the areas to be measured. In one embodiment, the viewing angle coverage relationship between the side camera, the central camera and the central projection is as follows: Figure 2 As shown, Figure 2 In the figure, Z1 is the shooting range of the side camera on the left, Z2 is the projection range of the central projection, and Z3 is the shooting range of the central camera.

[0092] The height map generation method provided by the embodiment of the present invention includes:

[0093] Step 101 : Projecting sinusoidal fringe patterns onto the surface of the object to be measured in sequence through central projection, wherein the sinusoidal fringe patterns include horizontal sinusoidal fringe patterns and vertical sinusoidal fringe patterns.

[0094] First, in this embodiment, it is necessary to project the sinusoidal fringe pattern onto the surface of the object to be measured in sequence through central projection. The sinusoidal fringe pattern is the above-mentioned sinusoidal fringe pattern. For details, please refer to the above description, which will not be repeated in this embodiment. Before projecting the sinusoidal fringe pattern, it is necessary to encode the sinusoidal fringe combination. When encoding the sinusoidal fringe combination, it is necessary to encode sinusoidal fringes of multiple steps at multiple frequencies. The number of steps of the sinusoidal fringe combination determines the accuracy and resolution of the measurement. In this embodiment, the sinusoidal fringe combination can be five frequencies and four steps, four frequencies and three steps, three frequencies and two steps, and higher frequencies and higher steps (at least three frequencies and two steps are required). Exemplarily, the sinusoidal fringe combination of five frequencies and four steps is used as an example for explanation. The five-frequency and four-step sinusoidal fringe combination is a sinusoidal fringe with five frequencies and four steps at each frequency. In this embodiment, the sinusoidal fringes can be encoded by the following formula:

[0095] In(x,y)=A(x,y)+B(x,y)cos[Φ(x,y)+2kπfx / 4]

[0096] Wherein, A is the background intensity, B is the amplitude, Φ is the phase, f is the frequency, k is the number of steps (in this embodiment, 1<=k<=4, k∈Z), and (x, y) are the coordinates in the central projection pixel coordinate system of the central projection.

[0097] In this embodiment, the sinusoidal fringe pattern includes a horizontally expanded horizontal sinusoidal fringe pattern and a vertically expanded vertical sinusoidal fringe pattern, so it is necessary to encode the sinusoidal fringe in two directions during encoding. In one embodiment, assuming that the projection resolution of the central projection is 1920*1080, the frequencies of the five horizontally expanded horizontal sinusoidal fringe are 240 Hz, 235 Hz, 192 Hz, 45 Hz and 44 Hz, respectively, wherein the highest frequency used by the horizontal sinusoidal fringe is 240 Hz, because 1920 / 240=8, that is, one cycle of the horizontal sinusoidal fringe includes 8 pixels (if the frequency is higher, the contrast of the fringe will decrease); 192 Hz means that one cycle includes 10 pixels; 235 is an empirical value; in addition, the frequency difference of 45 Hz and 44 Hz is 1, which is also an empirical value. The frequency difference of 1 can solve a complete expansion phase, and the expansion phase solved by the low frequency is used to guide the high frequency expansion. The frequencies of the five vertical sinusoidal stripes expanded vertically are 135 Hz, 130 Hz, 108 Hz, 25 Hz and 24 Hz respectively. The value selection principle is similar to that of the horizontal sinusoidal stripes. The combination of these frequencies can provide better measurement accuracy and stability for the subsequent measurement. It can be understood that the frequency of the five-frequency four-step sinusoidal stripes combination can also be set according to actual needs, and no specific limitation is made in this embodiment.

[0098] After encoding the five-frequency four-step sinusoidal fringe combination, it is necessary to sequentially project the sinusoidal fringe pattern of the five-frequency four-step sinusoidal fringe combination onto the surface of the object to be measured through central projection, wherein the object to be measured refers to the object to be three-dimensionally reconstructed, for example, the object to be measured may be a circuit board or a component, etc. That is, in this embodiment, it is necessary to sequentially project the sinusoidal fringe pattern of four steps at one frequency onto the surface of the object to be measured until the sinusoidal fringe patterns of all steps at all frequencies are projected. By projecting the sinusoidal fringe pattern of the five-frequency four-step sinusoidal fringe combination onto the object to be measured, the true shape of the object to be measured can be reflected according to the deflection of the sinusoidal fringe pattern on the object to be measured.

[0099] Step 102 : Each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured, a target image of the object to be measured is collected by each side camera.

[0100] After the sinusoidal fringe pattern is projected onto the surface of the object to be measured, four side cameras need to be started for image acquisition. Specifically, the texture image of the object to be measured can be acquired by the central camera, wherein the texture image includes the texture of the surface of the object to be measured. At the same time, after each central projection projects the sinusoidal fringe pattern onto the surface of the object to be measured, the four side cameras need to shoot the surface of the object to be measured, thereby obtaining a target image including the sinusoidal fringe pattern projected onto the surface of the object to be measured. That is, for a side camera, when the sinusoidal fringe combination is a five-frequency four-step sinusoidal fringe combination, for each frequency of the sinusoidal fringe pattern, it is necessary to acquire target images corresponding to four step lengths. Specifically, since the five-frequency four-step sinusoidal fringe combination includes two directions, namely, horizontal sinusoidal fringe and vertical sinusoidal fringe, the side cameras need to acquire corresponding target images for the horizontal sinusoidal fringe and vertical sinusoidal fringe, that is, for the five-frequency four-step sinusoidal fringe combination, each side camera needs to acquire 40 target images.

[0101] Step 103: determine a height map of the object to be measured in the pixel coordinate system of the central camera according to the target images captured by at least two side cameras and a preset coordinate transformation matrix.

[0102] After the side camera has captured all the target images corresponding to the combination of five-frequency four-step sinusoidal stripes, a height map of the object to be measured in the central camera pixel coordinate system can be generated according to the target images captured by at least two side cameras and the preset coordinate conversion matrix. The central camera pixel coordinate system is a pixel coordinate system corresponding to the central camera; the preset coordinate conversion matrix is ​​a preset coordinate conversion matrix, and the coordinate conversion matrix is ​​used to realize the coordinate conversion in different coordinate systems, for example, to realize the conversion of the coordinates in the side camera pixel coordinate system into the world coordinate system, the conversion of the projection coordinates in the central projection pixel coordinate system into the world coordinate system, and the conversion of the projection coordinates in the central projection pixel coordinate system into the central camera pixel coordinate system. In one embodiment, the preset coordinate conversion matrix is ​​used to realize the conversion of the coordinates of the pixel points in the central projection pixel coordinate system into the coordinates in the central camera pixel coordinate system. In addition, the reason for using at least two side cameras to collect the target image of the object to be measured in this embodiment is that by shooting the object to be measured at at least two angles, images of different sides of the object to be measured can be obtained, so as to more comprehensively understand the shape, size and surface defects of the object, so as to improve the accuracy of the height map generated subsequently.

[0103] Exemplarily, the unfolded phase of the sinusoidal fringes of the sinusoidal fringe pattern within one period can be first determined based on the target image, and then the first coordinates of the pixel points in the target image in the side camera pixel coordinate system are converted to coordinates in the central projection pixel coordinate system based on the unfolded phase, and then the coordinates of the pixel points in the central projection pixel coordinate system are converted to the second coordinates in the central camera pixel coordinate system through a preset coordinate conversion matrix. It should be noted that although the height map generation method provided in the prior art and the present embodiment both involve the process of converting coordinates, the coordinate conversion matrix in the present application does not introduce a rotation matrix and a translation vector. Finally, the height value corresponding to each second coordinate in the central camera pixel coordinate system can be further determined based on the coordinate conversion matrix, the second coordinate in the central camera pixel coordinate system, and the coordinate in the central projection pixel coordinate system, and finally the height map of the object to be measured in the central camera pixel coordinate system is generated based on the height value corresponding to the second coordinate. In one embodiment, the specific process of generating the height map is as follows. Figure 3 Compared with the prior art that directly converts the first coordinate of the pixel point in the side camera pixel coordinate system into the second coordinate in the center camera pixel coordinate system, this embodiment utilizes the coaxiality constraint between the center projection pixel coordinate system and the center camera coordinate system, and uses the center projection coordinate system as an intermediary to convert the first coordinate of the pixel point in the target image in the side camera pixel coordinate system into the coordinate in the center projection pixel coordinate system by unfolding the phase, and then converts the center projection pixel coordinate system into the coordinate in the center projection pixel coordinate system, which can effectively reduce the error introduced in the conversion process, so as to improve the accuracy of the height map generated subsequently.

[0104] In another embodiment, the height value corresponding to the first coordinate in the pixel coordinate system of each side camera can be determined based on the target image captured by each side camera, and the height value of the first coordinate in the pixel coordinate system of each side camera can be fused to obtain the target height value corresponding to the first coordinate. After that, the first coordinate of the pixel point in the pixel coordinate system of the side camera can be further converted into the second coordinate in the pixel coordinate system of the center camera according to the preset coordinate conversion matrix, and finally the second coordinate in the pixel coordinate system of the center camera is associated with the target height value to generate a height map of the object to be measured in the pixel coordinate system of the center camera.

[0105] As mentioned above, an embodiment of the present invention provides a method for generating a height map. The embodiment of the present invention projects a sinusoidal fringe pattern onto the surface of an object to be measured by central projection, and simultaneously turns on at least two side cameras to collect a target image of the surface of the object to be measured, and then generates a height map of the object to be measured in the central camera pixel coordinate system based on the target image and a preset coordinate transformation matrix. In the process of generating a height map, the embodiment of the present invention can use a preset coordinate transformation matrix, and there is no need to introduce a rotation matrix and a translation vector between the central projection and the central camera, thereby avoiding the influence of the calibration error of the rotation matrix and the translation vector, reducing the loss of imaging accuracy, and improving the accuracy of the generated height map. This solves the technical problem in the prior art that the accuracy of the generated height map is poor during the three-dimensional reconstruction of 3D AOI equipment.

[0106] The embodiment of the present invention further provides another method for generating a height map. The method for generating a height map provided by the embodiment of the present invention is a specific embodiment of the method for generating a height map. Figure 4 As shown, Figure 4 A schematic flow chart of another method for generating a height map provided by an embodiment of the present invention. The method for generating a height map provided by an embodiment of the present invention includes:

[0107] Step 201 : Projecting sinusoidal fringe patterns onto the surface of the object to be measured in sequence through central projection, wherein the sinusoidal fringe patterns include horizontal sinusoidal fringe patterns and vertical sinusoidal fringe patterns.

[0108] Step 202: Each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured, a target image of the object to be measured is collected by each side camera.

[0109] Step 203: Determine the second coordinate and height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera according to the target image captured by each side camera and the preset coordinate conversion matrix, wherein one side camera pixel coordinate system corresponds to one height value.

[0110] After the side camera has captured all the target images corresponding to the combination of five-frequency four-step sinusoidal stripes, the first coordinate in each target image in the side camera pixel coordinate system can be converted into the second coordinate in the center camera pixel coordinate system according to the target image captured by the side camera and the preset coordinate conversion matrix, and the height value corresponding to each second coordinate can be determined. It should be noted that the pixel coordinate system is a coordinate system used to describe the pixel position in the image. In the pixel coordinate system, the origin is usually set at the upper left corner of the image, that is, the intersection of the first row and the first column. In the pixel coordinate system of the camera, the coordinates of the pixel are usually expressed as (u, v), where u represents the horizontal coordinate of the pixel, and v represents the vertical coordinate of the pixel. The coordinates of the pixel determine the position of each pixel in the image. The pixel coordinate system of the side camera is the pixel coordinate system corresponding to the side camera. The pixel coordinate systems of cameras at different positions are different because the pixel coordinate system of the camera is related to the internal parameters and optical characteristics of the camera, and these parameters will vary depending on the position and angle of the camera.

[0111] In one embodiment, when converting the first coordinate in the side camera pixel coordinate system to the second coordinate in the center camera pixel coordinate system, the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one period can be determined first according to the target image, and then the first coordinate of the pixel point in the side camera pixel coordinate system in the target image is converted to the coordinate in the center projection pixel coordinate system based on the unwrapped phase, and then the coordinate of the pixel point in the center projection pixel coordinate system is converted to the second coordinate in the center camera pixel coordinate system through a preset coordinate conversion matrix. It should be noted that although the height map generation method provided in the prior art and the present embodiment both involve the process of converting coordinates, the coordinate conversion matrix in the present application does not introduce a rotation matrix and a translation vector. Finally, the height value corresponding to each second coordinate in the center camera pixel coordinate system can be further determined according to the coordinate conversion matrix, the second coordinate in the center camera pixel coordinate system, and the coordinate in the center projection pixel coordinate system, wherein the height value is used to characterize the distance between the surface of the object to be measured and the distance between the plane on which the object to be measured is placed. Compared with the prior art in which the first coordinate of the pixel point in the side camera pixel coordinate system is directly converted into the second coordinate in the central camera pixel coordinate system, the present embodiment utilizes the coaxiality constraint between the central projection pixel coordinate system and the central camera coordinate system, and uses the central projection coordinate system as an intermediary. By unfolding the phase, the first coordinate of the pixel point in the target image in the side camera pixel coordinate system is converted into the coordinate in the central projection pixel coordinate system, and then the central projection pixel coordinate system is converted into the coordinate in the central projection pixel coordinate system. This can effectively reduce the error introduced in the conversion process, so as to improve the accuracy of the height map generated subsequently.

[0112] In addition, it should be further explained that, in the present embodiment, each side camera corresponds to a side camera pixel coordinate system, and in the process of converting each side camera pixel coordinate system to the central camera pixel coordinate system, due to different calculation formulas and calculation data, the final calculated height value has a deviation, that is, each second coordinate corresponds to multiple height values, and the number of height values ​​corresponds to the number of side cameras. For example, when there are four side cameras, each second coordinate has a height value corresponding to the four side camera pixel coordinate systems, and each height value may be different.

[0113] On the basis of the above embodiment, in step 203, according to the target image captured by each side camera and the preset coordinate conversion matrix, the second coordinate and the height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera are determined, including:

[0114] Step 2031: Determine the unfolding phase of the sinusoidal fringes of the sinusoidal fringe pattern within one period according to the target image.

[0115] After the side camera captures the target image, the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle can be determined based on the target image captured by the side camera, wherein the unwrapped phase refers to restoring the original phase value from the wrapped phase. In one embodiment, the wrapped phase of each frequency can be determined based on the target image corresponding to each step of each frequency captured by each side camera, and then the wrapped phase of the frequency with the largest value is unwrapped to obtain the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle.

[0116] Based on the above embodiment, determining the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle according to the target image in step 2031 includes:

[0117] Step 20311: Determine the wrapping phase corresponding to each frequency based on the target image corresponding to the sinusoidal fringes of each frequency of the sinusoidal fringe pattern.

[0118] In this embodiment, it is first necessary to determine the wrapping phase corresponding to the sinusoidal stripes of each frequency based on the target image corresponding to the sinusoidal stripes of each frequency captured by the side camera, where the wrapping phase refers to the phase information obtained by the phase shifting technology, and the phase is constrained to be between: "-π, π", and this constraint is the origin of the "wrapping". Specifically, for the target images corresponding to the four steps of the sinusoidal stripes of each frequency captured by each side camera, the wrapping phase of each frequency can be determined using the following formula:

[0119]

[0120] Where (xs ,y s ) represents the coordinates of the pixel points in the target image, I i (x s ,y s ) represents the coordinates (x) in the target image s ,y s ) is the gray value corresponding to the i-th step size, Then the coordinate (x s ,y s ), after solving for the wrapped phase, since the wrapped phase is between [-π,π], it is necessary to further normalize the wrapped phase to between [0, 1].

[0121] Afterwards, by combining the target image captured by each side camera with the wrapped phase corresponding to each frequency of the sinusoidal fringes, the wrapped phase corresponding to each frequency in the sinusoidal fringes can be determined.

[0122] Step 20312: unfold the wrapped phase corresponding to the target frequency to obtain the unfolded phase of the sinusoidal fringe within one period, and the target frequency is the frequency with the largest value.

[0123] After determining the wrapped phase of each frequency of the sinusoidal stripes, it is necessary to further expand the wrapped phase of the target frequency, where the target frequency is the frequency with the largest value among all frequencies. The reason for selecting the frequency with the largest value is that the frequency with the largest value has the highest accuracy. The five frequencies in the five-frequency four-step sinusoidal stripes are decreasing, so the first frequency can be taken as the target frequency. When expanding the wrapped phase of the first frequency, multi-frequency heterodyne can be used to expand the wrapped phase. After expansion, an unambiguous one-cycle phase diagram is obtained. The specific formula is as follows:

[0124]

[0125] Among them, θ1 is the unwrapped phase of the first frequency, floor is the rounding function, is the difference between the wrapping phase of the first frequency and the wrapping phase of the second frequency. If the difference is greater than 0, it is retained, and if it is less than 0, it is increased by 1. λ1 is the value of the first frequency, λ 12 It is the difference between the first frequency and the second frequency. When the difference is less than 0, the absolute value is taken. is the wrapping phase of the first frequency.

[0126] The embodiment of the present invention calculates the wrapped phase of each frequency of the sinusoidal stripes, thereby limiting the phase value within a specific range, eliminating the phase discontinuity, and making the phase value easier to process and analyze, so that the unfolded phase of the sinusoidal stripes within one period can be accurately and effectively determined subsequently.

[0127] Step 2032: Determine the second coordinate and height value corresponding to the first coordinate in each side camera pixel coordinate system in the center camera pixel coordinate system according to the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the preset coordinate transformation matrix and the unfolding phase. One side camera pixel coordinate system corresponds to one height value.

[0128] After the unwrapped phase of the sinusoidal stripes is obtained, the first coordinate of each pixel point in each target image captured by the side camera in the pixel coordinate system of the side camera can be converted into the second coordinate of the pixel coordinate system of the central camera, and the height value corresponding to each second coordinate can be determined according to the first coordinate of each pixel point in each target image captured by the side camera in the pixel coordinate system of the side camera, the preset coordinate conversion matrix and the unwrapped phase. In one embodiment, the first coordinate of the pixel point in the pixel coordinate system of the side camera in the target image can be first converted into the coordinate in the central projection pixel coordinate system based on the unwrapped phase. For example, the unwrapped phase and the first coordinate of the pixel point in the pixel coordinate system of the side camera can be substituted into the preset formula to obtain the coordinate in the central projection pixel coordinate system, and then the coordinate of the pixel point in the central projection pixel coordinate system can be substituted into the coordinate conversion matrix to convert it into the second coordinate in the pixel coordinate system of the central camera.

[0129] On the basis of the above embodiment, in step 2032, according to the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the preset coordinate conversion matrix and the unfolding phase, the second coordinate and the height value corresponding to the first coordinate in each side camera pixel coordinate system in the center camera pixel coordinate system are determined, including:

[0130] Step 20321: Determine the projection coordinates corresponding to the first coordinates in the central projection pixel coordinate system according to the unfolded phase and the first coordinates of the pixel points in each target image in the corresponding side camera pixel coordinate system.

[0131] After determining the unwrapping phase of the target frequency, the first coordinate in each side camera pixel coordinate system can be further transformed into the central projection pixel coordinate system according to the unwrapping phase of the target frequency, wherein the central projection pixel coordinate system is the pixel coordinate system on the plane projected by the central projection. In one embodiment, during the conversion, it is first necessary to determine the first coordinate of each pixel point in each target image in the corresponding side camera pixel coordinate system, and then determine the unwrapping phase of the horizontal sinusoidal stripes and the unwrapping phase of the vertical sinusoidal stripes in the sinusoidal stripes according to the unwrapping phase of the target frequency, and combine the unwrapping phase of the horizontal sinusoidal stripes and the unwrapping phase of the vertical sinusoidal stripes to transform the first coordinate in each side camera pixel coordinate system into the central projection pixel coordinate system, and the obtained coordinate is the projection coordinate.

[0132] Based on the above embodiment, in step 20321, according to the unwrapped phase and the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the projection coordinate corresponding to the first coordinate in the central projection pixel coordinate system is determined, including:

[0133] Step 203211, determine the horizontal unwrapped phase corresponding to the horizontal sinusoidal stripes of the sinusoidal stripe pattern and the vertical unwrapped phase corresponding to the vertical sinusoidal stripes of the sinusoidal stripes.

[0134] First, it is necessary to determine the horizontal unfolding phase corresponding to the horizontal sinusoidal stripes of the sinusoidal stripes and the vertical unfolding phase corresponding to the vertical sinusoidal stripes of the sinusoidal stripes in the unfolding phase of the target frequency, wherein the horizontal sinusoidal stripes refer to the horizontally unfolded stripes in the sinusoidal stripes, and the vertical sinusoidal stripes refer to the vertically unfolded stripes in the sinusoidal stripes. The phase corresponding to the horizontal sinusoidal stripes in the unfolding phase is the horizontal unfolding phase, and the phase corresponding to the vertical sinusoidal stripes is the vertical unfolding phase.

[0135] Step 203212, determine the projection resolution of the central projection.

[0136] After that, it is necessary to further determine the projection resolution of the central projection, which is the resolution of the image projected by the central projection. The projection resolution can be set according to actual needs, for example, the projection resolution can be 1920*1080.

[0137] Step 203213: Determine the projection coordinates corresponding to the first coordinates of each side camera pixel coordinate system in the central projection pixel coordinate system according to the horizontal unfolding phase, the vertical unfolding phase, the projection resolution, and the first coordinates of each side camera pixel coordinate system.

[0138] Finally, the projection coordinates corresponding to the first coordinates of each side camera pixel coordinate system in the central projection pixel coordinate system can be determined according to the horizontal unfolding phase, the vertical unfolding phase, the projection resolution, and the first coordinates of each side camera pixel coordinate system. In one embodiment, the calculation can be performed using the following formula.

[0139]

[0140]

[0141] Among them, u p is the coordinate corresponding to the X axis in the projection pixel coordinate system, θ v is the transverse unwrapped phase corresponding to the transverse sinusoidal fringes in the unwrapped phase, u c ,v c is the first coordinate in the pixel coordinate system of the side camera, λv is the lateral resolution in the projection resolution, v p is the coordinate corresponding to the Y axis in the projection pixel coordinate system, θ h is the vertical unwrapped phase corresponding to the vertical sinusoidal fringes in the unwrapped phase, λ h It is the vertical resolution in the projection resolution.

[0142] The embodiment of the present invention can convert the first coordinate of the side camera pixel coordinate system into the central projection pixel coordinate system according to the lateral unfolding phase, vertical unfolding phase, projection resolution and the first coordinate of each side camera pixel coordinate system of the sinusoidal stripes, so as to facilitate the subsequent further conversion of the coordinates of the central projection pixel coordinate system into the central camera pixel coordinate system.

[0143] Step 20322: Determine the second coordinates and height value of the projection coordinates corresponding to each side camera pixel coordinate system in the center camera pixel coordinate system according to the projection coordinates and the preset coordinate conversion matrix.

[0144] After determining the projection coordinates corresponding to the first coordinates of each side camera pixel coordinate system in the central projection pixel coordinate system, the coordinates corresponding to the projection coordinates corresponding to the first coordinates of each side camera pixel coordinate system and the preset coordinate transformation matrix can be determined, and the height value corresponding to each second coordinate can be determined.

[0145] Based on the above embodiments, the preset coordinate transformation matrix includes multiple first transformation matrices for converting coordinates of different side camera pixel coordinate systems into coordinates in the world coordinate system, a second transformation matrix for converting coordinates in the central projection pixel coordinate system into coordinates in the world coordinate system, and a third transformation matrix for converting coordinates in the central projection pixel coordinate system into coordinates in the central camera pixel coordinate system.

[0146] In one embodiment, the preset coordinate conversion matrix includes multiple conversion matrices, which are multiple first conversion matrices that convert the first coordinates of different side camera pixel coordinate systems into coordinates in the world coordinate system, that is, different side camera pixel coordinate systems correspond to different first conversion matrices, and the number of first conversion matrices is consistent with the number of side cameras. In addition, the coordinate conversion matrix also includes a second conversion matrix for converting the projection coordinates in the center projection pixel coordinate system into the world coordinate system, and a third conversion matrix for converting the projection coordinates in the center projection pixel coordinate system into the coordinates in the center camera pixel coordinate system. In this embodiment, the first conversion matrix, the second conversion matrix and the third conversion can all be obtained in the process of calibrating the 3D AOI equipment.

[0147] In step 20322, the second coordinates and height values ​​of the projection coordinates corresponding to each side camera pixel coordinate system in the center camera pixel coordinate system are determined according to the projection coordinates and the preset coordinate conversion matrix, including:

[0148] Step 203221: According to the third transformation matrix, the projection coordinates in the central projection pixel coordinate system are converted into the second coordinates in the central camera pixel coordinate system.

[0149] When determining the height value, it is first necessary to convert the projection coordinates in the central projection pixel coordinate system into the second coordinates in the central camera pixel coordinate system according to the third conversion matrix. In one embodiment, the projection coordinates in the central projection pixel coordinate system corresponding to the pixel coordinate systems of different side cameras can be converted into the second coordinates in the central camera pixel coordinate system respectively. The specific conversion formula is as follows:

[0150]

[0151] Among them, H cp is the third transformation matrix, u k ,v k The second coordinate in the center camera pixel coordinate system.

[0152] It should be noted that the central projection and the central camera use the same lens. Therefore, by introducing the first transformation matrix obtained by calibration, due to the coaxiality constraint between the central projection and the central camera, the error will be much smaller than the rotation matrix and translation vector of the central projection and the central camera in the two systems, so that the final imaging accuracy is further improved, and the accuracy of subsequent alignment of height maps and texture maps is improved.

[0153] Step 203222: Determine multiple height values ​​corresponding to the second coordinate based on the second coordinate, the projection coordinate, the first transformation matrix corresponding to multiple side camera pixel coordinate systems, and the second transformation matrix, where one side camera pixel coordinate system corresponds to one height value.

[0154] After the projection coordinates in the central projection pixel coordinate system are converted to the second coordinates in the central camera pixel coordinate system, the height value corresponding to each second coordinate can be further calculated. Specifically, in this embodiment, it is necessary to calculate the height value corresponding to each side pixel coordinate system of each second coordinate according to the second matrix, the second coordinate corresponding to each side camera pixel coordinate system, the projection coordinate corresponding to each side camera pixel coordinate system, and the first transformation matrix corresponding to each side camera pixel coordinate system. The specific calculation formula is as follows:

[0155]

[0156] Among them, z k is the second coordinate (u k ,v k ) corresponding to the height value, h cij is the parameter of the i-th row and j-th column in the first transformation matrix, h pij is the parameter of the i-th row and j-th column in the second transformation matrix. It can be understood that for the second coordinates corresponding to different side camera pixel coordinate systems, the first transformation matrix corresponding to the corresponding side camera pixel coordinate system is used for calculation, so as to obtain the height values ​​corresponding to the different side camera pixel coordinate systems.

[0157] As described above, in the embodiment of the present invention, since the central projection and the central camera use the same lens, the first transformation matrix introduced makes the central projection and the central camera coaxial constraint, and the error will be much smaller than the rotation matrix and translation vector of the central projection and the central camera in two systems, thereby improving the accuracy of solving the height value. In addition, in this embodiment, the height value corresponding to the pixel coordinate system of each side camera is also solved, so that the height value corresponding to the pixel coordinate system of each side camera can be merged later, further improving the accuracy of calculating the height value.

[0158] In one embodiment, after determining the second coordinates and the height value of the projection coordinates corresponding to each side camera pixel coordinate system in the center camera pixel coordinate system in step 20322, the method further includes:

[0159] Step 20323: determine the zoom factor, where the zoom factor is the ratio of the pixel size of the center camera pixel coordinate system to the pixel size of the side camera pixel coordinate system.

[0160] Step 20324, scale the first coordinate according to the scaling factor.

[0161] In one embodiment, after converting the projection coordinates in the central projection pixel coordinate system into the second coordinates in the central camera pixel coordinate system, the first coordinates need to be further scaled so as to align the pixel size in the side camera pixel coordinate system with the pixel size in the central camera pixel coordinate system. Specifically, the scaling factor needs to be determined first. The scaling factor is the factor by which the first coordinates need to be scaled. In this embodiment, the scaling factor is the ratio of the pixel size in the central camera pixel coordinate system to the pixel size in the side camera pixel coordinate system. The pixel size in the central camera pixel coordinate system and the pixel size in the side camera pixel coordinate system can be obtained during the calibration process. After obtaining the scaling factor, the first coordinate (u c ,v c ) is multiplied by the scaling factor, so that the first coordinate (u c ,v c ) to scale and obtain the scaled first coordinate (u c′,v c ′), so as to align the pixel size of the side camera pixel coordinate system with the pixel size of the center camera pixel coordinate system.

[0162] As described above, after solving the second coordinate, the embodiment of the present invention further scales the first coordinate according to the ratio of the pixel size of the central camera pixel coordinate system to the pixel size of the side camera pixel coordinate system, thereby aligning the pixel size of the side camera pixel coordinate system with the pixel size of the central camera pixel coordinate system, so as to subsequently align the height map with the texture map of the central camera.

[0163] Step 204: determine a target height value corresponding to the second coordinate based on the height values ​​corresponding to the multiple side cameras.

[0164] After obtaining the height value corresponding to each second coordinate in the pixel coordinate system of the central camera and each side camera pixel coordinate system, the target height value corresponding to each second coordinate can be determined according to the height value of the second coordinate corresponding to each side camera, that is, the only height value corresponding to the second coordinate. Exemplarily, the invalid values ​​in the height value corresponding to each second coordinate can be eliminated first, for example, the minimum and maximum values ​​of the height value can be set, and the height values ​​outside the minimum and maximum values ​​can be determined as invalid values ​​and eliminated. Then, the height values ​​after the invalid values ​​are eliminated are merged to obtain the target height value corresponding to the second coordinate.

[0165] Based on the above embodiment, determining the target height value corresponding to the second coordinate based on the height values ​​corresponding to the multiple side cameras in step 204 includes:

[0166] Step 2041: Determine a valid height value among a plurality of height values ​​corresponding to the second coordinate.

[0167] After determining the height values ​​corresponding to the pixel coordinate systems of multiple side cameras in the second coordinate of the central camera pixel coordinate system, it is necessary to determine the valid height value among the multiple height values. The valid height value is the height value for which the height value is valid data. It is understandable that in the process of shooting with the side camera, the object to be measured in the captured target image may be blocked, or it may be overexposed or too dark, resulting in the height value generated subsequently being invalid data. Therefore, it is first necessary to filter out the invalid data in the height value to obtain the valid height value. In one embodiment, the valid height value in the height value can be filtered out by setting a threshold range.

[0168] Step 2042: Determine a target height value corresponding to the second coordinate according to the effective height value corresponding to the second coordinate.

[0169] After determining the effective height value, the target height value corresponding to each second coordinate in the central camera pixel coordinate system can be determined according to the effective height value, wherein the target height value is the only height value corresponding to the second coordinate. In one embodiment, the target height value can be generated by fusing the effective height values ​​corresponding to each second coordinate.

[0170] As described above, after solving the height value, the embodiment of the present invention further filters out the valid height value from the height value, thereby eliminating invalid data, and finally determines the target height value corresponding to the second coordinate based on the valid height value, thereby improving the accuracy of the subsequently generated target height value.

[0171] Based on the above embodiment, determining the target height value corresponding to the second coordinate according to the effective height value corresponding to the second coordinate in step 2042 includes:

[0172] Step 20421: Determine the maximum height value with the largest value and the minimum height value with the smallest value among the valid height values ​​corresponding to each second coordinate.

[0173] In one embodiment, when determining the target height value corresponding to each second coordinate, it is first necessary to determine the maximum height value with the largest value and the minimum height value with the smallest value among the valid height values ​​corresponding to each second coordinate.

[0174] Step 20422: When the difference between the maximum height value and the minimum height value is within a preset range, the effective height values ​​corresponding to the second coordinates are merged to obtain a target height value corresponding to the second coordinates.

[0175] After determining the maximum height value and the minimum height value corresponding to each second coordinate, it is necessary to further calculate the difference between the maximum height value and the minimum height value. It can be understood that in this embodiment, although there may be deviations in the effective height value corresponding to the second coordinate of each camera pixel coordinate system, the deviation is generally small, that is, the deviation generally does not exceed a certain range. If the deviation between the maximum height value and the minimum height value corresponding to a second coordinate is too large, it means that there is erroneous height data. In addition, the preset range is a pre-set range, and the preset range can be set according to actual needs, and is not specifically limited in this embodiment.

[0176] If the difference between the maximum height value and the minimum height value corresponding to the second coordinate is within a preset range, the valid height value corresponding to the second coordinate can be further fused to obtain the target height value corresponding to the second coordinate. Exemplarily, in the case of four side cameras, if the second coordinate has only one valid height value, the valid height value is directly used as the target height value corresponding to the second coordinate. If there are two valid height values, the mean of the two valid height values ​​is taken as the fused target height value. If there are three valid height values, the median is taken as the fused target height value; if there are four valid height values, the mean is taken as the fused target height value.

[0177] Step 20423: When the difference between the maximum height value and the minimum height value is not within the preset range, the target height value corresponding to the second coordinate is set to an invalid height value.

[0178] If the difference between the maximum height value and the minimum height value corresponding to the second coordinate is within the preset range, the target height value corresponding to the second coordinate is set to an invalid height value. In the subsequent filtering process, the target height value corresponding to the second coordinate is filled. Exemplarily, in the filtering process, the average of the target height values ​​of other second coordinates within the preset range with the second coordinate as the center can be calculated, and the average is used as the target height value of the second coordinate.

[0179] As described above, after screening out the valid height values, the embodiment of the present invention further determines whether there is erroneous data in the valid height values ​​corresponding to each second coordinate. If there is no erroneous data, the valid height values ​​corresponding to the second coordinates are fused to determine the final target height value. If there is erroneous data, the target height value corresponding to the second coordinate is set to an invalid height value, and the wireless height value is filled in during the subsequent filtering process, thereby further improving the accuracy of the target height value.

[0180] Step 205: Generate a height map according to the target height values ​​corresponding to the multiple second coordinates.

[0181] After obtaining the target height value corresponding to each second coordinate in the pixel coordinate system of the central camera, a height map can be generated according to the target height values ​​corresponding to the multiple second coordinates. After the height map is generated, the height map can be aligned with the texture map collected by the central camera. Exemplarily, the height map can be upsampled to the resolution of the image captured by the central camera, so as to align the height map with the texture image. In one embodiment, the overall process of generating a height map according to the target height value is as follows: Figure 5 shown.

[0182] In another embodiment, after the height map and the texture image captured by the central camera are aligned, the height map may be filtered to eliminate noise in the height map.

[0183] As described above, an embodiment of the present invention provides a method for generating a height map. The embodiment of the present invention projects a sinusoidal fringe pattern onto the surface of an object to be measured through central projection, and simultaneously turns on a side camera to collect a target image of the surface of the object to be measured. After that, after determining the unfolding phase of the sinusoidal fringes of the sinusoidal fringe pattern within one period based on the target image, each second coordinate in the central camera pixel coordinate system and the height value corresponding to the second coordinate in each side camera pixel coordinate system are determined according to the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, the preset coordinate conversion matrix and the unfolding phase, and the target height value corresponding to each second coordinate is obtained by fusing the height value corresponding to each second coordinate, and finally a height map is generated according to the target height value. In the process of generating a height map in an embodiment of the present invention, when converting data in the pixel coordinate system of the side camera into data in the pixel coordinate system of the center camera, since the center projection and the center camera use the same lens, a coaxial constraint exists between the center projection and the center camera through the first transformation matrix obtained in the calibration process. The center projection coordinate system is used as an intermediary. First, the first coordinates of the pixel points in the target image in the pixel coordinate system of the side camera are converted into coordinates in the center projection pixel coordinate system by phase unfolding, and then the center projection pixel coordinate system is converted into coordinates in the center projection pixel coordinate system. This can effectively reduce the error introduced in the conversion process to improve the accuracy of the height map generated subsequently. The error of the height map finally generated will be much smaller than the rotation matrix and translation vector of the center projection and the center camera in the two systems, so that the final imaging accuracy is further improved, and the accuracy of the generated height map is improved, which solves the technical problem of poor accuracy of the height map generated by the 3D AOI device in the prior art during the three-dimensional reconstruction process.

[0184] The embodiment of the present invention further provides a height map generating device, which is applicable to a 3D AOI device, wherein the 3D AOI device includes a central camera and a central projection that share a lens, and includes a plurality of side cameras that are respectively arranged around the central camera, such as Figure 6 As shown, Figure 6 A schematic diagram of the structure of a height map generating device provided by an embodiment of the present invention. The height map generating device provided by an embodiment of the present invention includes:

[0185] The fringe projection module 301 is used to sequentially project sinusoidal fringe patterns onto the surface of the object to be measured through central projection, where the sinusoidal fringe patterns include transverse sinusoidal fringe patterns and vertical sinusoidal fringe patterns.

[0186] An image acquisition module 302 is used to acquire a target image of the object to be measured through each side camera each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured;

[0187] The height map generation module 303 is used to determine the height map of the object to be measured in the central camera pixel coordinate system according to the target images captured by at least two side cameras and a preset coordinate transformation matrix.

[0188] Based on the above embodiment, the height map generation module 303 includes:

[0189] A height value determination submodule is used to determine the second coordinate and height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera according to the target image collected by each side camera and a preset coordinate conversion matrix, and one side camera pixel coordinate system corresponds to one height value;

[0190] a target height value determination submodule, configured to determine a target height value corresponding to the second coordinate based on a plurality of height values ​​corresponding to a plurality of side cameras;

[0191] The height map generation submodule is used to generate a height map according to target height values ​​corresponding to multiple second coordinates.

[0192] Based on the above embodiment, the height value determination submodule includes:

[0193] An unwrapping phase determination unit, used to determine the unwrapping phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle according to the target image;

[0194] The height value determination unit is used to determine the second coordinate and height value corresponding to the first coordinate in each side camera pixel coordinate system in the central camera pixel coordinate system based on the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system, a preset coordinate conversion matrix and an unfolding phase.

[0195] Based on the above embodiment, the phase determination unit includes:

[0196] a wrapping phase determination subunit, for determining a wrapping phase corresponding to each frequency according to a target image corresponding to the sinusoidal fringes of each frequency of the sinusoidal fringe pattern;

[0197] The phase unwrapping subunit is used to unwrap the wrapped phase corresponding to the target frequency to obtain the unwrapped phase of the sinusoidal fringe within one period, and the target frequency is the frequency with the largest value.

[0198] Based on the above embodiment, the height value determining unit includes:

[0199] The projection coordinate conversion subunit is used to determine the projection coordinate corresponding to the first coordinate in the central projection pixel coordinate system according to the unfolded phase and the first coordinate of the pixel point in each target image in the corresponding side camera pixel coordinate system;

[0200] The height value determination subunit is used to determine the second coordinates and height values ​​of the projection coordinates corresponding to each side camera pixel coordinate system in the central camera pixel coordinate system according to the projection coordinates and a preset coordinate conversion matrix.

[0201] Based on the above embodiment, the projection coordinate conversion subunit includes:

[0202] A horizontal and vertical unfolding phase determination component is used to determine the horizontal unfolding phase corresponding to the horizontal sinusoidal stripes of the sinusoidal stripe pattern and the vertical unfolding phase corresponding to the vertical sinusoidal stripes of the sinusoidal stripe pattern in the unfolding phase;

[0203] A resolution determination component, for determining a projection resolution of a central projection;

[0204] The projection coordinate conversion component is used to determine the projection coordinate corresponding to the first coordinate of each side camera pixel coordinate system in the central projection pixel coordinate system according to the horizontal unfolding phase, the vertical unfolding phase, the projection resolution and the first coordinate of each side camera pixel coordinate system.

[0205] On the basis of the above embodiment, the preset coordinate conversion matrix includes a plurality of first conversion matrices for converting coordinates of different side camera pixel coordinate systems into coordinates in a world coordinate system, a second conversion matrix for converting coordinates in a central projection pixel coordinate system into coordinates in a world coordinate system, and a third conversion matrix for converting coordinates in a central projection pixel coordinate system into coordinates in a central camera pixel coordinate system;

[0206] The height value determination subunit includes:

[0207] A second coordinate determination component, used for converting the projection coordinates in the central projection pixel coordinate system into second coordinates in the central camera pixel coordinate system according to a third transformation matrix;

[0208] The height value determination component is used to determine multiple height values ​​corresponding to the second coordinate based on the second coordinate, the projection coordinate, the first transformation matrix corresponding to multiple side camera pixel coordinate systems, and the second transformation matrix, where one side camera pixel coordinate system corresponds to one height value.

[0209] On the basis of the above embodiment, it further includes a scaling factor determination module, which is used to determine the scaling factor after determining the second coordinate and the height value of the projection coordinate corresponding to each side camera pixel coordinate system in the central camera pixel coordinate system, wherein the scaling factor is the ratio of the pixel size of the central camera pixel coordinate system to the pixel size of the side camera pixel coordinate system;

[0210] The coordinate scaling module is used to scale the first coordinate according to the scaling factor.

[0211] Based on the above embodiment, the target height value determination submodule includes:

[0212] A filtering unit, configured to determine a valid height value among a plurality of height values ​​corresponding to the second coordinate;

[0213] The target height value determining unit is used to determine the target height value corresponding to the second coordinate according to the effective height value corresponding to the second coordinate.

[0214] Based on the above embodiment, the target height value determination unit includes:

[0215] The maximum value determination subunit is used to determine a maximum height value with the largest value and a minimum height value with the smallest value among the valid height values ​​corresponding to each second coordinate;

[0216] A fusion subunit, for fusing the effective height values ​​corresponding to the second coordinates to obtain a target height value corresponding to the second coordinates when the difference between the maximum height value and the minimum height value is within a preset range;

[0217] The invalid height value determination subunit is used to set the target height value corresponding to the second coordinate to an invalid height value when the difference between the maximum height value and the minimum height value is not within a preset range.

[0218] The height map generating device provided in the embodiment of the present invention is included in the 3D AOI equipment, and can be used to execute the height map generating method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0219] It is worth noting that in the embodiment of the above-mentioned height map generating device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0220] This embodiment also provides a 3D AOI device, such as Figure 7 As shown, Figure 7A schematic diagram of the structure of a 3D AOI device provided in an embodiment of the present invention. The 3D AOI device 40 provided in an embodiment of the present invention includes a processor 400, a memory 401, a central camera 403 and a central projection 404 sharing a lens, and a plurality of side cameras 405 respectively arranged around the central camera 403;

[0221] The memory 401 is used to store the computer program 402 and transmit the computer program 402 to the processor 400;

[0222] The processor 400 is configured to execute the steps in the above-mentioned embodiment of a method for generating a height map according to instructions in the computer program 402 .

[0223] Exemplarily, the computer program 402 may be divided into one or more modules / units, one or more modules / units are stored in the memory 401, and executed by the processor 400 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 402 in the 4D AOI device 40.

[0224] The 4D AOI device 40 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The 4D AOI device 40 may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art will appreciate that Figure 7 It is only an example of the 4D AOI device 40 and does not constitute a limitation of the 4D AOI device 40. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the 4D AOI device 40 may also include input and output devices, network access devices, buses, etc.

[0225] The processor 400 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0226] The memory 401 may be an internal storage unit of the 4D AOI device 40, such as a hard disk or memory of the 4D AOI device 40. The memory 401 may also be an external storage device of the 4D AOI device 40, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the 4D AOI device 40. Further, the memory 401 may also include both an internal storage unit of the 4D AOI device 40 and an external storage device. The memory 401 is used to store computer programs and other programs and data required by the 4D AOI device 40. The memory 401 may also be used to temporarily store data that has been output or is to be output.

[0227] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0228] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0229] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0231] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store computer programs.

[0232] The embodiment of the present invention further provides a storage medium including computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute a height map generation method. The height map generation method is applicable to a 3D AOI device. The 3D AOI device includes a central camera and a central projection that share a lens, and includes a plurality of side cameras that are respectively arranged around the central camera. The method includes:

[0233] The sinusoidal fringe patterns are sequentially projected onto the surface of the object to be measured through central projection, wherein the sinusoidal fringe patterns include a horizontal sinusoidal fringe pattern and a vertical sinusoidal fringe pattern;

[0234] Each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured, a target image of the object to be measured is collected by each side camera;

[0235] A height map of the object to be measured in the pixel coordinate system of the central camera is determined according to the target images captured by at least two side cameras and a preset coordinate transformation matrix.

[0236] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the embodiments of the present invention are not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of the present invention. Therefore, although the embodiments of the present invention are described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the embodiments of the present invention, and the scope of the embodiments of the present invention is determined by the scope of the appended claims.

Claims

1. A sinusoidal fringe combination, characterized in that: include: A plurality of high-frequency sinusoidal stripes, wherein a plurality of sinusoidal stripes of the same frequency form a sinusoidal stripe pattern according to a preset period, wherein the number of projection pixels covered by the high-frequency sinusoidal stripes in one period is at least 8, and the horizontal sinusoidal stripe frequency and the vertical sinusoidal stripe frequency are determined according to the projection resolution of the projector, and the horizontal or vertical sinusoidal stripe pattern at a single frequency is projected in sequence by the projector until the horizontal sinusoidal stripe pattern and the vertical sinusoidal stripe pattern of all frequencies are projected.

2. The sinusoidal fringe combination according to claim 1, characterized in that: Also includes: At least two low-frequency sinusoidal stripes, multiple sinusoidal stripes of the same frequency form a sinusoidal stripe pattern according to a preset period, the number of projection pixels covered by the low-frequency sinusoidal stripes in one period ranges from 40 to 50, and the horizontal sinusoidal stripe frequency and the vertical sinusoidal stripe frequency are determined according to the projection resolution.

3. A sinusoidal fringe combination according to any one of claims 1 or 2, characterized in that: The number of steps of the sinusoidal fringe combination is at least three.

4. A height map generation method, applicable to a 3D AOI device, wherein the 3D AOI device comprises a central camera and a central projection sharing a lens, and comprises a plurality of side cameras respectively arranged around the central camera, characterized in that: The method comprises: sequentially projecting the sinusoidal fringe pattern according to any one of claims 1 to 3 onto the surface of the object to be measured through the central projection, wherein the sinusoidal fringe pattern includes a horizontal sinusoidal fringe pattern and a vertical sinusoidal fringe pattern; Each time the sinusoidal fringe pattern is projected onto the surface of the object to be measured, a target image of the object to be measured is collected by each of the side cameras; A height map of the object to be measured in the central camera pixel coordinate system is determined according to the target images captured by the at least two side cameras and a preset coordinate transformation matrix.

5. A method for generating a height map according to claim 4, characterized in that: The step of determining a height map of the object to be measured in a central camera pixel coordinate system according to the target images captured by the at least two side cameras and a preset coordinate transformation matrix includes: According to the target image captured by each of the side cameras and the preset coordinate conversion matrix, determine the second coordinate and the height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera, and one pixel coordinate system of the side camera corresponds to one height value; Determine a target height value corresponding to the second coordinate based on a plurality of the height values ​​corresponding to a plurality of the side cameras; A height map is generated according to the target height values ​​corresponding to the plurality of second coordinates.

6. A method for generating a height map according to claim 5, characterized in that: The determining, based on the target image captured by each of the side cameras and a preset coordinate conversion matrix, of a second coordinate and a height value corresponding to a first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the central camera includes: Determine the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle according to the target image; According to the first coordinates of the pixel points in each of the target images in the corresponding side camera pixel coordinate system, the preset coordinate transformation matrix and the unfolding phase, the second coordinates and height values ​​corresponding to the first coordinates in the side camera pixel coordinate system in the center camera pixel coordinate system are determined.

7. A method for generating a height map according to claim 6, characterized in that: The step of determining the unwrapped phase of the sinusoidal fringes of the sinusoidal fringe pattern within one cycle according to the target image comprises: Determining a wrapping phase corresponding to each frequency according to a target image corresponding to the sinusoidal fringes of each frequency of the sinusoidal fringe pattern; The wrapped phase corresponding to the target frequency is unfolded to obtain the unfolded phase of the sinusoidal fringes within one period, and the target frequency is the frequency with the largest value.

8. A method for generating a height map according to claim 6, characterized in that: The method of determining the second coordinate and height value corresponding to the first coordinate in the pixel coordinate system of each side camera in the pixel coordinate system of the center camera according to the first coordinate of the pixel point in each of the target images in the corresponding side camera pixel coordinate system, the preset coordinate conversion matrix and the unfolding phase, comprises: Determine, according to the unwrapped phase and the first coordinates of the pixel points in each of the target images in the corresponding side camera pixel coordinate system, the projection coordinates corresponding to the first coordinates in the central projection pixel coordinate system; According to the projection coordinates and the preset coordinate conversion matrix, the second coordinates and the height value of the projection coordinates corresponding to each of the side camera pixel coordinate systems in the central camera pixel coordinate system are determined.

9. A method for generating a height map according to claim 8, characterized in that: The determining, according to the unwrapped phase and the first coordinate of the pixel point in each of the target images in the corresponding side camera pixel coordinate system, the projection coordinate corresponding to the first coordinate in the central projection pixel coordinate system comprises: Determine a horizontal unwrapped phase corresponding to the horizontal sinusoidal fringes of the sinusoidal fringe pattern and a vertical unwrapped phase corresponding to the vertical sinusoidal fringes of the sinusoidal fringe pattern in the unwrapped phase; determining a projection resolution of the central projection; According to the lateral unfolding phase, the vertical unfolding phase, the projection resolution and the first coordinate of each of the side camera pixel coordinate systems, a projection coordinate in the central projection pixel coordinate system corresponding to the first coordinate of each of the side camera pixel coordinate systems is determined.

10. A method for generating a height map according to claim 8, characterized in that: The preset coordinate conversion matrix includes a plurality of first conversion matrices for converting coordinates of different side camera pixel coordinate systems into coordinates in a world coordinate system, a second conversion matrix for converting coordinates in the central projection pixel coordinate system into coordinates in a world coordinate system, and a third conversion matrix for converting coordinates in the central projection pixel coordinate system into coordinates in a central camera pixel coordinate system; The step of determining the second coordinates and the height value of the projection coordinates corresponding to each of the side camera pixel coordinate systems in the center camera pixel coordinate system according to the projection coordinates and the preset coordinate conversion matrix includes: According to the third transformation matrix, the projection coordinates in the central projection pixel coordinate system are converted into second coordinates in the central camera pixel coordinate system; According to the second coordinates, the projection coordinates, the first transformation matrix corresponding to the plurality of side camera pixel coordinate systems, and the second transformation matrix, a plurality of height values ​​corresponding to the second coordinates are determined, one height value corresponding to one side camera pixel coordinate system.

11. A method for generating a height map according to claim 10, characterized in that: After determining the second coordinates and the height value of the projection coordinates corresponding to each of the side camera pixel coordinate systems in the central camera pixel coordinate system, the method further includes: Determine a zoom factor, where the zoom factor is a ratio of a pixel size of the central camera pixel coordinate system to a pixel size of the side camera pixel coordinate system; The first coordinate is scaled according to the scaling factor.

12. A method for generating a height map according to claim 5, characterized in that: The determining a target height value corresponding to the second coordinate based on the plurality of height values ​​corresponding to the plurality of side cameras comprises: determining a valid height value among the plurality of height values ​​corresponding to the second coordinate; A target height value corresponding to the second coordinate is determined according to the effective height value corresponding to the second coordinate.

13. A method for generating a height map according to claim 12, characterized in that: The step of determining a target height value corresponding to the second coordinate according to the effective height value corresponding to the second coordinate comprises: Determine a maximum height value having the largest value and a minimum height value having the smallest value among the valid height values ​​corresponding to each of the second coordinates; When the difference between the maximum height value and the minimum height value is within a preset range, the effective height values ​​corresponding to the second coordinates are merged to obtain a target height value corresponding to the second coordinates; When the difference between the maximum height value and the minimum height value is not within a preset range, the target height value corresponding to the second coordinate is set as an invalid height value.

14. A 3D AOI device, characterized in that: The 3D AOI device includes a processor, a memory, a central camera and a central projection sharing a lens, and a plurality of side cameras respectively arranged around the central camera; The memory is used to store a computer program and transmit the computer program to the processor; The processor is used to execute a height map generation method as described in any one of claims 4-13 according to instructions in the computer program.

15. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform a height map generation method according to any one of claims 4 to 13.

Images