Structured light-assisted binocular stereo matching fusion method, system, equipment and medium

Abstract

The invention relates to a structured light-assisted binocular stereo matching fusion method, system and device and a medium. The method comprises the following steps: projecting a plurality of groups of high-frequency sine stripes to the surface of a target object, and superimposing and projecting a Gray code pattern; shooting the surface of the target object to obtain left and right images; and extracting a wrapped phase from the left and right images, and calculating to obtain an absolute phase according to the wrapped phase and the decoded information of the Gray code pattern. And calculating the matching cost of each pixel in the left image and the right image under different parallax, and carrying out cost aggregation processing in a parallax space by taking an absolute phase as a constraint to obtain a parallax map. And optimizing the disparity map to obtain the optimal disparity. And generating a three-dimensional point cloud according to the optimal parallax and the calibration parameters. According to the method, texture information can be enhanced, and the reconstruction integrity and geometric accuracy of weak-texture and high-reflection non-abandoned products are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to a structured light-assisted binocular stereo matching fusion method, system, device and medium. Background Art

[0002] Binocular stereo matching is a key research area in computer vision. It aims to calculate the depth of a scene using images captured by two cameras (a binocular system). Binocular stereo matching is a core technology for autonomous driving, robotic navigation, and augmented and virtual reality (AR / VR). Its accuracy has been significantly improved with the advancement of deep learning.

[0003] The algorithm currently used most in the industry is the SGM (Semi-Global Matching) algorithm. The SGM (Semi-Global Matching) algorithm is an efficient and high-precision binocular stereo matching algorithm that calculates the disparity map by combining the efficiency of local matching and the accuracy of global optimization. Its core idea is multi-path cost aggregation, that is, dynamic programming of pixel matching costs along multiple directions (such as horizontal, vertical, and diagonal), and then aggregating the results to enhance robustness. SGM calculates the initial matching cost through mutual information (MI) or Census transform, and introduces smoothness constraints (such as penalizing adjacent disparity discontinuities) to optimize the disparity space. However, this algorithm has the following disadvantages: areas with weak textures (such as white walls and solid-color objects) lack sufficient features, resulting in the matching algorithm being unable to generate reliable matching costs and being mismatched. In particular, the three-dimensional reconstruction of intangible cultural heritage products with weak textures is a difficult problem. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a structured light-assisted binocular stereo matching fusion method, system, device and medium in response to the above-mentioned defects.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a structured light-assisted binocular stereo matching fusion method, comprising the following steps:

[0006] Project multiple sets of high-frequency sinusoidal stripes onto the surface of the target object and superimpose the projected Gray code pattern;

[0007] photographing the surface of the target object to obtain left and right images;

[0008] Extracting a wrapping phase from the left and right images, and calculating an absolute phase based on the wrapping phase and information decoded from the Gray code pattern;

[0009] Calculating the matching cost of each pixel in the left and right images at different disparities, and performing cost aggregation processing in the disparity space with the absolute phase as a constraint to obtain a disparity map;

[0010] Optimizing the disparity map to obtain an optimal disparity;

[0011] A three-dimensional point cloud is generated according to the optimal disparity and calibration parameters.

[0012] Furthermore, in the structured light-assisted binocular stereo matching fusion method described in the present invention, the number of groups of the high-frequency sinusoidal stripes does not exceed 6 groups.

[0013] Furthermore, in the structured light-assisted binocular stereo matching fusion method described in the present invention, the number of groups of high-frequency sinusoidal stripes is determined according to the confidence of the three-dimensional model reconstructed by generating the three-dimensional point cloud.

[0014] Furthermore, in the structured light-assisted binocular stereo matching fusion method of the present invention, the step of extracting the wrapped phase from the left and right images includes:

[0015] The wrapping phase is extracted from the left and right images by a phase shift method.

[0016] Furthermore, in the structured light-assisted binocular stereo matching fusion method of the present invention, the step of optimizing the disparity map includes:

[0017] The disparity map is optimized by a dynamic programming method.

[0018] Furthermore, in the structured light-assisted binocular stereo matching fusion method described in the present invention, the calibration parameters are binocular camera calibration parameters.

[0019] Furthermore, in the structured light-assisted binocular stereo matching fusion method of the present invention, the information obtained after decoding the Gray code pattern is the fringe level of high-frequency sinusoidal fringes.

[0020] In addition, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor to execute the steps of the structured light-assisted binocular stereo matching fusion method as described above.

[0021] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the above-mentioned structured light-assisted binocular stereo matching fusion method by calling the computer program stored in the memory.

[0022] In addition, the present invention also provides a structured light-assisted binocular stereo matching fusion system, comprising a binocular camera, a structured light projector, and a computer device as described above, wherein the computer device is communicatively connected to the binocular camera and the structured light projector, respectively.

[0023] The structured light-assisted binocular stereo matching fusion method, system, device and medium of the present invention have the following beneficial effects: the method combines sinusoidal stripes and Gray code patterns to synchronously collect modulated high-precision phase information, improves the traditional binocular matching algorithm based on phase consistency constraints, and enhances texture information by introducing phase information in weak texture areas, significantly improving the reconstruction completeness and geometric accuracy of weak-textured and highly reflective intangible cultural heritage products. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 It is a flow chart of the structured light-assisted binocular stereo matching and fusion method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific embodiments of the present invention are now described in detail with reference to the accompanying drawings. In the following description, it should be noted that, unless otherwise clearly specified and limited, terms such as "installed", "connected", "connected", "fixed", and "set" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integrated connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. When an element is referred to as being "on" or "under" another element, the element can be "directly" or "indirectly" located above the other element, or there may be one or more intervening elements.

[0027] It should also be noted that highly reflective intangible cultural heritage products refer to intangible cultural heritage products with highly reflective properties, such as certain traditional handicrafts and metal products. During three-dimensional scanning and imaging, these products are prone to reflected light interference due to their high surface reflectivity, resulting in imaging difficulties. Structured light assistance is a technology used for three-dimensional measurement. It calculates the three-dimensional shape of the object by projecting a known light pattern (such as stripes, Gray code, etc.) onto the surface of an object and using a camera to capture changes in the reflected light pattern. This method is mainly used for high-precision industrial inspection and three-dimensional reconstruction. It should also be noted that sinusoidal stripes are a common structured light pattern. By projecting sinusoidal light stripes onto the surface of an object, the phase information is used to calculate the three-dimensional shape of the object. Wrapping phase refers to the phase discontinuity phenomenon caused by the periodic repetition of phase information. In binocular stereo matching and structured light measurement, the wrapping phase problem needs to be solved by a phase ambiguity resolution algorithm.

[0028] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0029] refer to Figure 1 In a preferred embodiment, the structured light-assisted binocular stereo matching fusion method of this embodiment includes the following steps:

[0030] S1. Project multiple sets of high-frequency sinusoidal stripes onto the surface of the target object and superimpose a Gray code pattern. Specifically, the target object is identified and, before imaging the target object, multiple sets of high-frequency sinusoidal stripes are projected onto the surface of the target object. Then, the Gray code pattern is projected onto the surface of the target object, thereby creating a superimposed effect between the high-frequency sinusoidal stripes and the Gray code pattern.

[0031] It will be appreciated that in some optional embodiments, the maximum number of high-frequency sinusoidal fringe groups is six. Furthermore, the number of high-frequency sinusoidal fringe groups can be determined based on the confidence level of the 3D model reconstructed from the generated 3D point cloud. In other words, different objects can use projected fringe patterns of varying accuracy, with confidence level used for judgment.

[0032] It should be noted that the frequency of the high-frequency sinusoidal fringes used in the embodiments of the present invention is much higher than the low-frequency fringes used in traditional fringe projection technology. The frequency range is not specifically limited and can be determined based on specific needs. For example, the frequency range of the high-frequency sinusoidal fringes can be higher than 50 Hz or between 50 Hz and 100 Hz.

[0033] S2. Photographing the surface of the target object to obtain left and right images. That is, after the optical structure projector completes the projection of the high-frequency sinusoidal stripes and the Gray code pattern, it photographs the surface of the target object to obtain left and right images.

[0034] S3. Extract the wrapped phase from the left and right images, and calculate the absolute phase based on the wrapped phase and the decoded Gray code pattern information. Specifically, the decoded Gray code pattern information is the fringe sequence of high-frequency sinusoidal fringes. In this embodiment, the wrapped phase can be extracted from the left and right images using a phase shifting method. The phase shifting method in this embodiment of the present invention can be referenced to existing technologies and will not be further described here.

[0035] S4. Calculate the matching cost of each pixel in the left and right images at different disparities, and perform cost aggregation in the disparity space with the absolute phase as a constraint to obtain a disparity map.

[0036] S5. Optimize the disparity map to obtain the optimal disparity. Specifically, in this embodiment, the disparity map can be optimized by a dynamic programming method. It should be noted that the dynamic programming method can refer to the existing technology and will not be described in detail here.

[0037] S6. Generate a 3D point cloud based on the optimal disparity and calibration parameters. The calibration parameters are the calibration parameters of the binocular camera. It will be understood that in this embodiment of the present invention, the disparity space refers to the image coverage area, while the disparity map is the distance information of the object from the camera. The disparity can be converted into distance information using the camera parameters, and with this distance information, a 3D point cloud can be generated.

[0038] It can be understood that the embodiment of the present invention first projects multiple groups of high-frequency sinusoidal stripes, extracts the wrapped phase through the phase shift method, and achieves sub-pixel accuracy. By superimposing the Gray code pattern and performing Gray code decoding to determine the stripe level, the phase ambiguity problem can be solved. Then, the absolute phase is calculated by combining the phase and Gray code decoding information, and the camera-projector pixel correspondence is established. Then, with the absolute phase as a constraint, cost aggregation is performed in the disparity space, and the disparity map is optimized through dynamic programming methods to retain details and smooth noise. Based on the disparity and calibration parameters, a high-precision three-dimensional point cloud is generated to achieve three-dimensional reconstruction. This embodiment

[0039] This embodiment combines sinusoidal stripes with Gray code patterns to synchronously capture high-precision phase information from the modulated state. Using phase consistency constraints, it improves the traditional binocular matching algorithm, introducing phase information in areas with weak texture. This method enhances texture information through phase information, significantly improving the reconstruction integrity and geometric accuracy of weakly textured and highly reflective intangible cultural heritage artifacts.

[0040] In another preferred embodiment, the computer-readable storage medium of this embodiment stores a computer program suitable for loading by a processor to execute the steps of the structured light-assisted binocular stereo matching fusion method described in the above embodiment. This embodiment combines sinusoidal stripes with Gray code patterns to synchronously capture modulated high-precision phase information, improves the traditional binocular matching algorithm based on phase consistency constraints, and introduces phase information in weakly textured areas. This method can enhance texture information through phase information, significantly improving the reconstruction integrity and geometric accuracy of weakly textured and highly reflective intangible cultural heritage artifacts.

[0041] In another preferred embodiment, the computer device of this embodiment includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps of the structured light-assisted binocular stereo matching fusion method of the above embodiment by calling the computer program stored in the memory. This embodiment combines sinusoidal stripes with Gray code patterns to synchronously collect modulated high-precision phase information, improves the traditional binocular matching algorithm based on phase consistency constraints, and introduces phase information in weak texture areas. This method can enhance texture information through phase information, significantly improving the reconstruction completeness and geometric accuracy of weak-texture, highly reflective intangible cultural heritage products.

[0042] In another preferred embodiment, the structured light-assisted binocular stereo matching fusion system of this embodiment includes a binocular camera, a structured light projector, and the computer device of the above embodiment, and the computer device is respectively connected to the binocular camera and the structured light projector. This embodiment first builds a hardware system of a binocular camera and a structured light projector, combines sinusoidal stripes (high precision) with Gray code (phase ambiguity resolution), and synchronously collects modulated high-precision phase information; then improves the traditional binocular matching algorithm based on phase consistency constraints, and introduces phase and other information in weak texture areas. This method enhances texture information through phase and other information, significantly improving the reconstruction completeness and geometric accuracy of weak-texture, highly reflective intangible cultural heritage products.

[0043] It can be understood that the embodiments of the present invention can improve the matching accuracy of weak texture areas. In addition to being used for weak texture areas of intangible cultural heritage products, it can also be used for flatness measurement of automobile rearview mirrors, etc.

[0044] The computer-readable storage medium of the present invention can be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0045] The processor of the present invention is used to provide computing and control capabilities to support the operation of the entire device. It should be understood that in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0046] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0047] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0048] It is understandable that the above embodiments only express the preferred implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the patent scope of the present invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, the above technical features can be freely combined, and several deformations and improvements can be made, all of which fall within the scope of protection of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should fall within the scope of coverage of the claims of the present invention.

Claims

1. A structured light-assisted binocular stereo matching fusion method, characterized in that: The following steps are involved: Project multiple sets of high-frequency sinusoidal stripes onto the surface of the target object and superimpose the projected Gray code pattern; photographing the surface of the target object to obtain left and right images; Extracting a wrapping phase from the left and right images, and calculating an absolute phase based on the wrapping phase and information decoded from the Gray code pattern; Calculating the matching cost of each pixel in the left and right images at different disparities, and performing cost aggregation processing in the disparity space with the absolute phase as a constraint to obtain a disparity map; Optimizing the disparity map to obtain an optimal disparity; A three-dimensional point cloud is generated according to the optimal disparity and calibration parameters.

2. The structured light-assisted binocular stereo matching fusion method according to claim 1, characterized in that: The number of groups of the high-frequency sinusoidal stripes does not exceed 6 groups.

3. The structured light-assisted binocular stereo matching fusion method according to claim 1, characterized in that: The number of groups of the high-frequency sinusoidal stripes is determined according to the confidence level of the three-dimensional model reconstructed by generating the three-dimensional point cloud.

4. The structured light-assisted binocular stereo matching fusion method according to claim 1, characterized in that: The step of extracting the wrapped phase from the left and right images includes: The wrapping phase is extracted from the left and right images by a phase shift method.

5. The structured light-assisted binocular stereo matching fusion method according to claim 1, characterized in that: The step of optimizing the disparity map includes: The disparity map is optimized by a dynamic programming method.

6. The structured light-assisted binocular stereo matching fusion method according to claim 1, characterized in that: The calibration parameters are binocular camera calibration parameters.

7. The structured light-assisted binocular stereo matching fusion method according to claim 1, characterized in that: The information after decoding the Gray code pattern is the fringe level of the high-frequency sinusoidal fringe.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor to execute the steps of the structured light-assisted binocular stereo matching fusion method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the steps of the structured light-assisted binocular stereo matching fusion method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A structured light-assisted binocular stereo matching fusion system, characterized in that: The system comprises a binocular camera, a structured light projector and the computer device according to claim 9, wherein the computer device is communicatively connected to the binocular camera and the structured light projector respectively.

Images