Environment perception calibration system and method based on image acquisition and structured light projection

Abstract

The application discloses an environment perception calibration system and method based on image acquisition and structured light projection, and belongs to the field of environment perception. In view of the problems of high cost and poor precision of existing environment perception, the application provides an environment perception calibration method based on image acquisition and structured light projection, and a structured light coding pattern with a preset known topological structure is projected to a space to be measured; an infrared image is obtained, which is an image formed by the structured light coding pattern being mapped on a medium surface; feature points are extracted from the infrared image, and two-dimensional pixel coordinates corresponding to the feature points are obtained; and the two-dimensional pixel coordinates are converted into actual physical depth information of the feature points through a multi-dimensional space depth mapping data set. When the application works, the depth information of the corresponding points is directly obtained through the multi-dimensional space depth mapping data set, so that the measurement precision is ensured, the online operation complexity and cost are reduced, and the binocular matching ambiguity problem caused by environmental light, texture repetition and other factors is avoided.

Description

Technical Field

[0001] This invention belongs to the field of optical environment perception and three-dimensional vision calibration, and more specifically, relates to an environment perception calibration system and method based on image acquisition and structured light projection. Background Technology

[0002] In intelligent sensing fields such as in-vehicle intelligent driving (e.g., reversing assistance for passenger cars, near-field collision avoidance for commercial vehicles, and near-field perception in accordance with EU MOIS regulations) and smart agriculture (e.g., harvesting robots, weeding robots), high-precision three-dimensional spatial depth acquisition is crucial for ensuring safety. However, existing near-field perception technologies suffer from the following fundamental pain points: 1. Poor environmental adaptability and narrow scene: Traditional time-of-flight (ToF) methods or pure optical binocular systems will experience severe drift or even failure in perception accuracy when facing complex natural environments such as extreme lighting (such as strong direct sunlight), mud obstruction, or reflection from vehicle windshields.

[0003] 2. High computational power consumption in online applications: Traditional structured light or multi-view vision solutions rely on real-time spatial 3D geometric reconstruction and complex mathematical models (such as high-order polynomial lens distortion models) for calculation, which makes it difficult to achieve high-frequency, low-latency real-time response on low-cost, low-computing-power automotive or industrial-grade chips.

[0004] 3. The mass production calibration process is complicated: The traditional sensor calibration process is extremely dependent on assembly accuracy. Once the sensor undergoes a slight mechanical deformation or displacement, the explicit mathematical model will fail, resulting in the inability to maintain long-term stable sensing accuracy after leaving the factory. Summary of the Invention

[0005] 1. The problem to be solved To address the issues of high computational cost and poor accuracy in existing environmental perception systems, this application provides an environmental perception calibration system and method based on image acquisition and structured light projection. During operation, this invention directly obtains the depth information of corresponding points through a multi-dimensional spatial depth mapping dataset, ensuring measurement accuracy while reducing online computational complexity and cost. It also avoids ambiguity in binocular matching caused by factors such as ambient lighting and texture repetition.

[0006] 2. Technical Solution To solve the above problems, the present invention adopts the following technical solution.

[0007] The environment perception calibration method based on image acquisition and structured light projection includes the following steps: Project a structured light coded pattern with a pre-defined known topology onto the space to be tested; Acquire an infrared image, which is the image formed by mapping the structured light coded pattern onto the surface of the medium; Feature points are extracted from infrared images, and the corresponding two-dimensional pixel coordinates are obtained. The two-dimensional pixel coordinates are transformed into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset; wherein the multi-dimensional spatial depth mapping dataset establishes a one-to-one mapping relationship between the image pixel coordinates and the actual physical depth of each known structured light coded pattern feature point.

[0008] Furthermore, converting the two-dimensional pixel coordinates into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset specifically includes: Determine the physical feature intersection index of the two-dimensional pixel coordinates; the physical feature intersection index is a pre-set unique number of the feature point in the structured light coding pattern; The multidimensional spatial depth mapping dataset is used to find two adjacent consecutive calibration sampling points in the coordinate trajectory of the physical feature intersection index where the two-dimensional pixel coordinates fall, and the corresponding calibration depth and the corresponding calibration pixel coordinates are obtained. Calculate the Euclidean distance between the two-dimensional pixel coordinates and the calibration pixel coordinates corresponding to the consecutive calibration sampling points in the two adjacent stages; Using the inverse Euclidean distance as a weighting coefficient, linear interpolation is performed on the calibration depth of two adjacent consecutive calibration sampling points to obtain the actual physical depth corresponding to the two-dimensional pixel coordinates.

[0009] Furthermore, the multidimensional spatial depth mapping dataset is acquired offline: The medium is driven to stop sequentially at N discrete calibration distance sampling points from near to far within the set working depth range; Collect and extract the two-dimensional pixel coordinates of feature points in infrared images at each calibrated distance sampling point; The two-dimensional pixel coordinates of each feature point are encoded and associated with the absolute depth value corresponding to the feature point to obtain a multi-dimensional spatial depth mapping dataset.

[0010] Furthermore, the medium is placed on a linear displacement stage, the direction of movement of which is parallel to the optical axis of the infrared image acquisition device; the medium is a target plate, the plane of which is perpendicular to the optical axis of the infrared image acquisition device.

[0011] Furthermore, when acquiring multidimensional spatial depth mapping datasets offline, a parallelism error compensation mechanism is also introduced: Determine the installation angle between the actual motion direction of the linear displacement stage and the optical axis of the camera; When constructing a multidimensional spatial depth mapping dataset, geometric projection relationships are used to perform geometric decoupling and compensation correction on the actual absolute depth values ​​of each sampling point.

[0012] Furthermore, when extracting the two-dimensional pixel coordinates of feature points, a topology tracking algorithm is used to uniformly number the feature points according to the topological geometric index order, thereby obtaining the mapping trajectory of the feature points as the distance changes.

[0013] Furthermore, after obtaining the two-dimensional pixel coordinates corresponding to the feature points in the infrared image, a dynamic anti-misalignment topology correction mechanism is performed on the two-dimensional pixel coordinates of the feature points before conversion.

[0014] Furthermore, a structured light coding pattern with a preset known topology is projected onto the space to be tested through a structured light projection module; infrared images are acquired through a monocular RGB-IR camera; and the monocular RGB-IR camera is fixedly connected to the structured light projection module.

[0015] Furthermore, after obtaining the actual physical depth information of the feature points, the process also includes converting the actual physical depth information of the feature points into the three-dimensional physical coordinates of the feature points in the infrared image under the coordinates of a monocular RGB-IR camera. Using the edge or color gradient texture in the RGB color image output by a monocular RGB-IR camera as guiding weights, the three-dimensional physical coordinates of feature points in the infrared image are used to generate the overall three-dimensional point cloud data of the infrared image.

[0016] A system for implementing the environment perception calibration method based on image acquisition and structured light projection as described in any of the above claims, comprising: Structured light projection module: used to project a structured light coded pattern with a preset known topology onto the space to be tested; Infrared image acquisition module: used to acquire infrared images, which are the images formed by mapping the structured light coding pattern onto the surface of the medium; The data processing module is used to extract feature points from infrared images and obtain the corresponding two-dimensional pixel coordinates of the feature points; and to convert the two-dimensional pixel coordinates into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset; wherein the multi-dimensional spatial depth mapping dataset establishes a one-to-one mapping relationship between the image pixel coordinates and the actual physical depth of each known structured light coded pattern feature point.

[0017] 3. Beneficial effects (1) This invention establishes a one-to-one mapping relationship between the image pixel coordinates of each feature point in the structured light coded pattern and its actual physical depth through an offline multidimensional spatial depth mapping dataset. This allows for the extraction of the two-dimensional coordinates of feature points from a single frame of infrared image during operation, enabling the direct acquisition of depth information of the corresponding points through table lookup and fast neighborhood interpolation. This eliminates the need for analytical geometric calculations such as online binocular matching, epipolar constraint search, or ray intersection. This fundamentally avoids the repetitive and time-consuming iterative optimization and geometric solution processes of traditional methods, significantly reducing the algorithm's dependence on processor computing power and memory bandwidth. It can run stably on low-computing-power and low-power embedded chip platforms, meeting the stringent requirements of automotive-grade or consumer-grade mass production solutions for real-time performance, power consumption and cost control, while ensuring the determinism and consistency of depth output; and the multi-dimensional spatial depth mapping dataset pre-solidifies the complex nonlinear mapping relationship between "pattern feature points - pixel coordinates - actual depth" in the structured light system into a set of discrete spatial lookup tables in an offline calibration method, which not only ensures measurement accuracy, but also greatly reduces the complexity of online computation, while avoiding the binocular matching ambiguity problem caused by factors such as ambient lighting and texture repetition. (2) This invention directly locks the target identity through the physical feature intersection index, so that the subsequent sampling point search is strictly constrained to the known depth coordinate trajectory of the feature point, avoiding the traditional steps of brute-force search or epipolar matching in the whole image or global space, eliminating the risk of mismatch from the source, and greatly reducing the data retrieval range; at the same time, searching on a single trajectory reduces the memory bandwidth occupation and cache miss rate, so that the data access efficiency remains highly stable on low computing power platforms; using the inverse Euclidean distance as the weighting coefficient for linear interpolation, compared with zero-order preservation, it can guarantee accuracy and smoothness; the entire conversion process only involves basic arithmetic operations, without iterative convergence or complex function differentiation, etc., which is efficient and can provide a highly predictable delay response for real-time environmental perception systems; (3) The offline calibration of the multidimensional spatial depth mapping dataset of this invention is achieved by using a high-precision linear displacement stage and target plate. It can complete the data acquisition of the full range and dense sampling points in an environment without time constraints and computing resource limitations. When working online, there is no need to repeat any calibration actions. The solidified dataset can be directly called to work, ensuring the stable output of real-time frame rate. At the same time, this point-by-point calibration based on physical benchmark replaces the indirect method of relying on mathematical models to calculate depth, laying the foundation for absolute accuracy of the dataset. Furthermore, a parallelism error compensation mechanism is introduced to implicitly include the nonlinear distortion of the lens and assembly error in the dataset, avoiding the cumulative error caused by inaccurate explicit modeling, and greatly improving the mass production reliability on the production line. (4) The introduction of the dynamic anti-misalignment topology correction mechanism of this invention enables the expected pixel position range of the broken edge node to be deduced in reverse by the continuous known nodes in the unbroken area when the local pattern is broken or partially occluded. Combined with displacement prediction on the time series, the nodes that may be misaligned are uniquely encoded and verified and online topology repair is performed, thereby greatly reducing the probability of false detection, ensuring the continuity in space and the stability in time, and significantly improving the reliability of depth perception in dynamic scenes or complex occlusion conditions. At the same time, a monocular RGB-IR camera and a structured light projection module are integrated and fixed. The RGB-IR camera can independently receive the encoded pattern projected by the structured light through the infrared channel, and is completely unaffected by changes in the color and brightness of the ambient visible light or direct strong light, ensuring that the feature point extraction accuracy remains stable under various extreme lighting conditions, providing a reliable data source for depth calculation. The fixed connection ensures the long-term stability of the calibration parameters, simplifying the use and maintenance costs for end users. This combination of equipment has extremely high environmental adaptability in smart agriculture (soil, occlusion, fruit maturity judgment) or vehicle reversing conditions. (5) The system of the present invention has a simple structure and realizes that online binocular matching or ray intersection analytical geometric calculation is not required when working online. By directly retrieving the pre-calibrated spatial mapping dataset and performing fast neighborhood interpolation, the algorithm consumes a lot of processor computing power, which perfectly meets the mass production requirements of low computing power chip platforms. At the same time, it ensures the determinism and consistency of depth output, which is conducive to reducing system response latency and improving overall robustness. Attached Figure Description

[0018] Figure 1 This is a schematic diagram showing the hardware spatial topology and rigid constraints of the monocular RGB-IR camera and the structured light projection module in this application; Figure 2 This is a schematic diagram illustrating the process of acquiring a multidimensional spatial depth mapping dataset offline. Figure 3 This is a flowchart illustrating the real-time online perception and calculation process for vehicle reversing and robot obstacle avoidance in this application. Detailed Implementation

[0019] The present invention will now be further described with reference to specific embodiments and accompanying drawings.

[0020] The environment perception calibration method based on image acquisition and structured light projection includes the following steps: A structured light coding pattern with a pre-defined known topology is projected onto the space to be tested. In this step, a grid pattern projector in the near-infrared 850nm or 940nm frequency band can be used to project the structured light coding pattern with the pre-defined known topology. The structured light coding pattern can be a standard geometric topology pattern, such as... The square grid pattern or other coded feature pattern; simply ensure that each feature point in the structured light coded pattern has a unique number; Acquiring an infrared image, which is the image formed by the structured light coded pattern mapped onto the surface of the medium; in this step, an infrared image can be acquired by a camera. Because the grid pattern projector emits invisible infrared lasers outward, these lasers are encoded to form a structured light coded pattern. When the structured light coded pattern is projected onto the surface of the medium (wall or object), because the surface of the medium has unevenness, this infrared light pattern will be distorted and deformed according to the shape of the object. The distorted and deformed infrared light pattern will be acquired by the camera, thereby obtaining the infrared image in this step; Feature points are extracted from the infrared image, and the corresponding two-dimensional pixel coordinates are obtained. It should be noted in this step that feature points are the intersections of various deformation patterns in the infrared image; for example, a structured light coded pattern... For a square grid pattern, the feature point is the grid point where the horizontal and vertical lines intersect; for a structured light coding pattern that is a dot matrix / speckle pattern, the feature point is the centroid of the light spot; for a structured light coding pattern that is a dot pattern, the feature point is the center point of the dot / ellipse; in short, the feature point is the smallest unit in the structured light coding pattern that is conspicuous, can be reliably detected, and can be assigned a unique identity; generally speaking, the feature point is the intersection of various deformation patterns. The two-dimensional pixel coordinates are transformed into the actual physical depth information of feature points using a multi-dimensional spatial depth mapping dataset. This multi-dimensional spatial depth mapping dataset establishes a one-to-one mapping relationship between the image pixel coordinates and the actual physical depth of each known structured light coded pattern's feature points. In this step, the known feature points of the structured light coded pattern refer to the grid intersections whose topological identities were preset by the coded pattern in the first step and whose real-time pixel coordinates were successfully extracted from the infrared image in the third step. For example, the structured light coded pattern is... The square grid pattern, where the feature points of each known structured light coded pattern are the grid points where horizontal and vertical lines intersect.

[0021] Meanwhile, in this step, the multidimensional spatial depth mapping dataset is a physical calibration database, which means that when working online, the system does not need to perform any cumbersome distortion correction or triangulation mathematical calculations. It only needs to input the two-dimensional pixel coordinates of the feature point, and it can directly translate the high-precision absolute depth of the point in the camera coordinate system by directly looking up the table and interpolating.

[0022] Therefore, in this embodiment, considering that the structured light coded pattern has a known and fixed topology, the identity of each feature point in the projected pattern (i.e., its position in the coded sequence) is uniquely determined. Thus, when imaging on any medium surface, the pixel coordinate offset of this feature point is determined only by the physical depth of its location. Therefore, by setting up a multi-dimensional spatial depth mapping dataset, the complex nonlinear mapping relationship between "pattern feature point—pixel coordinate—actual depth" in the structured light system is pre-fixed into a set of discrete spatial lookup tables using an offline calibration method. This eliminates the need for real-time geometric reconstruction or matching calculations, reducing the algorithm's dependence on processor computing power and memory bandwidth. This allows the entire environmental perception calibration process to run stably on low-computing-power, low-power embedded chip platforms. Simultaneously, it avoids binocular matching ambiguity caused by factors such as ambient lighting and texture repetition, ensuring measurement accuracy.

[0023] In one specific implementation, converting the two-dimensional pixel coordinates into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset specifically includes: The physical feature intersection index of the two-dimensional pixel coordinates is determined. The physical feature intersection index is a unique number preset for the feature point in the structured light coding pattern. The physical feature intersection index is explained as follows: the projected structured light coding pattern is, for example, an M-row × K-column grid pattern. The row and column intersection points have been predefined: for example, the first intersection point in the upper left corner is coded as j=1, its right neighbor is j=2, and so on. The last row and last column is j=M*K. This j is the physical feature intersection index, which represents "which point in the physical light spot emitted by the projector is this point". At the same time, it remains unchanged throughout the entire life cycle of the system. This step uses the physical feature intersection index to directly lock the target identity, so that the subsequent sampling point search is strictly constrained to the known depth coordinate trajectory of the feature point, eliminating the risk of mismatch from the source of the algorithm and greatly reducing the data retrieval range and improving efficiency. By using a multidimensional spatial depth mapping dataset, we can find two adjacent consecutive calibration sampling points in the coordinate trajectory of the physical feature intersection index where the two-dimensional pixel coordinates fall, and obtain their corresponding calibration depth and corresponding calibration pixel coordinates. Finding two adjacent consecutive calibration sampling points in a single trajectory does not require building and maintaining a complex high-dimensional spatial index structure, so that the data access efficiency remains highly stable on low computing power platforms. Calculate the Euclidean distance between the two-dimensional pixel coordinates and the calibration pixel coordinates corresponding to the two adjacent consecutive calibration sampling points; Using the inverse Euclidean distance as a weighting coefficient, linear interpolation is performed on the calibration depth of two adjacent consecutive calibration sampling points to obtain the actual physical depth corresponding to the two-dimensional pixel coordinates. By using the inverse Euclidean distance as a weighting coefficient for linear interpolation, a smooth and monotonic depth transition is constructed between adjacent calibration sampling points. This allows for the recovery of sub-sampling point level continuous depth resolution with only a minimal number of multiply-accumulate and square-root operations, ensuring the fineness and accuracy of physical measurements. While this embodiment uses linear interpolation with the inverse Euclidean distance, in practical applications, the neighborhood interpolation can be extended to employ bicubic interpolation, spline interpolation, or polynomial approximation algorithms.

[0024] To further understand this transformation step, the following detailed explanation is provided: For any set of extracted real-time pixel coordinates First, identify that it belongs to the first The nth feature point is then retrieved from the pre-stored offline calibration dataset to obtain the nth feature point. Trajectory of a dataset of feature points: Find the adjacent calibration sampling intervals that the real-time pixel falls into in the dataset trajectory, that is, find two adjacent calibration depths that meet the conditions. and The calibration pixel coordinates corresponding to its two calibration depths are as follows: and .

[0025] Its high-precision continuous depth value is calculated using a linear interpolation algorithm based on the inverse Euclidean distance ratio. The Euclidean distance between the real-time point and its two neighboring continuous calibration points is calculated as the basis for the weighting coefficients.

[0026]

[0027] Depth value The solution formula is:

[0028] That is, the closer the real-time pixel coordinates are to a trajectory point at a certain calibration depth, the larger the weighting coefficient of that calibration depth.

[0029] In one specific implementation, such as Figure 2 As shown, the multidimensional spatial depth mapping dataset is obtained offline: The medium is driven to stop sequentially at N discrete calibrated distance sampling points within a set working depth range, from near to far. Specifically, the medium is placed on a linear displacement stage, and the movement direction of the linear displacement stage is parallel to the optical axis of the infrared image acquisition device. The medium is a target plate, and the plane of the target plate is perpendicular to the optical axis of the infrared image acquisition device. Collect and extract the two-dimensional pixel coordinates of feature points in infrared images at each calibrated distance sampling point; The two-dimensional pixel coordinates of each feature point are encoded and associated with the absolute depth value corresponding to the feature point to obtain a multi-dimensional spatial depth mapping dataset.

[0030] Specifically, in this embodiment, the multidimensional spatial depth mapping dataset can be obtained offline in the following way: 1.1 Calibration Environment and Equipment Setup: The rigidly fixed "monocular RGB-IR camera + structured light projection module" system is secured to one end of the optical calibration stage. A linear guide rail (i.e., a linear displacement stage) driven by a high-precision servo motor is positioned directly in front of the system, with the guide rail's movement direction parallel to the camera's optical axis. Axis. A highly flat, standard isotropic diffuse reflection target plate (such as a high-aluminate white board) is vertically mounted on the sliding slider of the guide rail, ensuring that the target plate plane is aligned with the camera's optical axis. Strict verticality. This step also includes introducing a parallelism error compensation mechanism: determining the installation angle between the actual motion direction of the linear displacement stage and the camera optical axis; and using geometric projection relationships to geometrically decouple and compensate for the actual absolute depth values ​​of each sampling point when constructing a multidimensional spatial depth mapping dataset.

[0031] Specifically, the parallelism error compensation mechanism can be exemplified as follows: In the offline calibration initialization stage, a high-precision laser interferometer is first used to determine the actual motion vector direction of the linear displacement stage and the optical axis of the camera. There is a small installation angle θ between the axes (e.g., θ = 0.05° as measured). When constructing the multidimensional spatial depth mapping dataset, a geometric projection correction model is introduced: for a calibration sampling point that should theoretically be at distance Z, its actual spatial coordinates are offset due to the installation angle θ. This is addressed using a geometric decoupling formula. Geometric decoupling and compensation correction are performed on the actual absolute depth values ​​at each sampling point. Through this projection correction, the cumulative projection error in the depth direction caused by the non-strict parallelism between the displacement axis and the optical axis due to assembly (for example, reducing the ranging deviation by about 2 mm at 5 meters) is mapped into a lookup table, thereby achieving implicit compensation for mechanical installation errors.

[0032] The parallelism error compensation mechanism is introduced to take into account the actual movement direction of the linear guide rail and the camera optical axis during actual production line setup. The shaft may have an extremely small installation angle θ. In this embodiment, during offline initialization, the small angle θ is measured by a laser interferometer. When writing data points later, the geometric projection relationship is used to perform geometric decoupling and compensation correction on the actual absolute depth value of the sampling point position, thereby eliminating the linear cumulative error caused by hardware misalignment and reducing the mechanical rigidity threshold of production line construction.

[0033] 1.2 Multi-stage distance propulsion and dynamic data acquisition Set the system's operating depth range as follows (For vehicle-mounted near-field sensing, the setting can be from 0.2m to 5m). Set within this range. A set of discrete calibration distance sampling points in ascending order:

[0034] Meanwhile, to ensure consistent depth resolution across the entire field of view, in areas with large nonlinear distortion of the lens (such as edge regions where the FoV exceeds 70% of the entire field of view, and the absolute value of the distortion rate slope is greater than a preset threshold), this embodiment employs a non-uniform densification sampling strategy to eliminate the influence of severe nonlinear deformation at the edges. Specifically, the interval between the calibration distance sampling points in this region is increased from the central region Δ... (e.g., 20cm) Narrow and densify sampling to (e.g., 5cm).

[0035] The control displacement stage moves the target plate from near to far, stopping sequentially at each designated calibration distance. At each location, perform the following operations: 1) Activate the structured light projection module and project a known standard geometric topology (such as...) onto the target plate. Infrared patterns (square grid patterns or other coded feature patterns).

[0036] 2) Control the monocular RGB-IR camera to perform exposure, acquire composite spectral images, and extract pure, high-contrast infrared channel images through its hardware color filter array (CFA). .

[0037] 1.3 Subpixel-level pattern feature point extraction and topology tracking: Infrared images at each sampling distance : 1) Employ a feature detection operator based on the Hessian matrix or a sub-pixel edge intersection extraction algorithm to capture feature intersections in the pattern. For the th The first order distance For each feature point, obtain its high-precision sub-pixel two-dimensional coordinates in the image coordinate system:

[0038] 2) Run the topology tracing algorithm: based on the inherent geometric index order of the pattern (e.g., row numbers) , column number The extracted feature points are sorted and uniformly numbered to lock a series of discrete sub-pixel coordinate points of the same physical feature point at different calibrated distances, thus forming the mapping trajectory of the feature point as the distance changes.

[0039] 1.4 Construction of Multidimensional Spatial Depth Mapping Dataset (LUT): The absolute pixel coordinates of each feature node at each known depth and the corresponding... The values ​​are written to the system's non-volatile memory, forming a high-density three-dimensional control mesh lookup table with full field-of-view (FoV) coverage. The storage mapping logic is as follows:

[0040] This method implicitly incorporates the severe nonlinear distortion caused by the lens and mechanical assembly errors into the data mapping, eliminating the need for explicit distortion correction calculations during online solution.

[0041] Therefore, in this embodiment, the acquisition of the multidimensional spatial depth mapping dataset is achieved because the calibration method does not require the absolute parallel installation of the displacement stage and the camera optical axis, and does not require high-precision separate calibration of lens distortion. This significantly reduces the stringent precision requirements on the production line assembly process, thereby enhancing the robustness of the calibration algorithm to incoming material tolerances and assembly fluctuations, and effectively ensuring yield and performance stability during large-scale mass production. Simultaneously, offline calibration can be pre-completed as an independent workstation on the production line, and the generated standard dataset can be directly burned or solidified into the storage unit of each device, greatly simplifying the final inspection process on the production line and ensuring the consistency and traceability of depth perception performance across the same batch of products.

[0042] In one specific implementation, when extracting the two-dimensional pixel coordinates of feature points, a topology tracking algorithm is used to uniformly number the feature points according to the topological geometric index order. This locks a sequence of discrete sub-pixel coordinates of the same physical feature intersection at different calibration distances, obtaining the mapping trajectory of the feature point as a function of distance, i.e., the coordinate trajectory of the physical feature intersection index. This embodiment utilizes the topology tracking algorithm to perform numbering based on the inherent geometric adjacency relationships of the projection pattern (such as grid row and column order, connected component adjacency relationships), ensuring that each index number strictly follows the inherent order of the physical projection pattern, thereby eliminating ambiguity in neighboring point matching and providing higher accuracy for subsequent depth calculation.

[0043] In one specific implementation, after obtaining the two-dimensional pixel coordinates corresponding to the feature points in the infrared image, a dynamic anti-misalignment topology correction mechanism is applied to the two-dimensional pixel coordinates of the feature points before conversion. Specifically, considering the pattern deformation and breakage problems caused by sudden tall obstacles, terrain step edges, or severe local soil occlusion during the online sensing stage, this embodiment introduces a spatially adjacent node topological geometric constraint and time-series displacement prediction mechanism when extracting the feature coordinates of the current frame. Since the projected pattern has a globally unique topological index, when a local pattern breaks or is partially occluded, the expected pixel position range of the broken edge node is deduced backward from the continuous known nodes in the unbroken area. If the feature extraction algorithm captures a feature point within this range, displacement prediction is performed by combining the motion vector of the node in the previous frame of the time series. Unique encoding verification and online topology repair are performed on nodes that may be misaligned, effectively avoiding depth misjudgment caused by mis-finding neighbors.

[0044] Specifically, the topological geometric constraints of spatially adjacent nodes and the time-series displacement prediction mechanism can be illustrated by the following example: 1. Application example of the mechanism: Suppose the current image is the first Feature point (whose topological neighbors are) and Feature point extraction failed due to partial soil obstruction. The system was restarted to fix the issue as follows: Spatial constraints: The system extracts adjacent nodes. The pixel coordinates are ,node The pixel coordinates are Based on the known geometric adjacency relationships of the topology, the system is located at its theoretical position. Establish the expected pixel search range in the vicinity.

[0045] Time prediction: Retrieve the previous frame ( (Time) The motion vector of node number is Based on the global motion trend of this region in the current frame, predict... The theoretical coordinates of node number in the current frame are: .

[0046] Verification and Repair: The system finds high-contrast feature points within the prediction area, verifies their uniqueness through unique coding, and confirms their identity. The node is used as the valid coordinate input into the depth calculation process.

[0047] 2. Implementation Results: This mechanism effectively avoids node coordinate loss or mismatch due to occlusion. Test data shows that before introducing this mechanism, the node mismatch rate in complex dynamic occlusion scenarios was approximately 5%, leading to "jumps" or "holes" in depth perception. After introducing this mechanism, the success rate of repairing breaks in a single node within three consecutive frames exceeds 98%, and the spatial smoothness index of depth measurement is improved by approximately 15%, ensuring the continuity and robustness of depth perception output during robot obstacle avoidance. In one specific implementation, such as Figure 1 As shown, a structured light coded pattern with a preset known topology is projected onto the space to be tested through a structured light projection module; an infrared image is acquired through a monocular RGB-IR camera; and the monocular RGB-IR camera is fixedly connected to the structured light projection module.

[0048] Specifically, in this embodiment, the structured light projection module projects a near-infrared structured light coded pattern into the unknown environmental space. When this pattern encounters the object to be measured or the terrain surface, its geometric topology deforms (modulates) with the three-dimensional undulations of the object. A monocular RGB-IR camera captures images using pulse-synchronized exposure and separates the original composite image into an RGB color image stream and an IR infrared image stream with pixel-level alignment.

[0049] Furthermore, the monocular RGB-IR camera and the structured light projection module are integrated and fixed together using a high-rigidity automotive-grade metal bracket. The optical center of the monocular RGB-IR camera is set as the origin of the camera coordinate system. Its optical axis is The structured light projection module's emission optical center is... The baseline distance (shortest distance) between its emission axis and the camera optical axis in space is: The relative angle between the two axes in space is Due to the use of rigid constraints, the baseline distance is [not specified] after the system assembly is completed. and included angle The spatial relative pose relationship is locked and serves as a fixed prior geometric constant for the system.

[0050] This embodiment utilizes the hardware color separation characteristics of the RGB-IR camera. The infrared (IR) channel is specifically used to extract pattern feature points with high contrast that are not affected by ambient color or extreme lighting. At the same time, the RGB channel can synchronously acquire pure ambient visible light textures, achieving time-difference-free and coaxial "color + depth" information fusion, which has extremely high environmental adaptability in smart agriculture (soil, shading, fruit ripeness judgment) or vehicle reversing conditions.

[0051] In one specific implementation, after obtaining the actual physical depth information of the feature points, the method further includes converting the actual physical depth information of the feature points into the three-dimensional physical coordinates of the feature points in the infrared image under the coordinates of a monocular RGB-IR camera. Using the edge or color gradient texture in the RGB color image output by a monocular RGB-IR camera as guiding weights, the three-dimensional physical coordinates of feature points in the infrared image are used to generate the overall three-dimensional point cloud data of the infrared image.

[0052] In this embodiment, after obtaining the actual physical depth information of the feature points, it is aligned and fused with the RGB color image to output multimodal perception data, as follows: Combined with the intrinsic principal distance of a monocular camera and optical center coordinates , two-dimensional pixel coordinates Convert to 3D physical coordinates in the camera coordinate system:

[0053]

[0054]

[0055] It's also worth noting that after obtaining the 3D physical coordinates, since these are only the 3D physical coordinates of feature points, what is obtained is a sparse 3D point cloud at the feature intersections. After obtaining the sparse 3D point cloud at the feature intersections, an image texture-guided depth densification algorithm is further initiated. Using the edges and color gradient texture of a clean coaxial RGB image as guiding weights, secondary region interpolation is performed on the blank areas inside the pattern that are not covered by light points, thereby generating a high-density dense 3D point cloud from the sparse point cloud, and finally outputting the result.

[0056] Once a dense 3D point cloud is obtained, it can be applied to the following aspects: In-vehicle applications (such as reversing / near-field collision avoidance): The calculated 3D point cloud spatial distance is used to detect the obstacle boundaries in the field of view in real time, calculate the minimum safe distance of the obstacle, and output the obstacle bounding box or trigger an alarm signal.

[0057] Agricultural robot applications (such as grading and harvesting): Dense three-dimensional physical coordinates are fused with pixel-aligned RGB color images. Multispectral visual information from the RGB channels is used to distinguish crops from weeds or determine fruit maturity. Simultaneously, aligned depth information provides accurate 3D positioning, outputting 3D color point cloud data with spectral classification attributes to guide the robotic arm in intelligent and precise operations.

[0058] A system for implementing the environment perception calibration method based on image acquisition and structured light projection as described in any of the above embodiments includes: Structured light projection module: used to project a structured light coded pattern with a preset known topology onto the space to be tested; Infrared image acquisition module: used to acquire infrared images, which are the images formed by mapping the structured light coding pattern onto the surface of the medium; The data processing module is used to extract feature points from infrared images and obtain the corresponding two-dimensional pixel coordinates of the feature points; and to convert the two-dimensional pixel coordinates into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset; wherein the multi-dimensional spatial depth mapping dataset establishes a one-to-one mapping relationship between the image pixel coordinates and the actual physical depth of each known structured light coded pattern feature point.

[0059] The system in this embodiment has a simple structure and achieves online operation without the need for online binocular matching or ray intersection analytical geometric calculations. By directly retrieving the pre-calibrated spatial mapping dataset and performing fast neighborhood interpolation, the algorithm's consumption of processor computing power is greatly reduced, perfectly meeting the mass production requirements of low-computing-power chip platforms. At the same time, it ensures the determinism and consistency of depth output, which is conducive to reducing system response latency and improving overall robustness.

[0060] To further understand this application, as Figure 3 As shown, when the system is in a vehicle (e.g., reversing) or operating as a smart agricultural robot, it performs the following operations: Step 1: The structured light projection module projects a near-infrared structured light coded pattern into the unknown environment. When this pattern encounters the object or terrain surface being measured, its geometry deforms (modulates) with the object's three-dimensional undulations. A monocular RGB-IR camera captures images using pulse-synchronized exposure and separates the original composite image into pixel-aligned RGB color image streams and IR infrared image streams. Step 2: Efficiently extract the real-time two-dimensional pixel coordinates of the intersection points of each pattern feature in the current frame image from the real-time separated IR infrared image. ; Step 3: For any set of extracted real-time pixel coordinates The data processing module first identifies that it belongs to the first... The system calculates the actual physical depth information of each feature point by retrieving a pre-stored offline calibration dataset and then performing neighborhood interpolation depth calculation based on the dataset. Step 4: Multimodal sensing data and application layer output Combined with the intrinsic principal distance of a monocular camera and optical center coordinates , two-dimensional pixel coordinates The data is converted to three-dimensional physical coordinates in the camera coordinate system. Based on the calculated three-dimensional point cloud spatial distance, the data processing module detects the obstacle boundaries in the field of view in real time, calculates the minimum safe distance of the obstacle, and outputs the obstacle bounding box or triggers an alarm signal.

[0061] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

Claims

1. An environmental perception calibration method based on image acquisition and structured light projection, characterized in that: Includes the following steps: Project a structured light coded pattern with a pre-defined known topology onto the space to be tested; Acquire an infrared image, which is the image formed by mapping the structured light coded pattern onto the surface of the medium; Feature points are extracted from infrared images, and the corresponding two-dimensional pixel coordinates are obtained. The two-dimensional pixel coordinates are converted into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset. The multidimensional spatial depth mapping dataset establishes a one-to-one mapping relationship between the image pixel coordinates and the actual physical depth for each known structured light coded pattern feature point.

2. The environmental perception calibration method based on image acquisition and structured light projection according to claim 1, characterized in that: The process of converting the two-dimensional pixel coordinates into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset specifically includes: Determine the physical feature intersection index of the two-dimensional pixel coordinates; the physical feature intersection index is a pre-set unique number of the feature point in the structured light coding pattern; The multidimensional spatial depth mapping dataset is used to find two adjacent consecutive calibration sampling points in the coordinate trajectory of the physical feature intersection index where the two-dimensional pixel coordinates fall, and the corresponding calibration depth and the corresponding calibration pixel coordinates are obtained. Calculate the Euclidean distance between the two-dimensional pixel coordinates and the calibration pixel coordinates corresponding to the consecutive calibration sampling points in the two adjacent stages; Using the inverse Euclidean distance as a weighting coefficient, linear interpolation is performed on the calibration depth of two adjacent consecutive calibration sampling points to obtain the actual physical depth corresponding to the two-dimensional pixel coordinates.

3. The environmental perception calibration method based on image acquisition and structured light projection according to claim 1 or 2, characterized in that: The multidimensional spatial depth mapping dataset was obtained offline. The medium is driven to stop sequentially at N discrete calibration distance sampling points from near to far within the set working depth range; Collect and extract the two-dimensional pixel coordinates of feature points in infrared images at each calibrated distance sampling point; The two-dimensional pixel coordinates of each feature point are encoded and associated with the absolute depth value corresponding to the feature point to obtain a multi-dimensional spatial depth mapping dataset.

4. The environmental perception calibration method based on image acquisition and structured light projection according to claim 3, characterized in that: The medium is set on a linear displacement stage, and the direction of movement of the linear displacement stage is parallel to the optical axis of the infrared image acquisition device; the medium is a target plate, and the plane of the target plate is perpendicular to the optical axis of the infrared image acquisition device.

5. The environmental perception calibration method based on image acquisition and structured light projection according to claim 3, characterized in that: When acquiring multidimensional spatial depth mapping datasets offline, a parallelism error compensation mechanism is also introduced. Determine the installation angle between the actual motion direction of the linear displacement stage and the optical axis of the camera; When constructing a multidimensional spatial depth mapping dataset, geometric projection relationships are used to perform geometric decoupling and compensation correction on the actual absolute depth values ​​of each sampling point.

6. The environmental perception calibration method based on image acquisition and structured light projection according to claim 4 or 5, characterized in that: When extracting the two-dimensional pixel coordinates of feature points, a topology tracking algorithm is used to uniformly number the feature points according to the topological geometric index order, thereby obtaining the mapping trajectory of the feature points as the distance changes.

7. The environmental perception calibration method based on image acquisition and structured light projection according to claim 1, characterized in that: After obtaining the two-dimensional pixel coordinates corresponding to the feature points in the infrared image, a dynamic anti-misalignment topology correction mechanism is performed on the two-dimensional pixel coordinates of the feature points before conversion.

8. The environmental perception calibration method based on image acquisition and structured light projection according to claim 1, characterized in that: A structured light coded pattern with a preset known topology is projected onto the space to be tested using a structured light projection module; infrared images are acquired using a monocular RGB-IR camera; and the monocular RGB-IR camera is fixedly connected to the structured light projection module.

9. The environmental perception calibration method based on image acquisition and structured light projection according to claim 1, characterized in that: After obtaining the actual physical depth information of the feature points, the process also includes converting the actual physical depth information of the feature points into the three-dimensional physical coordinates of the feature points in the infrared image under the coordinates of a monocular RGB-IR camera. Using the edge or color gradient texture in the RGB color image output by a monocular RGB-IR camera as guiding weights, the three-dimensional physical coordinates of feature points in the infrared image are used to generate the overall three-dimensional point cloud data of the infrared image.

10. A system for implementing the environmental perception calibration method based on image acquisition and structured light projection as described in any one of claims 1-9, characterized in that: include: Structured light projection module: used to project a structured light coded pattern with a preset known topology onto the space to be tested; Infrared image acquisition module: used to acquire infrared images, which are the images formed by mapping the structured light coding pattern onto the surface of the medium; Data processing module: used to extract feature points from infrared images, obtain the two-dimensional pixel coordinates corresponding to the feature points; and convert the two-dimensional pixel coordinates into the actual physical depth information of the feature points through a multi-dimensional spatial depth mapping dataset; The multidimensional spatial depth mapping dataset establishes a one-to-one mapping relationship between the image pixel coordinates and the actual physical depth for each known structured light coded pattern feature point.

Images