Environment perception detection method and system based on beam shaping and visual fusion
By embedding planar geometric feature patterns into an infrared grating grid and combining it with the U2net deep learning model, the accuracy and robustness issues of existing environmental perception technologies in complex environments are solved, achieving high-precision 3D obstacle localization and supporting real-time perception for intelligent driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-07
AI Technical Summary
Existing environmental perception technologies lack accuracy and robustness in complex environments, especially in nighttime, low-light, and featureless environments, making it difficult to achieve high-precision target detection and 3D positioning.
By employing a method based on beam shaping and visual fusion, uniquely identifiable planar geometric feature patterns are embedded in an infrared grating grid. The U2net deep learning model is then used for multi-class instance segmentation and feature point extraction to establish a 2D-3D mapping and achieve 3D localization of obstacles.
It improves the accuracy and robustness of environmental perception, enabling centimeter-level obstacle localization in complex scenarios, and meets the real-time and reliability requirements of advanced driver assistance systems.
Smart Images
Figure CN121811367A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of environmental perception technology, and more specifically, relates to an environmental perception detection method and system based on beam shaping and visual fusion. Background Technology
[0002] With the rapid development of intelligent driving and assisted driving technologies, environmental perception, as a core component, is becoming increasingly important. Currently, mainstream environmental perception technologies mainly include: I. Vision Perception: Utilizing monocular or multi-view cameras, deep learning algorithms are used to identify roads, vehicles, pedestrians, traffic signs, etc., and to perform target detection and classification. However, visual perception is highly susceptible to ambient lighting conditions (such as nighttime, backlighting, tunnels) and weather conditions (such as rain, fog), and it cannot directly obtain accurate 3D depth information of the target. Its 3D reconstruction relies on complex binocular vision or structure-of-motion (SfM) algorithms, which are computationally intensive and have limited accuracy.
[0003] II. LiDAR (Light Detection and Ranging): LiDAR generates high-precision point cloud data by emitting laser beams and measuring the time it takes for them to reflect back, thus enabling centimeter-level 3D modeling of the environment. Although LiDAR provides extremely high spatial accuracy, it is expensive and susceptible to severe weather (such as snow and heavy rain). At the same time, the sparsity of point cloud data poses a challenge to the detection of small targets and fine-grained recognition.
[0004] 3. Millimeter-wave radar (MMW Radar): It has strong penetration capability and all-weather operation capability, and can accurately measure the target's distance, velocity, and azimuth. However, its angular resolution is low, making it difficult to identify and distinguish multiple small targets at close range, and it cannot provide detailed geometric information about the target.
[0005] IV. Infrared Gratings: Infrared gratings with specific patterns are projected, and their reflected images in the environment are captured by a camera. The three-dimensional spatial position of the target is calculated by extracting and processing the refraction information of the grating lines on obstacles. Due to the unique advantages of infrared gratings in near-field environmental perception, they have become a mainstream trend. However, existing infrared grating sensing technologies still have the following problems: 1. Single Feature Dependency: Most existing technologies rely on sub-pixel extraction of grating lines, using deep learning to establish a 2D-to-3D mapping model through the intersections or parallel lines between the lines. This method has extremely high requirements for the integrity and continuity of the grating lines. Once the lines are interrupted or deformed due to obstacles, ground reflections, or environmental noise, it is very easy to cause the intersections or matching to fail, thus causing the 3D solution to fail.
[0006] 2. Line number matching problem: In complex environments, especially when there are multiple obstacles blocking the view, the segmented raster lines may become disconnected from the original line sequence, making it impossible to accurately match with the known DOE template. Consequently, it is impossible to determine the true line number, which seriously affects the accuracy and robustness of 3D reconstruction.
[0007] 3. Insufficient model generalization ability: Traditional image processing methods, such as Hough transform and skeleton extraction, can effectively extract lines, but their robustness is limited by algorithm parameters and image quality, making it difficult to adapt to complex scenes with different lighting, different surface materials, and different obstacle shapes. Summary of the Invention
[0008] 1. The problem to be solved To address the issues of poor accuracy and robustness in existing environmental perception methods, this invention provides an environmental perception and detection method and system based on beam shaping and visual fusion. The method overcomes the limitations of single feature dependence and line number matching in existing technologies, achieving high-precision, omnidirectional three-dimensional spatial localization of obstacles in complex environments, significantly improving the system's robustness, real-time performance, and generalization ability. The system of this invention meets the real-time and reliability requirements of Advanced Driver Assistance Systems (ADAS), providing robust perception support for intelligent driving.
[0009] 2. Technical Solution To solve the above problems, the present invention adopts the following technical solution.
[0010] An environment perception and detection method based on beam shaping and visual fusion includes the following steps: S1: Obtain the target image: The target image is a grating image with a planar geometric feature pattern; wherein the planar geometric feature pattern is used to locate and match the sequence number of the grating lines in the target image; S2: Multi-class instance segmentation: Input the target image into the U2net deep learning model to perform pixel-level instance segmentation of multiple classes of objects in the target image, and obtain the first image corresponding to each class of objects; S3: 3D Reconstruction: Extract feature points from the first image of each type of object, and establish a 2D-3D mapping based on the position of the feature points and planar geometric feature patterns in the target image; based on the 2D-3D mapping relationship, perform sequence matching and 3D spatial positioning on each grating line in the first image to determine the 3D coordinates of the obstacle.
[0011] By employing the above technical solution, the problem of matching grating line numbers is solved from a physical level by embedding uniquely identifiable planar geometric feature patterns into the grating grid. The U2net deep learning model enables pixel-level instance segmentation of multi-category targets. Compared to skeleton-level feature extraction, pixel-level instance segmentation overcomes the strong dependence on line integrity in existing methods, robustly handling grating interruptions and occlusions, and significantly enhancing overall robustness under complex conditions such as occlusion and interruptions. Through 3D reconstruction based on multiple feature points, the number of each grating line can be accurately determined, and its position in 3D space can be calculated, ultimately achieving centimeter-level localization of obstacles. The entire method effectively addresses the problems of insufficient target detection and 3D localization accuracy and poor robustness in existing vehicle perception solutions in complex and variable scenarios, especially at night, in low light, and in featureless environments.
[0012] Furthermore, step S1, acquiring the target image, specifically includes: S11: Obtain the original image: Based on the laser emitter and diffractive optical elements, an infrared grating is projected toward the sensing area, and the image projected by the infrared grating is obtained based on the camera module to obtain the original image; wherein the diffractive optical elements project a planar geometric feature pattern with a predefined arrangement at a specific position of the grid; the planar geometric feature pattern is a triangular pattern and a circular pattern, and the triangular pattern and the circular pattern have a known and precise relative position in the physical world; S12: Extract the infrared grating image from the original image, and crop and grayscale the infrared grating image to obtain the target image.
[0013] Using the above technical solution, diffractive optical elements project triangular and circular patterns at specific positions on the grating grid. These patterns have known and precise relative positions in the physical world, enabling each grating line to be uniquely identified and matched with adjacent feature patterns. This physically solves the matching ambiguity problem caused by line repetition in traditional grating projection, providing a reliable basis for subsequent 3D reconstruction. At the same time, by separating the infrared grating image from the original image, visible light background interference can be effectively removed, focusing on the structured grating information and reducing data redundancy. Cropping can remove invalid areas at the image edges, allowing for centralized processing of regions of interest and reducing computational load. Grayscale conversion converts the image into a single channel, simplifying data dimensions while preserving the intensity information of the grating lines, which is beneficial for subsequent feature extraction and deep learning model processing.
[0014] Furthermore, step S12 includes obtaining the image of the separated infrared grating from the camera module through an image acquisition device; and cropping the image of the infrared grating by selecting a ROI region, and converting the cropped color image into a grayscale image.
[0015] Furthermore, the training process of the U2net deep learning model in step S2 is as follows: Define multiple categories of objects: Set up 6 categories of objects, namely: 0: background, 1: triangle, 2: line, 3: point, 4: objection, 5: circle; Acquiring training data: The training data is labeled data: the image containing infrared gratings and obstacles is labeled at the pixel level; The U2net network structure is adopted, and its loss function is set. The weighted cross-entropy loss function is used as the model's loss function, specifically:
[0016] Among them, y i p is the true category label of i pixels. yi w is the probability that the model predicts it to be of that category. yi This represents the weight of the category; N is the total number of pixels. Based on the training data, the U2net network structure is used for training to obtain the final U2net deep learning model.
[0017] By employing the aforementioned technical solution, the lightweight U2net network, with its unique U-shaped structure and RSU (ReSidual U-blocks) module, excels in capturing multi-scale contextual information. The design of the RSU module enables it to effectively extract features at different scales without increasing computational cost, which is crucial for segmenting objects of different sizes, such as raster lines, intersections, triangles, and circles. Simultaneously, a weighted cross-entropy loss function is introduced during training to address the imbalance problem between different object categories. The category weights can be dynamically adjusted based on the number of pixels in each category in the dataset, assigning higher weights to smaller categories to guide the model to better learn these key features and ensure the accuracy of the model in subsequent use.
[0018] Furthermore, when the target image is input into the U2net deep learning model, the U2net deep learning model transforms the target image into a probability map with 6 channels, and then performs post-processing on each probability map to obtain the mask and mask image for each category; where each channel corresponds to a category object.
[0019] By adopting the above technical solution, the mask of each category contains pixel-level information of each instance, providing robust underlying data for subsequent 3D reconstruction.
[0020] Furthermore, step S3 specifically includes the following steps: S31: Feature point extraction: Extract the set of pixel coordinates for each raster line and the center pixel coordinates for each triangle and circle from the mask of each category; S32: Template matching: Establish a one-to-one object relationship between 2D points in the target image and 3D points in the diffractive optical element template based on the diffractive optical element template, and obtain the mapping between 2D and 3D; the diffractive optical element template includes the position information and arrangement sequence of triangular patterns and circular patterns; S33: Line number matching and 3D reconstruction: The grating lines segmented by the U2net deep learning model are reconstructed in 3D to obtain the grating mesh; S34: 3D obstacle localization: Extract pixel contours from object category instances segmented by the U2net deep learning model; match the pixel set of the pixel contours with the raster mesh to calculate the position, size and orientation of the obstacle in 3D space, thus achieving 3D obstacle localization.
[0021] Using the above technical solution, feature point extraction is based on pixel-level instance segmentation masks output by U2Net. This ensures that the extracted feature points completely belong to the target object, greatly eliminating interference from background noise, shadows, reflections, etc., resulting in high accuracy. An accurate and unique 2D-3D point pair correspondence is established through diffractive optical element templates, providing an absolute reference and solving the fundamental problem of sequence number matching. With known template information, the sequence number of each grating line in the image and its equation in three-dimensional space can be uniquely determined, achieving high-precision 3D scene reconstruction. By using the reconstructed, high-precision grating mesh as a scale, the segmented 2D obstacles are directly upgraded and measured, achieving low-cost, high-precision visual 3D perception.
[0022] Furthermore, step S33 specifically includes: Skeletonization or central axis extraction is performed on all line category instances segmented by the U2net deep learning model to obtain the pixel set of the line; For each segmented triangle and circle category instance, calculate its center point or feature corner point, project its 2D image coordinates onto the 3D plane where the diffractive optical element is located, and obtain its 3D spatial coordinates. By using the known arrangement rules of the diffractive optical element templates, the true serial numbers of these 3D spatial coordinates are determined, thereby establishing the correspondence between the 2D segmented features and the 3D templates; Using 3D spatial coordinates with real serial numbers as anchor points, the serial numbers of adjacent lines are matched. For each 2D line with a defined sequence number, the 3D coordinates of any pixel on the 2D line are calculated using the camera projection model in the coordinate system of the diffractive optical element.
[0023] By employing the above technical solution, the integrity of the lines is ensured and the influence of width variations in the original raster lines caused by uneven lighting or noise on positioning is avoided by skeletalizing or extracting the central axis of the line category instances. The center point or feature corner point is calculated for the segmented triangle and circle category instances. The adjacent 2D lines are matched by number using 3D feature anchor points with determined sequence numbers and positions. The principle is simple, reliable, and resistant to local interference, which greatly improves the robustness in partially occluded scenarios. The entire step S34 forms a data closed loop, ensuring consistency and meeting real-time requirements while maintaining high accuracy.
[0024] Furthermore, step S4 is included: Result Visualization: The 3D coordinate information of the reconstructed obstacle, along with the 3D model of the raster lines, is projected back into a 2D image or presented as a 3D point cloud. This visualization provides intuitive and understandable perceptual results, supporting real-time monitoring and decision-making.
[0025] A system for environmental perception and detection using the beam shaping and visual fusion-based method as described in any of the above technical solutions includes: Infrared grating projection module: used to project infrared gratings with planar geometric patterns into the environment; Image acquisition module: used to acquire target images; The computational processing module is used to perform multi-class instance segmentation and 3D reconstruction on the target image to obtain the 3D coordinates of the obstacles. Human-computer interaction module: used to display processing results and system status.
[0026] Using the above technical solution, the system solves the problem of matching grating line numbers from a physical level by introducing planar geometric feature patterns into the infrared grating projection module; it achieves multi-class instance segmentation, 3D reconstruction, and obstacle 3D localization through the computing processing module, realizing centimeter-level positioning of obstacles; the entire system has demonstrated excellent performance in real vehicle testing, especially in minimum obstacle detection and precision control, which can meet the real-time and reliability requirements of advanced driver assistance systems (ADAS), providing strong perception support for intelligent driving. Attached Figure Description
[0027] Figure 1 A schematic diagram of a diffractive optical element projecting a target onto a target. Figure 2 A schematic diagram of an image from which the infrared grating has been extracted from the original image; Figure 3 This is a schematic diagram showing the processed image of the infrared grating. Figure 4 A schematic diagram of the raster mesh image used to visualize the annotation results; Figure 5 This is a schematic diagram illustrating the visualization results of the binary detection. Figure 6 This is a schematic diagram of the detection results for the line sequence of the grating grid. Figure 7 This is a schematic diagram of the 3D reconstruction results (using LiDAR as the ground truth). Figure 8 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0028] The present invention will now be further described with reference to specific embodiments and accompanying drawings.
[0029] like Figure 8 As shown, an environment perception and detection method based on beam shaping and visual fusion includes the following steps: S1: Obtain the target image: The target image is a grating image with a planar geometric feature pattern; wherein the planar geometric feature pattern is used to locate and match the sequence number of the grating lines in the target image; Step S1 differs from traditional steps by embedding planar geometric feature patterns, such as triangles and circles, into specific positions within the grid lines instead of traditional parallel grid lines. Triangles and circles are characterized by their known relative positional relationships and precise geometric shapes in physical space, serving as unique identifiers. This provides absolute spatial anchor points for subsequent processing, fundamentally avoiding matching ambiguities caused by the periodic repetition of the grating, and enabling precise positioning and sequence matching of grating lines in the image.
[0030] S2: Multi-class instance segmentation: Input the target image into the U2net deep learning model to perform pixel-level instance segmentation of multiple classes of objects in the target image, and obtain the first image corresponding to each class of objects; using the U2net deep learning model to perform multi-class instance segmentation of the target image is more powerful and more stable than traditional skeleton extraction or simple binarization. In step S2, the U2Net deep learning model is selected. With its nested U-shaped structure, U2Net can capture multi-scale contextual information. The output is a multi-channel binary mask image with the same resolution as the input. Each channel corresponds to the accurate pixel-level segmentation result of a class of objects. In this embodiment, the objects can be divided into six categories: background, line, point, objection, triangle, and circle. This step performs pixel-level segmentation, which is different from traditional skeleton extraction. Skeleton extraction is extremely sensitive to the continuity of lines. Once the lines are interrupted due to occlusion, the skeleton breaks, and matching and reconstruction fail. This step can robustly handle raster interruption and occlusion, which significantly improves the robustness of the whole method under complex conditions such as occlusion and interruption.
[0031] S3: 3D Reconstruction: Extract feature points from the first image of each type of object, and establish a 2D-3D mapping based on the position of the feature points and planar geometric feature patterns in the target image; based on the 2D-3D mapping relationship, perform sequence matching and 3D spatial positioning on each grating line in the first image to determine the 3D coordinates of the obstacle.
[0032] Step S3 involves fusing all identifiable features (including line, point, triangle, circle, and object contours) segmented by the U2net deep learning model; then using feature points in the raster image with planar geometric patterns as reliable reference points to identify the line sequence; next, determining the position of the raster mesh in the camera coordinate system, i.e., solving the relative relationship between the camera and the DOE through the raster projection equation, thereby establishing a robust 2D to 3D mapping; then performing sequence matching and 3D spatial positioning on the raster mesh lines; and finally, accurately determining the 3D coordinates of the obstacle.
[0033] Step S3 specifically involves matching the extracted feature points with a known projection pattern template (which refers to the location of a planar geometric feature pattern) to uniquely determine the absolute sequence number of the grating lines in the local area. Then, by calculating the corresponding 3D world coordinates of each pixel on the 2D lines with determined sequence numbers, the 3D location of the obstacle is ultimately achieved. The accuracy of the matching in this step no longer relies on traditional local texture or continuity assumptions, but is based on global, unique physical features, resulting in extremely high reliability. It can accurately determine the sequence number of each grating line, thereby calculating its position in 3D space, ultimately achieving centimeter-level positioning of the obstacle.
[0034] In summary, this embodiment generates refractive information of specific infrared grating lines on obstacles and uses deep learning models and multi-feature fusion methods to extract and process this refractive information in real time, thereby achieving accurate three-dimensional perception of the environment and obstacles. This solves the problems of insufficient target detection and three-dimensional positioning accuracy and poor robustness of existing vehicle perception solutions in complex and variable scenarios, especially at night, in low light and featureless environments, and provides an all-weather and high-precision solution for environmental perception.
[0035] In one specific implementation, step S1, acquiring the target image, specifically includes: S11: Obtain the original image: Based on the laser emitter and diffractive optical elements, an infrared grating is projected toward the sensing area, and the image projected by the infrared grating is obtained based on the camera module to obtain the original image; wherein the diffractive optical elements project a planar geometric feature pattern with a predefined arrangement at a specific position of the grid; the planar geometric feature pattern is a triangular pattern and a circular pattern, and the triangular pattern and the circular pattern have a known and precise relative position in the physical world; S12: Extract the infrared grating image from the original image, and crop and grayscale the infrared grating image to obtain the target image.
[0036] In this embodiment, the image from which the infrared grating is extracted from the original image is called the preprocessed image. The preprocessed image is as follows: Figure 2 As shown, with a resolution of 1280*960 pixels, a series of preprocessing operations are required to optimize image quality and prepare for subsequent model inference. Specifically, since the raster grid is mainly distributed in the central part of the image, Region of Interest (ROI) cropping is performed on the preprocessed image from the region X=172, Y=200, W=932, H=700. This effectively removes redundant information, reduces computational load, and improves processing speed. The cropped color image is then converted to a grayscale image to better highlight the raster lines and patterns. The processed image is shown below. Figure 3 As shown.
[0037] Specifically, step S12 includes obtaining the image of the separated infrared grating from the camera module through the image acquisition device; and cropping the image of the infrared grating by selecting a ROI region, and converting the cropped color image into a grayscale image.
[0038] The image from which the infrared grating is separated from the original image, i.e., the preprocessed image, can effectively remove visible light background interference, focus on structured grating information, and reduce data redundancy; cropping can remove invalid areas at the image edges, concentrate on processing regions of interest, and reduce computational load; grayscale conversion converts the image into a single channel, simplifies data dimensions, and at the same time preserves the intensity information of the grating lines, which is beneficial for subsequent feature extraction and deep learning model processing.
[0039] In one specific implementation, the training process of the U2net deep learning model in step S2 is as follows: Define multiple categories of objects: Set up 6 categories of objects, namely: 0: background, 1: triangle, 2: line, 3: point, 4: objection, 5: circle; Acquiring training data: The training data is labeled data: the image containing infrared gratings and obstacles is labeled at the pixel level; The U2net network structure is adopted, and its loss function is set. The weighted cross-entropy loss function is used as the model's loss function, specifically:
[0040] Among them, y i p is the true category label of i pixels. yi w is the probability that the model predicts it to be of that category. yi This represents the weight of the category; N is the total number of pixels. Based on the training data, the U2net network structure is used for training to obtain the final U2net deep learning model.
[0041] Specifically, this embodiment selects U2net (U2netP, i.e., a lightweight network), a network specifically designed for salient object detection. Its unique U-shaped structure and RSU (ReSidual U-blocks) modules make it excellent at capturing multi-scale contextual information. The design of the RSU modules enables it to effectively extract features at different scales without increasing computational cost, which is crucial for segmenting targets of different sizes such as raster lines, intersections, triangles, and circles.
[0042] Traditional U2Net is typically used for binary classification (foreground / background). This embodiment extends it to a multi-class instance segmentation task. The training process is as follows: Six segmentation categories are defined: 0: background, 1: triangle, 2: line, 3: point, 4: objection, and 5: circle. LabelMe and other annotation tools are then used to perform pixel-level annotation on images containing infrared gratings and obstacles. For each image, based on the grating image with planar geometric features and the actual situation of the obstacles, the polygonal region for each category is accurately delineated. The visualization effect of the annotated image is as follows: Figure 4 As shown: background: Default background; triangle: Use the polygon tool to label each triangle; line: Use the polyline tool to mark each raster line; point: Use the control point tool to mark each intersection point; Objection: Use the polygon tool to label each obstacle; circle: Use the polygon tool to label each circle.
[0043] Meanwhile, to address the class imbalance issue during training (such as line and background having a larger proportion while triangle and circle have a smaller proportion), this embodiment introduces a weighted cross-entropy loss. The class weights in this loss function can be dynamically adjusted based on the number of pixels in each class in the dataset, giving higher weights to smaller classes to guide the model to better learn these key features and ensure the accuracy of the model in subsequent use.
[0044] In one specific implementation, when the target image is input into the U2net deep learning model, the U2net deep learning model transforms the target image into a probability map with six channels. Each probability map is then post-processed (e.g., thresholding) to obtain a mask for each category and a mask image. The mask contains pixel-level information for each instance, providing robust underlying data for subsequent 3D reconstruction. By binarizing the probability map of each channel, a mask image is obtained as follows: Figure 5 The resulting image is shown.
[0045] In one specific embodiment, step S3 specifically includes the following steps: S31: Feature point extraction: Extract the set of pixel coordinates for each raster line and the center pixel coordinates for each triangle and circle from the mask of each category; specifically, this step is as follows: extract the set of pixel coordinates for each raster line from the mask segmented by U2net; from the triangle and circle masks segmented by U2net, use OpenCV's contour lookup and moment calculation to obtain the sub-pixel level center pixel coordinates (ui, vi) of each geometric pattern; S32: Template Matching: Establish a one-to-one object relationship between 2D points in the target image and 3D points in the diffractive optical element template based on the template, obtaining a 2D-3D mapping; the diffractive optical element template includes the position information and arrangement sequence of triangular and circular patterns; utilizing the known relative positional relationship and precise geometric shape of triangles and circles in physical space, a DOE template is formed; based on the physical design of the DOE template, the precise three-dimensional coordinates Pi=(Xi, Yi, Zi) of each triangle and circle in the DOE coordinate system are known in advance. To establish an accurate correspondence, the KDTree (K-Dimensional Tree) algorithm is used to spatially index and match feature points: a KDTree is constructed using the theoretical coordinates of each geometric pattern in the template, and the image feature points extracted in step S31 are used as query points. Nearest Neighbor Search is performed using KDTree, and the 2D observation points in the image are automatically matched with the known 3D points in the template based on the spatial proximity, thereby establishing a one-to-one correspondence between 2D points in the image and 3D points in the template, obtaining a 2D-3D mapping; In terms of specific steps: 1.1 Center of gravity calculation (get_centroid) Used to simplify polygons (such as triangles, circular outlines) or sets of points to a single central coordinate.
[0046]
[0047] Where Cx and Cy are the calculated centroid coordinates, N is the total number of coordinate points contained in the shape, and xi and yi are the coordinates of the i-th vertex. 2.1 Context-based determination (get_triangle_type_context) (New logic / High priority) The determination is made by using the position of the nearest neighbor "point" relative to the "triangle".
[0048] logic: 1) Calculate the centroid (Tx, Ty) of the triangle.
[0049] 2) Find the point (Px, Py) that is closest to the centroid among all the "points".
[0050] 3) Compare Py and Ty
[0051] If Py < Ty: The point is above the triangle (Y increases downward in the image coordinate system), and it is determined to be an inverted triangle; If Py > Ty: The point is below the triangle, and it is determined to be a regular triangle.
[0052] Among them, Tx, Ty are the centroid coordinates of the triangle. Px, Py are the centroid coordinates of the reference point (Point) that is the closest. D is the Euclidean distance; 2.2 Judgment based on geometric shape When no reference point can be found around, it is judged only based on the vertex distribution of the triangle itself.
[0053] Formula: Geometric vertical midpoint (filtering noise points using percentiles):
[0054] Among them, Pk%(Y) is the k-th percentile value of the set of Y coordinates of all vertices. Cy is the Y coordinate of the centroid of the triangle.
[0055] 3. Spatial Registration, which is used to calculate the local offset of the data to be classified relative to the template.
[0056] Define how far the feature points in the current sliding window are considered valid reference points. The formula is as follows:
[0057] Among them, R is the search radius. W is the window width, set to 40, and H is the window height, set to 30.
[0058] 3.2 Calculation of local offset (find_local_offset) Calculate the average displacement of the feature points in the target image and the corresponding feature points in the template image.
[0059] For each target feature point T in the current window, find the feature point A of the same type (such as both being regular triangles) and the closest distance in the template. The formula is as follows: Find the best match:
[0060] Constraints: Type(A) = Type(T) and distance < 50 pixels.
[0061] Calculate a single offset:
[0062] Calculate the average offset:
[0063] Where Tx and Ty are the coordinates of the feature points in the data to be classified. Ax and Ay are the coordinates of the feature points in the template data. M is the number of successfully matched feature point pairs. ΔX and ΔY are the final correction offsets applied to the current point.
[0064] 3.3 Coordinate correction, the formula is as follows:
[0065] 4. Line Classification and Prediction 4.1 Determining the line axis (expected_axis) To determine whether the current line is primarily horizontal or vertical, and to filter KDTree search results, the formula is as follows:
[0066] Where Δxlocal represents the horizontal direction, and Δylocal represents the vertical direction. i is the index of the current point. xi+1 and xi-1 are the coordinates of adjacent points.
[0067] 4.2 Template Matching Classification (get_closest_line_label) At the corrected coordinate position, search for the nearest line label in the KDTree template.
[0068] formula:
[0069] Constraints: dist < 30 (distance threshold). The label suffix must match expected_axis (e.g., if determined to be the x-axis, only horizontal labels are accepted).
[0070] 5. Result Smoothing To eliminate individual classification errors (noise), a sliding window is used for majority voting, as shown in the following formula: For the i-th point in the sequence, take the window range:
[0071] Where w is the window size (window_size=5). Mode is the mode function (the element that appears most frequently).
[0072] S33: Line Number Matching and 3D Reconstruction: The grating lines segmented by the U2net deep learning model are reconstructed in 3D to obtain the grating mesh, as shown below. Figure 7 As shown; given that traditional line number matching methods based on intersections or parallel lines are prone to failure due to occlusion, a new line number matching method is proposed, using triangles and circles segmented by U2net as anchor points; the details are as follows: S331: Skeletonize or extract the central axis of all line category instances segmented by the U2net deep learning model to obtain the pixel set of the line; S332: For each segmented triangle and circle category instance, calculate its center point or feature corner point, project its 2D image coordinates onto the 3D plane where the diffractive optical element is located, and obtain its 3D spatial coordinates. S333: By using the known arrangement rules of the diffractive optical element template (e.g., triangles are always located in odd-numbered rows of a specific column, and circles are located in even-numbered rows), the true sequence number of these 3D spatial coordinates is determined, thereby establishing the correspondence between the 2D segmented features and the 3D template; S334: Using 3D spatial coordinates with actual serial numbers as anchor points, adjacent lines are matched for serial number matching. For example, a raster line that passes through a triangle with serial number i and a circle with serial number i+1 has an actual serial number of i. In this way, even if some lines are interrupted due to obstacles, their serial numbers can still be determined by the feature points (triangles, circles, or intersections) at their two ends, achieving seamless 3D reconstruction. The step number can be further illustrated with the following example: a. If a raster line intersects a triangle numbered i and a circle numbered j in a DOE template, the true index of the line can be determined using the known geometric rules of the DOE. For example, if the DOE design specifies that the i-th row and the j-th row are identified by a triangle and a circle respectively, then the index of the line can be uniquely determined; b. Even if a line is interrupted by an obstacle, the intersections of its two ends with feature points (such as triangles or circles) can still provide sufficient clues to infer its original sequence number. For example, if an interrupted line intersects with a triangle numbered i at one end, the line's sequence number can be inferred to be i. Based on this, a template-matching-like approach is used to identify line sequences, with results such as... Figure 6 As shown.
[0073] S335: For each 2D line with a defined sequence number, the 3D coordinates of any pixel on the 2D line are calculated using the camera projection model in the coordinate system of the diffractive optical element; specifically as follows: a. For any pixel point p=(u, v)T in a 2D image, its coordinates in the normalized camera coordinate system are...
[0074] It can be calculated using the inverse of the camera intrinsic parameter matrix K, as follows:
[0075] b. The 3D coordinates of this point in the camera coordinate system can be represented as: Where λ is the depth value; R and t obtained through PnP can be used to transform 3D points in the camera coordinate system to the DOE coordinate system: ; c. Since this point is located on the 3D plane containing the raster line with the known index (this plane is Z=0 or a known constant Z0 in the DOE coordinate system), this information can be used to calculate the depth value λ, according to the plane equation:
[0076] Transform the camera coordinate system to the DOE coordinate system (assuming RT is the transpose of the rotation matrix). (where t is the normalized pixel coordinate and t is the translation vector).
[0077] Therefore, we can solve the problem as follows:
[0078] d. After obtaining the depth value λ, the precise 3D coordinates P of the pixel in the DOE coordinate system can be obtained. DOE ; Furthermore, in step S345, for each 2D line with a determined sequence number, each pixel is projected backward into three-dimensional space using a camera projection model to form a 3D ray. Using the pre-calibrated ground equation Ax+By+Cz+D=0, the intersection point of each 3D ray with the ground is calculated, thereby obtaining the 3D coordinates of each line on the ground; where x, y, and z are coordinate points in three-dimensional space; and A, B, C, and D are plane equation coefficients. S34: 3D obstacle localization: Extract pixel contours from object category instances segmented by the U2net deep learning model; match the pixel set of the pixel contours with the raster mesh to calculate the position, size and orientation of the obstacle in 3D space, thus achieving 3D obstacle localization.
[0079] Specifically, step S34 includes the following steps: a. Extract the pixel outlines from the objection instances segmented from U2net; b. Match the set of pixels of the outline with the 3D reconstructed raster mesh; the intersection of the points on the obstacle outline and the raster lines is the contact point between the obstacle and the raster in space; c. Based on the line number of the intersection point and the pixel position of the point in the 2D image, the position of the intersection point in the 3D spatial coordinate system is accurately calculated using the above-mentioned 3D reconstruction method; d. By using at least two such intersection points, the position, size, and orientation of the obstacle in three-dimensional space can be calculated, achieving precise three-dimensional positioning of the obstacle. Specifically, by calculating the three-dimensional coordinates of all points on the obstacle's outline, the point cloud data of the obstacle can be obtained, and its bounding box can be further fitted to obtain its three-dimensional dimensions and spatial orientation.
[0080] In one specific implementation, the method further includes step S4: result visualization: projecting the 3D coordinate information of the reconstructed obstacle, along with the 3D model of the raster lines, back into a 2D image or presenting it as a 3D point cloud. This visualization provides intuitive and understandable perceptual results, supporting real-time monitoring and decision-making.
[0081] A system using the environment perception and detection method based on beam shaping and visual fusion as described in any of the above embodiments includes: Infrared grating projection module: Used to project an infrared grating with planar geometric feature patterns into the environment; specifically, the infrared grating projection module includes a light source and a diffractive optical element (DOE). The light source is an infrared laser source with a wavelength of 940nm. When the distance from the center of the DOE along the central optical axis to the equivalent receiving surface is 196.7cm, it can project a grid pattern onto the target in the correct orientation, with a maximum horizontal distance of 1000cm and a maximum vertical distance of 520cm. The innovation of this DOE lies in embedding pre-designed planar geometric feature patterns into the grid pattern. In this embodiment, triangular and circular patterns are preferred. These patterns have known and precise relative positions in the physical world. A schematic diagram of the DOE is shown below. Figure 1 As shown; Specifically, in this implementation, in addition to projecting conventional parallel grid lines, DOE also projects geometric feature patterns with predefined arrangements at specific positions of the grid. These feature patterns have known relative positional relationships and precise geometric shapes in physical space, serving as unique identifiers for precise positioning and sequence matching of raster lines in the image; thus solving the problem of raster line sequence matching from a physical perspective.
[0082] Image acquisition module: used to acquire target images; in this embodiment, a 1M camera with 940nm infrared band sensing capability is used to acquire grating images projected by DOE onto the environment in real time.
[0083] The computational processing module is used to perform multi-class instance segmentation and 3D reconstruction on the target image to obtain the 3D coordinates of the obstacle. This module adopts a high-performance industrial control computer, such as the DT-5309-BQ470MA, equipped with an Intel Core i7-10700 processor and an NVIDIA RTX 3060 12G graphics card to meet the real-time requirements of deep learning model inference and 3D reconstruction algorithms.
[0084] Human-computer interaction module: used to display processing results and system status.
[0085] The system in this embodiment embeds uniquely identifiable triangular and circular features into the raster mesh using a self-designed DOE, solving the problem of raster line number matching at the physical level. It employs an improved U2net model to achieve pixel-level instance segmentation of multiple target categories, significantly enhancing the system's robustness under complex conditions such as occlusion and interruption. It innovatively integrates multiple visual features such as lines, intersections, triangles, circles, and obstacle contours, and uses a multi-feature anchor point-based 3D reconstruction algorithm to accurately determine the number of each raster line, thereby calculating its position in 3D space and ultimately achieving centimeter-level obstacle localization.
[0086] It is also worth noting that the system demonstrated excellent performance in real-vehicle testing, especially in minimum obstacle detection and precision control, meeting the real-time and reliability requirements of advanced driver assistance systems (ADAS) and providing strong perception support for intelligent driving.
[0087] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.
Claims
1. An environmental perception and detection method based on beam shaping and visual fusion, characterized in that: Includes the following steps: S1: Obtain the target image: The target image is a grating image with a planar geometric feature pattern; wherein the planar geometric feature pattern is used to locate and match the sequence number of the grating lines in the target image; S2: Multi-class instance segmentation: Input the target image into the U2net deep learning model to perform pixel-level instance segmentation of multiple classes of objects in the target image, and obtain the first image corresponding to each class of objects; S3: 3D Reconstruction: Extract feature points from the first image of each type of object, and establish a 2D-3D mapping based on the position of the feature points and planar geometric feature patterns in the target image; based on the 2D-3D mapping relationship, perform sequence matching and 3D spatial positioning on each grating line in the first image to determine the 3D coordinates of the obstacle.
2. The environmental perception and detection method based on beam shaping and visual fusion according to claim 1, characterized in that: Step S1, which involves acquiring the target image, specifically includes: S11: Obtain the original image: Based on the laser emitter and diffractive optical elements, an infrared grating is projected toward the sensing area, and the image projected by the infrared grating is obtained based on the camera module to obtain the original image; wherein the diffractive optical elements project a planar geometric feature pattern with a predefined arrangement at a specific position of the grid; the planar geometric feature pattern is a triangular pattern and a circular pattern, and the triangular pattern and the circular pattern have a known and precise relative position in the physical world; S12: Extract the infrared grating image from the original image, and crop and grayscale the infrared grating image to obtain the target image.
3. The environmental perception and detection method based on beam shaping and visual fusion according to claim 2, characterized in that: Step S12 includes obtaining the image of the separated infrared grating from the camera module through an image acquisition device; and cropping the image of the infrared grating by selecting a ROI region, and converting the cropped color image into a grayscale image.
4. The environmental perception and detection method based on beam shaping and visual fusion according to claim 2, characterized in that: The training process of the U2net deep learning model in step S2 is as follows: Define multiple categories of objects: Set up 6 categories of objects, namely: 0: background, 1: triangle, 2: line, 3: point, 4: objection, 5: circle; Acquiring training data: The training data is labeled data: the image containing infrared gratings and obstacles is labeled at the pixel level; The U2net network structure is adopted, and its loss function is set. The weighted cross-entropy loss function is used as the model's loss function, specifically: Among them, y i p is the true category label of i pixels. yi w is the probability that the model predicts it to be of that category. yi This represents the weight of the category; N is the total number of pixels. Based on the training data, the U2net network structure is used for training to obtain the final U2net deep learning model.
5. The environmental perception and detection method based on beam shaping and visual fusion according to claim 4, characterized in that: When the target image is input into the U2net deep learning model, the U2net deep learning model converts the target image into a probability map with 6 channels, and then performs post-processing on each probability map to obtain the mask and mask image for each category; where each channel corresponds to a category object.
6. The environmental perception and detection method based on beam shaping and visual fusion according to claim 5, characterized in that: Step S3 specifically includes the following steps: S31: Feature point extraction: Extract the set of pixel coordinates for each raster line and the center pixel coordinates for each triangle and circle from the mask of each category; S32: Template matching: Establish a one-to-one object relationship between 2D points in the target image and 3D points in the diffractive optical element template based on the diffractive optical element template, and obtain the mapping between 2D and 3D; the diffractive optical element template includes the position information and arrangement sequence of triangular patterns and circular patterns; S33: Line number matching and 3D reconstruction: The grating lines segmented by the U2net deep learning model are reconstructed in 3D to obtain the grating mesh; S34: 3D obstacle localization: Extract pixel contours from object category instances segmented by the U2net deep learning model; match the pixel set of the pixel contours with the raster mesh to calculate the position, size and orientation of the obstacle in 3D space, thus achieving 3D obstacle localization.
7. The environmental perception and detection method based on beam shaping and visual fusion according to claim 6, characterized in that: Step S33 specifically includes: Skeletonization or central axis extraction is performed on all line category instances segmented by the U2net deep learning model to obtain the pixel set of the line; For each segmented triangle and circle category instance, calculate its center point or feature corner point, project its 2D image coordinates onto the 3D plane where the diffractive optical element is located, and obtain its 3D spatial coordinates. By using the known arrangement rules of the diffractive optical element templates, the true serial numbers of these 3D spatial coordinates are determined, thereby establishing the correspondence between the 2D segmented features and the 3D templates; Using 3D spatial coordinates with real serial numbers as anchor points, the serial numbers of adjacent lines are matched. For each 2D line with a defined sequence number, the 3D coordinates of any pixel on the 2D line are calculated using the camera projection model in the coordinate system of the diffractive optical element.
8. An environmental perception and detection method based on beam shaping and visual fusion according to claim 6 or 7, characterized in that: It also includes step S4: result visualization: projecting the 3D coordinate information of the obstacle after 3D reconstruction, together with the 3D model of the raster lines, back to a 2D image or presenting it in the form of a 3D point cloud.
9. A system using the environmental perception and detection method based on beam shaping and visual fusion as described in any one of claims 1-8, characterized in that: include: Infrared grating projection module: used to project infrared gratings with planar geometric patterns into the environment; Image acquisition module: used to acquire target images; The computational processing module is used to perform multi-class instance segmentation and 3D reconstruction on the target image to obtain the 3D coordinates of the obstacles. Human-computer interaction module: used to display processing results and system status.