Environment sensing method based on target detection and stereo matching

Abstract

The invention relates to an environment sensing method based on target detection and stereo matching. According to the environment perception method based on target detection and stereo matching, targets in an environment image can be recognized through model training, and category information and position information of each target can be recognized, so that a target detection model can recognize the category information and the position information of the targets in the environment image in real time; through a local matching strategy of a target area delineated by focusing on the position information of the target, huge calculation overhead caused by full-image matching is avoided; pixel-level parallax information in the two-dimensional image is accurately converted into a physical depth distance in a three-dimensional space in combination with calibration parameters of a binocular camera in an environment sensing system; and finally outputting an environment perception image capable of displaying a detailed perception result of each detected target through information fusion. According to the environment sensing method, the efficiency and robustness of the whole sensing process are remarkably improved while the accuracy and integrity of environment sensing are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of environment perception technology, in particular to an environment perception method based on target detection and stereo matching. BACKGROUND

[0002] In the application fields of industrial detection, automatic driving and real-time vision analysis, the conventional target detection algorithm can usually only output the category information and two-dimensional position information of multiple targets in the image, and lacks the perception ability of target depth or distance information. In order to obtain the spatial distance of the target, binocular vision becomes an effective technical path. However, the traditional stereo matching method usually needs to generate a dense depth map for the entire field of view, which has high computational complexity and large resource consumption, and is difficult to meet the needs of high real-time scenarios. SUMMARY

[0003] Therefore, it is necessary to provide an environment perception method based on target detection and stereo matching, which can quickly lock the region of interest through target detection, and then perform stereo matching and depth calculation only in the key region, so as to realize the integrated perception of target category, position and distance information while significantly reducing the computational load.

[0004] An environment perception method based on target detection and stereo matching, comprising the steps of:

[0005] acquiring binocular images and performing fine target labeling on the binocular images to generate training set data with target category information and target position information;

[0006] using the training set data to perform end-to-end model training on a deep learning network model to obtain a target detection model;

[0007] using a binocular camera in an environment perception system to collect binocular images in the surrounding environment as environment images in real time;

[0008] inputting the environment images into the target detection model to output recognition images with target category information and target position information;

[0009] inputting the local images circled by the target position information in the recognition images into a stereo matching model adopting a multi-channel independent matching strategy to perform matching cost calculation, so as to obtain the disparity value of the local images where the target is located;

[0010] combining the calibration parameters of the binocular camera in the environment perception system to convert the disparity value of the local images where the target is located into the depth value of the target; the depth value of the target is the spatial distance between the target and the plane where the binocular camera of the environment perception system is located;

[0011] The class information and the location information of the target output by the target detection model are integrated and fused with the depth value of the target and the environment image to obtain an environment perception image with the class information of the target, the location information of the target and the depth value of the target.

[0012] The above-mentioned environment perception method based on target detection and stereo matching can accurately detect and identify the target in the surrounding environment through the target detection model obtained through model training, and identify the class information and the location information of the target; in the stereo matching stage, the local matching strategy focusing on the target region circled by the location information of the target is adopted to avoid the huge calculation overhead caused by full-image matching, thereby significantly improving the environment perception efficiency; the improved stereo matching model adopting the multi-channel independent matching strategy is applied to enhance the matching robustness; the pixel-level disparity information in the two-dimensional image is accurately converted into the physical depth distance in the three-dimensional space by combining the calibration parameters of the binocular camera in the environment perception system; and the environment perception image capable of displaying the detailed perception results of each detected target in the surrounding environment of the environment perception system is finally output through the integration and fusion of the identification information, the calculation information and the environment image.

[0013] Therefore, the above-mentioned environment perception method based on target detection and stereo matching can significantly improve the efficiency and robustness of the overall perception process while ensuring the accuracy and integrity of the environment perception, and is especially suitable for complex dynamic application scenarios with high real-time requirements. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 FIG. 1 is a flowchart of an environment perception method based on target detection and stereo matching according to an embodiment of the present application;

[0015] Figure 2 FIG. 2 is a flowchart of step S10 in the environment perception method based on target detection and stereo matching shown in FIG. 1; Figure 1

[0016] Figure 3 FIG. 4 is a flowchart of step S20 in the environment perception method based on target detection and stereo matching shown in FIG. 1; Figure 1

[0017] Figure 4 FIG. 5 is a flowchart of step S21 in step S20 shown in FIG. 4; Figure 1

[0018] Figure 5 FIG. 6 is a flowchart of step S80 in the environment perception method based on target detection and stereo matching shown in FIG. 1; Figure 4

[0019] Figure 6 Figure 5 ​​​​​Flowchart of step S213 in step S21 shown in the figure;

[0020] Figure 7 For Figure 5 Flowchart of step S215 in step S21 shown in the figure;

[0021] Figure 8 For Figure 1 Flowchart of step S50 in the environment perception method based on target detection and stereo matching shown in the figure. DETAILED DESCRIPTION

[0022] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, some designs, manufacturing or production changes made on the basis of the technical content disclosed in the present application are only routine technical means for those of ordinary skill in the art related to the content disclosed in the present application, and should not be understood as insufficient disclosure of the content disclosed in the present application.

[0023] In the present application, the phrase "embodiments" means that the specific features, structures or characteristics described in combination with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase at various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0024] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be inclusive of both the singular and the plural, unless otherwise indicated. The terms "comprising", "comprises" and "comprised of" as well as "having", "including", "containing" and "characterized by" are inclusive and do not exclude other steps, elements or ingredients. The terms "a", "an" and "the" followed by a reference to an element are used to describe all elements or the element comprising one or more elements unless otherwise indicated. The term "or" as used herein is used to mean, and is used interchangeably with, the inclusive or. The designation of terms "first", "second", "third", etc. are used to distinguish similar elements and do not imply a specific order or a specific spatial or temporal sequence.

[0025] Referring to Figure 1 The environment perception method based on target detection and stereo matching in an embodiment of the present application can be applied to the fields of industrial detection and automatic driving. The present application is described by taking the application in the field of automatic driving as an example. The environment perception method based on target detection and stereo matching comprises steps S10 to S70.

[0026] In step S10, binocular images are acquired and fine target labeling is performed on the binocular images to generate training set data with position information of targets and position information of the targets. In order to ensure the comprehensiveness of the labeled targets in the training set data, a large number of pairs of binocular images are often acquired as initial data for target labeling. The execution of step S10 provides high-quality supervision signals for model training in subsequent step S20.

[0027] The labeled targets can be freely selected according to different use scenarios. For example, in the field of automatic driving of vehicles, the targets can be vehicles (e.g., cars, bicycles, motorcycles, electric vehicles, tricycles, trucks, etc.), pedestrians, animals (e.g., dogs, cats, etc.), stationary objects (e.g., houses, road curbs, ditches, fences, flower beds, trees, green belts, etc.), and all obstacles that can affect the safe driving of vehicles; for example, in the field of industrial detection, the targets can be detection items of products to be detected, such as various "unqualified" items of products.

[0028] Referring to Figure 2 Specifically, step S10 comprises steps S11 to S13.

[0029] Step S11: Obtain stereo images.

[0030] Specifically, stereo images can be obtained through ontology acquisition or online public dataset acquisition.

[0031] Step S12: Label and convert the binocular images.

[0032] Step S13: Crop, stretch, rotate, and flip the labels and the converted binocular images to obtain training set data with target location information and target category information.

[0033] By executing steps S11 and S12, a training set database is constructed, and by executing step S13, the quality of the training set database is improved, thereby obtaining training set data.

[0034] Step S20 involves using the training set data to perform end-to-end model training on the deep learning network model to obtain an object detection model. By executing step S20, a target training model is trained that can quickly identify the category information and location information of targets in environmental images.

[0035] The deep learning network model can be either the YOLO series or other commonly used object detection models. The YOLO series includes YOLOv5, YOLOv8, and YOLOv9-Nano, all of which are single-stage object detection models. Their main characteristic is that they can achieve extremely high detection speed while maintaining detection accuracy, making them suitable for real-time environmental perception tasks. Specifically, the YOLOv5 basic model adopts a lightweight design and has a fast inference speed, making it suitable for rapid environmental detection in autonomous driving or industrial inspection. YOLOv8 further optimizes the network structure and feature fusion strategy, improving the accuracy of small and multiple object detection in complex scenes. YOLOv9-Nano, while maintaining high accuracy, further compresses the number of model parameters and computational cost, making it suitable for embedded devices with limited computing power.

[0036] In addition, two-stage object detection models can be used, such as Cascade R-CNN, Faster R-CNN with FPN, RetinaNet, and the ViT-Enhanced model based on the visual Transformer. Among them, Cascade R-CNN optimizes the positional information of the target step by step through a multi-level regressor, which can significantly improve the accuracy of high-quality detection, especially suitable for the accurate detection of large targets and targets in complex backgrounds; Faster R-CNN with FPN combines a feature pyramid network (FPN) to achieve multi-scale feature fusion, improving the detection capability of small targets; RetinaNet introduces Focal Loss to solve the problem of severe imbalance between positive and negative samples, improving the detection accuracy of rare targets; the ViT-Enhanced model is based on the visual Transformer structure and models the global information of the image through a self-attention mechanism, which can enhance the feature representation capability in multi-target and complex scenes, and improve the reliability of the environmental perception system in complex environments.

[0037] Specifically, in this embodiment, the deep learning network model is the YOLOv5 basic model, which achieves a balance between speed and accuracy, and can meet the real-time and accuracy requirements of autonomous driving environment perception.

[0038] Step S30 involves using a binocular camera in the environmental perception system to acquire binocular images of the surrounding environment in real time, which are then used as environmental images. Step S30 is typically performed during the autonomous driving process of a vehicle or during industrial inspection, to perceive the environment around the vehicle in real time during autonomous driving, or to perceive the inspection image of the product to be inspected during industrial inspection.

[0039] Step S40: Input the environmental image into the target detection model to output a recognized image with target category information and target location information.

[0040] Step S50: Input the local image defined by the location information of all targets in the identified image into the improved stereo matching model using a multi-channel independent matching strategy to calculate the matching cost, so as to obtain the disparity value of the local image where the target is located.

[0041] Step S60: Combining the calibration parameters of the binocular camera in the environmental perception system, the disparity value of the local image where the target is located is used as the depth value of the target. The depth value of the target is the spatial distance between the target and the plane where the binocular camera of the environmental perception system is located.

[0042] Specifically, step S60 is: using the formula The disparity value of the target image is obtained by using the calibration parameters of the binocular camera in the environmental perception system. Convert to the target depth value .in, For the focal length in the calibration parameters of the binocular camera in the environmental perception system, The baseline length is used in the calibration parameters of a stereo camera in an environmental perception system. Therefore, the depth value Z and disparity value D of a target in physical space satisfy an inverse proportional relationship.

[0043] Step S70: Integrate and fuse the target category information and target location information output by the target detection model with the target depth value and the environmental image to obtain an environmental perception image containing the target category information, target location information, and target depth value. The target location information refers to the specific location of the target in the environmental image or environmental perception image, i.e., the coordinate information of the area occupied by the target in the environmental image or environmental perception image.

[0044] By executing steps S10 to S70, the categories of targets in the environment surrounding the environmental perception system, the positions of targets in the environmental perception images, and the depth values ​​between targets and the stereo cameras in the environmental perception system can be obtained quickly and accurately in real time.

[0045] Steps S10 to S40 are executed to accurately identify targets in the surrounding environment, including their category and location information. Step S50 utilizes a local matching strategy focused on the target region defined by the target's location information, avoiding the significant computational overhead of full-image matching and significantly improving environmental perception efficiency. The application of an improved stereo matching model with a multi-channel independent matching strategy enhances matching robustness. Step S60 establishes a mapping relationship from two-dimensional image pixels to three-dimensional physical space based on triangulation principles, accurately converting pixel-level disparity information in the two-dimensional image into physical depth distance in three-dimensional space. Step S70, through coordinate transformation and data association, finally outputs an environmental perception image displaying detailed perception results for each detected target in the surrounding environment.

[0046] Therefore, the above-mentioned environment perception method based on target detection and stereo matching significantly improves the efficiency and robustness of the overall perception process while ensuring the accuracy and completeness of environment perception. It is especially suitable for complex dynamic application scenarios with high real-time requirements.

[0047] Please refer to it again. Figure 1 In some embodiments, after step S70, step S80 is further included: using the location information of the target in the environmental perception image and the depth value of the target, the distance information between two adjacent targets in the environmental perception image and the relative position information between two adjacent targets are calculated.

[0048] Thus, in an autonomous driving environment, after knowing what types of targets exist in the surrounding environment, where each target is located in the environmental image, and what the spatial depth of each target is relative to the binocular camera in the environmental perception system, it is also possible to obtain the distance between adjacent targets in the surrounding environment and the relative positions between adjacent targets. This information is used to help determine whether an autonomous vehicle can safely pass between two adjacent targets, thereby further improving the driving safety and reliability of autonomous driving.

[0049] Please refer to the following: Figure 3 Specifically, the step of calculating the distance information between two adjacent targets in the environmental perception image includes steps S81 to S83:

[0050] Step S81: Establish a three-dimensional coordinate system using the plane where the binocular camera in the environmental perception system is located as the XY plane.

[0051] Step S82: Based on the target's location information and depth value, obtain the coordinates of adjacent targets A and B in the three-dimensional coordinate system within the surrounding environment of the environmental perception system, denoted as (X... a Y a Z a ) and (X b Y b Z b ).

[0052] Step S83: Calculate the distance between target A and target B in the surrounding environment according to the following formula. : .

[0053] By executing steps S81 to S83, the distance between two adjacent targets in the surrounding environment can be calculated in real time based on the target's location information and depth value in the environmental perception image.

[0054] Please refer to the following: Figure 4 In some embodiments, step S20 includes steps S21 and S22.

[0055] Step S21: Introduce an attention-based FCM module into the Head part of the deep learning network model to obtain an initial detection model.

[0056] Step S22: Input the training set data into the initial detection model for training to obtain the target detection model.

[0057] The Feature Enhancement Module (FCM module) focuses on mitigating the information imbalance problem caused by the loss of information about small targets in deep networks. Its purpose is to integrate the spatial location information of targets more deeply into the network, better aligning it with deeper semantic information and improving the localization ability of small targets. Therefore, by introducing an attention-based FCM module into the architecture of a deep learning network model, the feature extraction capability of the target detection model for difficult targets such as small-sized and weakly textured targets is effectively enhanced, improving detection recall and accuracy. This, in turn, enables a comprehensive perception of all targets in the environment, improving environmental awareness.

[0058] Please refer to the following: Figure 5 Furthermore, in some embodiments, step S21 includes steps S211 to S217.

[0059] Step S211, using formula (X) 1 ,X 2 =Split(X) splits the input feature X (X∈C×H×W) into channels X. 1 =αC and channel X 2 =(1-α)C. α is the splitting ratio, 0≤α≤1; C is the number of channels of input feature X; H and W are the height and width of input feature X, respectively.

[0060] The input features here refer to the feature maps obtained after the labeled data has undergone a series of convolution and pooling operations after being input into the target training model.

[0061] Step S212, using formula (X) C ,X S )=φ1(X 1 ,X 2 ), X 1 The branch processing unit φ1, composed of a standard 3×3 convolution, is fed into the branch processing unit to obtain the spatial branch X. C , will X 2 The data is fed into a branch processing unit φ1 composed of pointwise convolutions to obtain the positional branch X. S .

[0062] Step S213, spatial branch X C Introduced into channel interaction, weights are assigned to important information on each channel of the channel interaction.

[0063] Step S214: The spatial branch X of channel interaction is selected through the channel attention mechanism. C Mapped to location branch X S By fusing complementary features, the first fusion branch is obtained.

[0064] Step S215, set the location branch X SIntroduced into spatial interaction, it assigns weights to spatial information on each channel of spatial interaction.

[0065] Step S216: Use spatial attention mechanism to locate the positional branch X of spatial interaction. S Mapped to spatial branch X C The two sides complement each other and merge to obtain the second fusion branch.

[0066] Step S217: Perform residual connection on the first fusion branch and the second fusion branch to obtain feature X. FCM Among them, feature X FCM Features that include both spatial and semantic mappings.

[0067] Specifically, step S217 is: using formula X FCM =(X C ω2)⊕(X S ω1) Connect the first fusion branch and the second fusion branch together to obtain feature X. FCM .

[0068] Where ω1 is the weight of important information on each channel of channel interaction, and ω2 is the weight of spatial information on each channel of spatial interaction.

[0069] As the network deepens, branches with low-level spatial information become more prominent, and more and more low-level spatial information is implicitly encoded into high-dimensional vectors. Therefore, enhancing the acquisition of low-level information at the appropriate time can improve performance. By executing step S212, spatial branches X that can extract richer feature information on each channel are obtained. C The location branch X, which can extract relatively weak information through pointwise convolution and retain a large amount of shallow spatial location information, is... S To obtain spatial mappings of semantic and location information respectively.

[0070] After separating the branches, the acquired features and spatial branch X C and location branch X S Although effective, it is discrete, which can lead to inaccurate matching of target features. Therefore, by executing steps S213 to S216, in the spatial branch X... C and location branch X S They are mapped to each other to compensate for their respective missing feature mappings, achieve efficient feature matching, and enable strong features to guide weak features, so as to alleviate the problem of information imbalance.

[0071] Please refer to the following: Figure 6 Specifically, step S213 includes steps S2131 to S2133.

[0072] Step S2131: Perform a 3×3 depthwise separable convolution operation on each channel of the channel interaction to cut off the information between channels and obtain the convolution feature X. i D The calculation formula is: X i D =φ2(k i ,X i C ), where X i C k represents the original input features that will be fed into the spatial branch convolution operation. i For the integer number of channels, there are independent 3×3 depth-separable convolution kernels, where φ2 represents the mathematical notation for the convolution operation. Specifically, k i Each convolutional kernel can only process one channel of the input feature. Specifically, φ2 defines the complete processing flow: depthwise convolution → batch normalization → ReLU activation.

[0073] Step S2132, for the convolutional feature X i D Global average pooling is performed to obtain global convolutional features.

[0074] Step S2133: The global convolutional features are input into the sigmoid layer to output the weights ω1 of key information. The calculation formula is: ω1 = R[ R represents the activation function. Index coordinates representing spatial location, This represents a depthwise convolutional feature map. Specifically, R is the Sigmoid function.

[0075] Please refer to the following: Figure 7 Specifically, step S215 includes steps S2151 and S2152.

[0076] Step S2151, set the location branch X S Introduced into spatial interaction, it generates a spatial attention image.

[0077] Step S2152: Map the spatial attention image onto the branch after a 3×3 standard convolution to generate the spatial information weight ω2, calculated as: ω2=R(F(X S R represents the activation function, F(X) S ) represents the location branch feature X S Spatial attention map of the transformation.

[0078] Please refer to the following: Figure 8 In some embodiments, step S50 includes steps S51 to S56.

[0079] Step S51: Obtain the total pixel area of ​​the recognized image and the area of ​​the target pixel region for each target.

[0080] Step S52: Determine the maximum disparity value n for each target based on the proportion of the target pixel area in the total pixel area.

[0081] Specifically, if the area of ​​the target pixel region accounts for ≤5% of the total pixel area, then n=4 is determined;

[0082] If 5% < the proportion of the target pixel area in the total pixel area ≤ 10%, then n = 8;

[0083] If 10% < the proportion of the target pixel area in the total pixel area ≤ 20%, then n = 16 is determined;

[0084] If the area of ​​the target pixel region accounts for more than 20% of the total pixel area, the stereo matching condition is not met, and an invalid result is output. If the proportion is too large, it means that the target is too close to the stereo camera in the environmental perception system, and there is no point in performing stereo matching calculations on the target. In this case, the existing technology for autonomous driving has already implemented a way to handle close targets (i.e., immediately braking the vehicle to avoid obstacles when a close target is detected), so there is no need to obtain the spatial depth information of the target.

[0085] Of course, in other embodiments, the maximum disparity value n of the target can also be determined by multiplying the proportion of the target pixel area in the total pixel area by a specific coefficient.

[0086] Step S53: Determine the horizontal disparity d of the left and right images of each target in the recognition image based on the maximum disparity value n, where d = 0, 1, 2...n.

[0087] Step S54: Separate the three color channels of the local image defined by the location information of the target in the identified image, and independently calculate the matching cost of all targets on each color channel based on the horizontal disparity d between the left and right images to obtain the matching cost of the three color channels for each target. The three color channels are red (R), green (G), and blue (B).

[0088] Specifically, the steps for independently performing matching calculations for all targets on each color channel based on the horizontal disparity d of the left and right images are as follows: using the formula Calculate pixels Matching cost of the c-th channel .

[0089] in, R represents the red channel, G represents the green channel, and B represents the blue channel. The pixels in the local image defined to identify the location information of the target in the left image of the image. In the passage grayscale value, The pixels in the local image defined to identify the location information of the target in the right image of the image. In the passage grayscale value

[0090] Step S55: For the matching cost of the three color channels of each target, under the same disparity value of the horizontal disparity d of the left and right images, the matching cost of the three color channels is compared, and the minimum matching cost is selected as the final matching cost of the target under the same disparity value of the horizontal disparity d of the left and right images.

[0091] Specifically, step S55 is: using the formula The matching costs of the three color channels under the same disparity value d in the left and right images are compared, and the minimum matching cost is selected as the final matching cost for the target under the same disparity value d in the left and right images. .

[0092] Step S56: Compare the final matching costs of the horizontal disparity d of the left and right images under all disparity values, and select the minimum matching cost as the disparity value of the local image where the target is located.

[0093] Specifically, step S56 is: using the formula The disparity value of the local image where the target is located is calculated. .

[0094] Thus, by executing steps S51 to S53, based on the distance of the target from the binocular camera in the environment perception system, the horizontal disparity d of the left and right images in the recognition image is adaptively adjusted, making the horizontal disparity d of the left and right images positively correlated with the relative size of the target region. This ensures matching efficiency while taking into account the reasonable allocation of computing resources. By executing steps S51 to S56, a multi-channel independent matching and cost fusion strategy is implemented. The actual matching quality of the stereo matching model is dynamically adjusted by fusing the matching cost and execution degree weights of the three RGB color channels, thereby enhancing matching robustness. The depth estimation accuracy is improved through matching cost fusion and constraint optimization.

[0095] In the field of stereo vision measurement and depth estimation, feature matching algorithms have always been a crucial step. Traditional local feature algorithms, such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features), achieve relatively robust image matching by extracting keypoints and their descriptors that are scale- and rotation-invariant. However, compared with SGBM (Semi-Global Block Matching) in stereo matching algorithms, these two types of algorithms still have significant limitations in depth estimation and dense disparity reconstruction tasks, mainly in terms of matching density, computational efficiency, illumination sensitivity, and 3D geometric consistency.

[0096] From the perspective of matching density, SIFT and SURF are sparse feature matching algorithms. They extract keypoints only at significant corners or texture changes in an image, resulting in discrete keypoint pairs. This feature point matching method has advantages in tasks such as scene recognition, image stitching, or pose estimation, but it is insufficient for stereo measurement that requires continuous disparity distribution. In contrast, the SGBM algorithm computes a dense disparity map across the entire image through pixel-level or block-level semi-global optimization, reflecting the complete depth changes of an object's surface. Therefore, SIFT and SURF cannot directly generate continuous depth information; their results often require additional interpolation or sparse-to-dense transformation processes, leading to increased computational complexity and unstable accuracy.

[0097] From a computational efficiency perspective, both SIFT and SURF involve significant filtering and statistical operations during feature extraction and description. SIFT requires constructing a multi-scale Gaussian pyramid and searching for extreme points in scale space, while simultaneously calculating the gradient direction histogram to generate a 128-dimensional descriptor. Although SURF accelerates computation through integral images and Hessian matrices, its keypoint detection and 64-dimensional descriptor generation still require substantial convolution and integration operations. These steps are computationally burdensome when performed on high-resolution images or video sequences, making it difficult to meet real-time requirements. SGBM, through cost aggregation and dynamic programming, achieves high computational efficiency while maintaining a certain level of accuracy, especially with hardware platform acceleration (such as GPUs or FPGAs), enabling real-time stereo matching. Therefore, in terms of speed and practicality, neither SIFT nor SURF is as suitable as SGBM for real-time 3D reconstruction or dynamic scene depth measurement.

[0098] From the perspective of illumination and texture sensitivity, while SIFT and SURF possess illumination invariance to some extent, this invariance primarily targets local brightness variations or affine transformations. They remain highly sensitive to large-scale uneven illumination, reflections, shadows, or highlighted areas. Furthermore, their feature extraction relies on local gradient information of the image; when the target surface lacks significant texture, the number of feature points decreases significantly, even making effective matching impossible. In contrast, SGBM, through a cost aggregation mechanism based on assumptions of brightness consistency and smoothness, can achieve relatively continuous disparity estimation results even in low-texture regions through neighborhood constraints and global optimization. Therefore, in weakly textured or non-uniformly illuminated environments, SGBM exhibits significantly higher robustness than SIFT and SURF, which are based on local features.

[0099] In terms of 3D geometric consistency and adaptability to viewpoint changes, SIFT and SURF mainly rely on the scale and orientation invariance of keypoints to achieve robust matching. When the disparity is large or the baseline length is too long, the viewpoint difference between the left and right images causes significant changes in the projection of the same object in the two images, making it difficult for traditional feature descriptors to maintain a stable correspondence. In contrast, the SGBM algorithm directly constrains the consistency of disparity during the matching process and combines left-right consistency checks and global smoothing constraints, which can better address the geometric mismatch problem under conditions of large disparity or tilted viewpoints.

[0100] From an algorithm design perspective, SIFT and SURF are essentially feature recognition algorithms, focusing on the identification and matching quality of key points rather than depth recovery; while SGBM is a typical disparity calculation algorithm, its core design aiming to obtain high-precision pixel-level depth information. The different application focuses of these two types of algorithms lead to significant differences in their performance in stereo measurement. The feature point matching outputs of SIFT and SURF are more suitable for tasks such as camera calibration, image registration, or target recognition, while the dense disparity map output by SGBM is more suitable for scenarios such as 3D reconstruction, motion detection, or spatial measurement.

[0101] SIFT and SURF algorithms have significant advantages in the stability and invariance of feature extraction, but compared with the SGBM stereo matching algorithm, they have limitations in dense depth reconstruction, real-time performance, illumination robustness, and geometric consistency. These limitations make it difficult for traditional feature algorithms to be used alone for high-precision, continuous depth estimation tasks.

[0102] Therefore, the aforementioned environment perception method based on target detection and stereo matching, with its innovative three-channel separation matching and sufficient channel selection mechanism, fully utilizes the channel differences in color images, effectively improving matching accuracy under complex lighting and texture conditions. Particularly in the depth calculation stage, the parallax-depth conversion model based on triangulation principles provides the system with reliable geometric constraints from two-dimensional visual information to three-dimensional spatial distance, ensuring the physical accuracy of depth perception.

[0103] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An environment perception method based on target detection and stereo matching, characterized in that, The method comprises the steps of: acquiring binocular images and performing fine target labeling on the binocular images to generate training set data with class information of the target and position information of the target; training a deep learning network model end-to-end using the training set data to obtain a target detection model; acquiring binocular images in the surrounding environment in real time using a binocular camera in an environment perception system as environment images; inputting the environment images into the target detection model to output recognition images with class information of the target and position information of the target; inputting a local image circled by the position information of the target in the recognition image into an improved stereo matching model adopting a multi-channel independent matching strategy to perform matching cost calculation and obtain a disparity value of the local image where the target is located; combining calibration parameters of the binocular camera in the environment perception system to convert the disparity value of the local image where the target is located into a depth value of the target; the depth value of the target is a spatial distance between the target and a plane where the binocular camera in the environment perception system is located; integrating and fusing the class information of the target and the position information of the target output by the target detection model, the depth value of the target, and the environment images to obtain an environment perception image with the class information of the target, the position information of the target, and the depth value of the target.

2. The method for environment perception based on target detection and stereo matching according to claim 1, characterized in that, The step of obtaining the training set data with the class information of the target and the position information of the target comprises the steps of: acquiring binocular images; labeling and converting the binocular images; performing cropping, stretching, rotating, and reversing processing on the labeled and converted binocular images to obtain the training set data with the class information of the target and the position information of the target; and / or after integrating and fusing the class information of the target and the position information of the target output by the target detection model, the depth value of the target, and the environment images, the step of calculating distance information between adjacent two targets and relative position information between the adjacent two targets in the environment perception image using the position information of the target and the depth value of the target in the environment perception image. 3.The method of environment perception based on target detection and stereo matching according to claim 1, characterized in that, The step of training the deep learning network model end-to-end using the training set data comprises the steps of: introducing an FCM module based on an attention mechanism into a Head part of the deep learning network model to obtain an initial detection model; inputting the training set data into the initial detection model for training to obtain the target detection model.

4. The method for environment perception based on target detection and stereo matching according to claim 3, characterized in that, The step of introducing the FCM module based on the attention mechanism into the Head part of the deep learning network model to obtain the initial detection model comprises the steps of: by formula (X 1 ,X 2 )=Split(X) splits the channels of an input feature X (X e C x H x W) into channels X 1 = aC and channels X 2 =(1-a)C; a is a split ratio, 0≤a≤1; C is the number of channels of the input feature X, H and W are the height and width of the input feature X, respectively. by equation (X C ,X S )=φ1(X 1 ,X 2 ), X 1 is sent into a branch processing unit φ1 consisting of a standard 3x3 convolution to obtain a spatial branch X C , X 2 is sent into a branch processing unit φ1 consisting of a point-wise convolution to obtain a position branch X S ; The spatial branch X C introduced into the channel interaction to assign a weight to important information on each lane of the channel interaction; The spatial branch X of channel interaction is mapped to the position branch X through a channel attention mechanism C The spatial branch X of channel interaction is mapped to the position branch X through a channel attention mechanism S Complementary feature fusion is performed to obtain a first fusion branch; The position leg X S is introduced into the spatial interaction such that the spatial information on each channel of the spatial interaction is weighted; The spatial interaction position branch X is mapped to the spatial branch X by a spatial attention mechanism S C complementary fusion in the spatial branch X to obtain a second fusion branch;​ residually connecting the first fusion branch and the second fusion branch to obtain a feature X FCM .

5. The method for environment perception based on target detection and stereo matching according to claim 4, characterized in that, The spatial branch X C The step of introducing into the channel interaction a weighting of the important information on each channel of the channel interaction comprises: Each channel of the channel interaction is subjected to a convolution operation using a 3*3 depth separable convolution to cut off the information between the channels to obtain a convolution feature X i D , and the calculation formula is: X i D = φ2(k i , X i C ); X i C is the original input feature about to be subjected to a spatial branch convolution operation, k i is an integer after rounding, and φ2 represents a mathematical symbol of a convolution operation; global average pooling is performed on the convolution feature X i D to obtain a global convolution feature; The global convolution feature is input into a sigmoid layer to output a weight ω1 of key information, and a calculation formula is: ω1=R[ ]; R represents an activation function, an index coordinate representing a spatial position, a deep convolution feature map.

6. The target detection and stereo matching based environment perception method of claim 4, wherein, The position leg X S The step of introducing into the spatial interaction, to weight the important information on each channel of the spatial interaction, comprises: The position branch X S introduced into the spatial interaction, generating a spatial attention image; The spatial attention image is mapped to the branch passed through 3*3 standard convolution to generate the weight ω2 of spatial information, and the calculation formula is: ω2=R(F(X S )); R represents an activation function, F(X S ) represents a spatial attention map converted from the position branch feature X S .

7. The target detection and stereo matching based environment perception method of claim 4, wherein, residual connecting the first fusion branch and the second fusion branch to obtain a feature X FCM the steps are: using the formula X FCM = (X C ω2)⊕(X S ω1) to obtain the feature X FCM ; where ω1 is an allocation weight of important information on each channel of channel interaction, and ω2 is an allocation weight of spatial information on each channel of spatial interaction. 8.The method of environment perception based on target detection and stereo matching according to claim 1, characterized in that, The step of inputting the local image circled by the position information of the target in the recognition image into the improved stereo matching model adopting the multi-channel independent matching strategy to perform matching cost calculation and obtain the disparity value of the local image where the target is located comprises the steps of: acquiring a total pixel area of the recognition image and an area of a target pixel region of each target; According to the proportion of the area of the target pixel region of each target in the total pixel area, determine the maximum disparity value n of each target; According to the maximum disparity value n, determine the left-right image horizontal disparity d of each target in the identification image, d=0,1,2……n; Separate the image color three channels of the local image circled by the position information of the target in the identification image, and independently perform matching calculation of all targets on each color channel based on the left-right image horizontal disparity d, to obtain the matching cost of three color channels of each target; For the matching cost of three color channels of each target, compare the matching cost of three color channels under the same disparity value of left-right image horizontal disparity d, and select the minimum matching cost as the final matching cost of the target under the same disparity value of left-right image horizontal disparity d. Compare the final matching cost of left-right image horizontal disparity d under all disparity values, and select the minimum matching cost as the disparity value of the local image where the target is located.

9. The target detection and stereo matching based environment perception method of claim 8, wherein, The step of performing matching calculation of all targets independently on each color channel based on the horizontal disparity d of the left and right images is: using the formula calculating the pixel point in the matching cost of the cth channel ; wherein, R is the red channel among the three color channels, G is the green channel among the three color channels, and B is the blue channel among the three color channels, is the pixel point in the local image circled by the position information of the target in the left image of the identification image in the gray value of the channel , is the pixel point in the local image circled by the position information of the target in the right image of the identification image in the gray value of the channel . The step of comparing the matching costs of the three color channels under the same disparity value of the horizontal disparity d of the left and right images to select the minimum matching cost as the final matching cost of the target under the same disparity value of the horizontal disparity d of the left and right images for each target is: using the formula The matching costs of the three color channels under the same disparity value of the horizontal disparity d of the left and right images are compared to select the minimum matching cost as the final matching cost of the target under the same disparity value of the horizontal disparity d of the left and right images ; The step of comparing the final matching cost of the target at all disparity values of the horizontal disparity d of the left and right images to select the minimum matching cost as the disparity value of the local image where the target is located is: using the formula The disparity value of the local image where the target is located is calculated .

10. The target detection and stereo matching based environment perception method of claim 9, wherein, The steps of converting the disparity value of the local image where the target is located into the depth information of the target in combination with the calibration parameters of the binocular camera in the environment perception system are as follows: Using the formula to convert the disparity values of the local image where the target is located into depth values by the calibration parameters of the binocular camera in the environment perception system ; wherein, is the focal length in the binocular camera calibration parameters in the environment perception system, is the baseline length in the binocular camera calibration parameters in the environment perception system.

Images