Vehicle driving obstacle distance detection method and system based on vehicle-mounted depth camera

Abstract

The application provides a vehicle driving obstacle distance detection method and system based on a vehicle-mounted depth camera, and relates to the technical field of vehicle obstacle detection.The method comprises the following steps: acquiring RGB image information, depth point cloud information and 3D coordinate information of an obstacle to be identified during vehicle driving; extracting the geometric features of key points in the RGB image information, and identifying the obstacle; aligning and integrating the 3D coordinate information and the depth point cloud information according to timestamps, and combining the identified obstacle to obtain 3D coordinates of the obstacle with timestamps; learning the spatial features of the 3D coordinates of the obstacle with timestamps at different scales, and performing feature extraction to obtain the distance between the obstacle and the vehicle body.The depth camera installed on the vehicle body is used to acquire the RGB image information, depth point cloud information and 3D coordinate information of the obstacle to be identified during vehicle driving, and a hybrid network model is used to classify and regress after different feature information is extracted, so that the effects of obstacle identification and obstacle-vehicle body distance prediction can be efficiently and accurately considered.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle obstacle detection, and in particular relates to a method and system for detecting the distance of a vehicle to an obstacle based on a vehicle-mounted depth camera. Background Art

[0002] Currently, the challenges faced by vehicle-mounted obstacle detection technology are mainly reflected in the limitations of its various components and the detection subject itself, such as:

[0003] Although inexpensive, ultrasonic radar has limited ability to identify obstacles at long distances and in complex environments. It also reacts slowly to high-speed objects, potentially leading to lags or misjudgments. Ultrasonic radar is also affected by factors such as its transmission angle, placement, and design. In practice, software often modifies the ultrasonic radar envelope, preventing optimal sensor utilization.

[0004] Although lidar provides high-precision three-dimensional spatial information, its high equipment cost, large size and requirement for high-performance computer resources make its deployment and maintenance complicated, and it is inefficient when processing large-scale data.

[0005] Although millimeter-wave radar has strong penetration ability and certain medium-range detection capabilities, its accuracy is relatively low. It is not as accurate as lidar in identifying the shape and size of the target, and there may be blind spots in certain directions.

[0006] Infrared thermal imagers have unique advantages at night and in low-temperature environments, but their resolution is low and they mainly focus on detecting temperature differences, which limits their ability to identify object details and capture color information.

[0007] In summary, when the existing technology uses the above-mentioned devices to detect vehicle obstacles, there are some deficiencies caused by the limitations of the devices themselves, and it is difficult to simultaneously meet the goals of efficient and accurate recognition. Summary of the Invention

[0008] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a method and system for detecting the distance to obstacles while a vehicle is moving based on a vehicle-mounted depth camera. The method uses a depth camera installed on the vehicle body to obtain RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is moving. A hybrid network model is used to extract different feature information and then perform classification and regression. This method can achieve both efficient and accurate obstacle identification and distance prediction between the obstacle and the vehicle body.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] A first aspect of the present invention provides a method for detecting the distance to an obstacle while a vehicle is traveling based on a vehicle-mounted depth camera.

[0011] A method for detecting the distance of a vehicle to an obstacle based on a vehicle-mounted depth camera includes the following steps:

[0012] Obtain RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is moving;

[0013] Extract the geometric features of key points in RGB image information and identify obstacles;

[0014] Align and integrate the 3D coordinate information and depth point cloud information according to the timestamp, and combine it with the identified obstacles to obtain the 3D coordinates of the obstacles with timestamps;

[0015] The spatial features of the obstacle’s 3D coordinates with timestamps at different scales are learned and feature extracted to obtain the distance between the obstacle and the vehicle body.

[0016] As a further technical solution, after obtaining the distance between the obstacle and the vehicle body, the following is also included:

[0017] Pre-set multiple warning distance levels between obstacles and the vehicle body;

[0018] Determine the specific alarm distance level that the distance between the obstacle and the vehicle body falls into, and issue a corresponding alarm message.

[0019] As a further technical solution, a depth camera installed on the vehicle body is used to obtain RGB image information, depth point cloud information and 3D coordinate information of obstacles to be identified while the vehicle is moving.

[0020] As a further technical solution, the geometric features of key points in the RGB image information are extracted to identify obstacles therein, specifically including:

[0021] Transposed convolution is used to process RGB images to achieve supersampling;

[0022] The super-sampling information of the RGB image is input into the graph neural network model to extract the geometric features of the key points. The residual block is introduced into the graph neural network model to solve the gradient vanishing and exploding problems in deep neural networks.

[0023] The geometric features of the key points in the extracted RGB image information are input into the residual convolutional neural network model to identify and classify the RGB image information to obtain the obstacle recognition results.

[0024] As a further technical solution, the 3D coordinate information and depth point cloud information are aligned and integrated according to the timestamp, and combined with the identified obstacles to obtain the 3D coordinates of the obstacles with timestamps. Specifically, it includes:

[0025] Convert 3D coordinate information and depth point cloud information to the same coordinate system, and use point cloud alignment technology to achieve timestamp alignment operation;

[0026] Use the optical flow fusion algorithm to fuse 3D coordinate information and depth point cloud information, and perform filtering operations on the fused data;

[0027] Based on the filtered data and combined with the specific location of the identified obstacle, the 3D coordinates of the obstacle with a timestamp are obtained.

[0028] As a further technical solution, we learn the spatial features of the obstacle's 3D coordinates with timestamps at different scales and perform feature extraction to obtain the distance between the obstacle and the vehicle body. Specifically, we include:

[0029] The 3D coordinates of the obstacles with timestamps are input into the temporal convolutional network model. Dilated Convolutions are first used to process the image time series to increase the receptive field. Residual modules are added between network layers to identify and classify RGB image information, enabling the network to learn deeper features. A bidirectional TCN is added in the middle of the network layer, and the past and future information of the sequence is simultaneously utilized to improve the model's predictive ability and obtain obstacle recognition results.

[0030] As a further technical solution, it also includes:

[0031] Through the conversion between world coordinates, the obtained camera coordinates of the obstacle are converted into two-dimensional pixel coordinates, and the corresponding obstacle visual warning information is displayed on the two-dimensional RGB image.

[0032] A second aspect of the present invention provides a vehicle obstacle distance detection system based on a vehicle-mounted depth camera.

[0033] The vehicle obstacle distance detection system based on the vehicle-mounted depth camera includes:

[0034] The information acquisition module is configured to: acquire RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is traveling;

[0035] The obstacle recognition module is configured to: extract geometric features of key points in the RGB image information and identify obstacles therein;

[0036] The integration module is configured to: align and integrate the 3D coordinate information and the depth point cloud information according to the timestamp, and combine it with the identified obstacles to obtain the 3D coordinates of the obstacles with the timestamp;

[0037] The feature learning module is configured to learn the spatial features of the obstacle's 3D coordinates at different scales with timestamps, and perform feature extraction to obtain the distance between the obstacle and the vehicle body.

[0038] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the method for detecting the distance to an obstacle while the vehicle is traveling based on a vehicle-mounted depth camera as described in the first aspect of the present invention.

[0039] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for detecting the distance to an obstacle in a vehicle based on a vehicle-mounted depth camera as described in the first aspect of the present invention are implemented.

[0040] One or more of the above technical solutions have the following beneficial effects:

[0041] The present invention provides a method and system for detecting the distance to obstacles while a vehicle is moving based on a vehicle-mounted depth camera. The method uses a depth camera installed on the vehicle body to obtain RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is moving. A hybrid network model is used to extract different feature information and then perform classification and regression. This method can achieve both efficient and accurate obstacle identification and distance prediction between the obstacle and the vehicle body.

[0042] The hybrid network model proposed in this paper includes a graph neural network model, a residual convolutional neural network model, and a temporal convolutional network model. The graph neural network model is used to extract the geometric features of key points in RGB image information. The extracted geometric features of key points in the RGB image information are input into the residual convolutional neural network model to identify and classify the RGB image information to obtain obstacle recognition results. The 3D coordinates of the obstacles with timestamps are input into the temporal convolutional network model. Multiple stacked temporal convolutional layers are used to extract features from the 3D coordinates of the obstacles with timestamps, and spatial features at different scales are learned. Through this hybrid network model, more useful information can be integrated to accurately predict the distance to the obstacle.

[0043] The application applies the idea of residual network to the traditional graph convolutional neural network, the residual block can solve the gradient vanishing and explosion problem in the deep neural network, so that the network can learn deeper features more effectively. Before the image data input is recognized, the algorithm will first adopt the Transposed convolution to process the image, and the function can realize the super sampling, so that the network can generate high-resolution output.

[0044] The application fuses a plurality of time sequence convolutional networks, first adopts Dilated Convolutions (dilated convolution) to process the image time sequence to increase the receptive field, so that the model can capture longer time dependence; residual modules are added between network layers to help solve the gradient vanishing problem in deep network, so that the network can learn deeper features. In addition, the idea of bidirectional TCN network is added in the middle of the network layer, specifically, the bidirectional TCN is added in the middle of the network layer to utilize the past and future information of the sequence, thereby improving the prediction ability of the model.

[0045] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the application and are incorporated herein for explanation by illustrating a preferred embodiment of the application.

[0047] Figure 1 The method flowchart of the first embodiment. DETAILED DESCRIPTION

[0048] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0049] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the application.

[0050] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0051] Embodiment one

[0052] Before the method for preventing rear-end collision of the automobile provided by the embodiments of the application is explained in detail, the application scenarios provided by the embodiments of the application are introduced.

[0053] On-board obstacle detection is an important aspect of intelligent vehicle identification. Existing methods for on-board obstacle detection using ultrasonic radar, lidar, millimeter-wave radar, and infrared thermal imagers often suffer from limitations in the equipment itself, resulting in detection results that cannot balance efficiency, accuracy, and cost savings.

[0054] Based on such an application scenario, an embodiment of the present application provides a method for detecting the distance of a vehicle obstacle based on a vehicle-mounted depth camera. The vehicle-mounted depth camera installed on the vehicle body is used to collect information, and the collected information is processed using a hybrid network model to obtain the distance between the obstacle and the vehicle body. Based on this, an alarm is issued to provide a certain warning to the driver.

[0055] Next, the vehicle obstacle distance detection method based on the vehicle-mounted depth camera provided in the embodiment of the present application will be explained in detail with reference to the accompanying drawings.

[0056] like Figure 1 As shown, the method for detecting the distance of a vehicle to an obstacle based on a vehicle-mounted depth camera includes the following steps:

[0057] Obtain RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is moving;

[0058] Extract the geometric features of key points in RGB image information and identify obstacles;

[0059] Align and integrate the 3D coordinate information and depth point cloud information according to the timestamp, and combine it with the identified obstacles to obtain the 3D coordinates of the obstacles with timestamps;

[0060] The spatial features of the obstacle’s 3D coordinates with timestamps at different scales are learned and feature extracted to obtain the distance between the obstacle and the vehicle body.

[0061] In this application, a hybrid network model is designed, including a graph neural network model, a residual convolutional neural network model, and a temporal convolutional network model. It performs feature extraction and obstacle recognition on the RGB image information, depth point cloud information, and 3D coordinate information obtained by the vehicle-mounted depth camera, and ultimately obtains accurate distance information between the obstacle and the vehicle body. Specifically, it includes:

[0062] In some embodiments, after obtaining the distance between the obstacle and the vehicle body, the method further includes:

[0063] Pre-set multiple warning distance levels between obstacles and the vehicle body;

[0064] Determine the specific alarm distance level that the distance between the obstacle and the vehicle body falls into, and issue a corresponding alarm message.

[0065] It can be understood that a depth camera installed on the vehicle body can be used to obtain RGB image information, depth point cloud information and 3D coordinate information of obstacles to be identified while the vehicle is driving.

[0066] In some embodiments, extracting geometric features of key points in RGB image information and identifying obstacles therein specifically includes:

[0067] Transposed convolution is used to process RGB images to achieve supersampling;

[0068] The super-sampling information of the RGB image is input into the graph neural network model to extract the geometric features of the key points. The residual block is introduced into the graph neural network model to solve the gradient vanishing and exploding problems in deep neural networks.

[0069] The geometric features of the key points in the extracted RGB image information are input into the residual convolutional neural network model to identify and classify the RGB image information to obtain the obstacle recognition results.

[0070] In some embodiments, the 3D coordinate information and the depth point cloud information are aligned and integrated according to the timestamp, and combined with the identified obstacles to obtain the 3D coordinates of the obstacles with timestamps, specifically including:

[0071] Convert 3D coordinate information and depth point cloud information to the same coordinate system, and use point cloud alignment technology to achieve timestamp alignment operation;

[0072] Use the optical flow fusion algorithm to fuse 3D coordinate information and depth point cloud information, and perform filtering operations on the fused data;

[0073] Based on the filtered data and combined with the specific location of the identified obstacle, the 3D coordinates of the obstacle with a timestamp are obtained.

[0074] In some embodiments, spatial features of different scales of the obstacle's 3D coordinates with timestamps are learned and feature extraction is performed to obtain the distance between the obstacle and the vehicle body, specifically including:

[0075] The 3D coordinates of the obstacles with timestamps are input into the temporal convolutional network model. Dilated Convolutions are first used to process the image time series to increase the receptive field. Residual modules are added between network layers to identify and classify RGB image information, enabling the network to learn deeper features. A bidirectional TCN is added in the middle of the network layer, and the past and future information of the sequence is simultaneously utilized to improve the model's predictive ability and obtain obstacle recognition results.

[0076] In some embodiments, further comprising:

[0077] The obtained camera coordinates of the obstacles are converted into two-dimensional pixel coordinates through conversion between world coordinates, and corresponding obstacle visual warning information is displayed on a two-dimensional RGB image.

[0078] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and the embodiments of the present application will not be described one by one.

[0079] More specifically:

[0080] (1) The improvement of the graph neural network model structure in this embodiment is that the idea of residual network is applied to the traditional graph convolutional neural network. The residual block can solve the gradient vanishing and explosion problem in the deep neural network, so that the network can learn deeper features more effectively. In addition, before the image data input is recognized, the algorithm will first use the Transposed convolution to process the image. This function can realize super sampling, so that the network can generate high-resolution output.

[0081] (2) The improvement of the time series convolution network model is that multiple time series convolution networks are fused. The specific structure includes: first, Dilated Convolutions (dilated convolution) is used to process the image time series to increase the receptive field, so that the model can capture longer time dependence. Similarly, residual modules are added between network layers to help solve the gradient vanishing problem in deep networks, so that the network can learn deeper features. In addition, the idea of adding bidirectional TCN network in the middle of the network layer is added, which specifically adds bidirectional TCN in the middle of the network layer to utilize past and future information of the sequence, thereby improving the prediction ability of the model.

[0082] The 3D coordinates of the obstacles with timestamps are input into the time series convolution network model, and the 3D coordinates of the obstacles with timestamps are extracted through multiple stacked time series convolution layers to learn spatial features of different scales. The data processing process specifically uses different convolution kernels to perform convolution operations on the image to obtain feature attributes.

[0083] (3) The 3D coordinate information and the depth point cloud information are aligned and integrated according to the timestamps, and combined with the recognized obstacles, the 3D coordinates of the obstacles with timestamps are obtained, wherein:

[0084] The 3D coordinate information and the depth point cloud information are integrated, specifically: first, the 3D coordinate information and the depth point cloud information need to be converted to the same coordinate system.

[0085] Next, point cloud alignment technology is used, utilizing the transformation matrix between the two datasets. This matrix transforms one dataset into the reference coordinate system of the other. Then, an optical flow fusion algorithm is used to fuse the 3D coordinate information and the depth point cloud information. Because the fused data contains noise, two filtering algorithms, Gaussian filtering and median filtering, are used to improve data quality.

[0086] Finally, the fused data can be visualized and analyzed using 3D visualization tools to extract useful information or perform further processing.

[0087] This embodiment uses a depth camera installed on the car to capture RGB image information and depth point cloud information. The graph convolutional neural network can extract key point geometric features from the RGB information in the image and combine it with the residual convolutional neural network structure to identify and classify the image information. In the data preprocessing stage, the obtained 3D coordinates, depth information and timestamps are integrated to improve the diversity and completeness of the data set.

[0088] This embodiment uses a mixture of multiple network models to extract the obtained information, and uses GNN to perform dense sampling evenly at different locations in the image;

[0089] Convolution is used to predict and extract test results from different feature maps. The homogeneous transformation matrix obtained by convolution can establish the relationship between the two-dimensional mask and the three-dimensional point. This method uses the camera projection matrix algorithm to project the feature center of each voxel onto the image plane.

[0090] The temporal convolutional network (TCN) is used to extract features of 3D coordinates with timestamps. By stacking multiple TCN layers, the network can learn spatial features of different scales.

[0091] After using the hybrid network model to extract different feature information, classification and regression are performed. After the target is confirmed, the NMS algorithm is performed to filter out the prediction boxes with large overlap. The remaining prediction boxes are the target locations we have identified.

[0092] Finally, the algorithm defines the need to categorize and classify the frequency and level of obstacle distance warnings. By converting the camera coordinates of obstacles into 2D pixel coordinates through world coordinate conversion, the corresponding obstacle visual warning information is generated on the 2D RGB image. When obstacles are within different detection ranges, different levels of warning are issued on the monitoring map. Simultaneously, the distance between the vehicle and the obstacle is displayed on the image using 3D coordinate information.

[0093] Example 2

[0094] This embodiment discloses a vehicle obstacle distance detection system based on a vehicle-mounted depth camera.

[0095] A vehicle driving obstacle distance detection system based on a vehicle-mounted depth camera, comprising:

[0096] An information acquisition module configured to acquire RGB image information, depth point cloud information and 3D coordinate information of an obstacle to be identified in vehicle driving;

[0097] An obstacle identification module configured to extract geometric features of key points in the RGB image information and identify the obstacle therein;

[0098] An integration module configured to align and integrate the 3D coordinate information and the depth point cloud information according to time stamps, and obtain 3D coordinates of the obstacle with time stamps in combination with the identified obstacle;

[0099] A feature learning module configured to learn spatial features of the 3D coordinates of the obstacle with time stamps in different scales, and perform feature extraction to obtain a distance between the obstacle and the vehicle body.

[0100] Embodiment Three

[0101] An object of the embodiment is to provide a computer-readable storage medium.

[0102] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the vehicle driving obstacle distance detection method based on a vehicle-mounted depth camera according to Embodiment 1 of the present disclosure.

[0103] Embodiment Four

[0104] An object of the embodiment is to provide an electronic device.

[0105] An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, the processor implementing the steps in the vehicle driving obstacle distance detection method based on a vehicle-mounted depth camera according to Embodiment 1 of the present disclosure when executing the program.

[0106] The steps and methods involved in the above embodiments two, three and four correspond to Embodiment One, and the specific embodiments can be referred to the relevant description part of Embodiment One. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any method in the present disclosure.

[0107] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0108] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A vehicle obstacle distance detection method based on a vehicle-mounted depth camera is characterized in that: The following steps are involved: Obtain RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is moving; Extract the geometric features of key points in RGB image information and identify obstacles; Align and integrate the 3D coordinate information and depth point cloud information according to the timestamp, and combine it with the identified obstacles to obtain the 3D coordinates of the obstacles with timestamps; Learn the spatial features of the obstacle's 3D coordinates at different scales with timestamps, perform feature extraction, and obtain the distance between the obstacle and the vehicle body; The spatial features of the obstacle's 3D coordinates with timestamps at different scales are learned and extracted to obtain the distance between the obstacle and the vehicle body. Specifically, the following are performed: The 3D coordinates of the obstacles with timestamps are input into the temporal convolutional network model. Dilated Convolutions are first used to process the image time series to increase the receptive field. Residual modules are added between network layers to identify and classify RGB image information, enabling the network to learn deeper features. A bidirectional TCN is added in the middle of the network layer, and the past and future information of the sequence is simultaneously utilized to improve the model's predictive ability and obtain obstacle recognition results.

2. The method for detecting the distance to an obstacle while the vehicle is traveling based on a vehicle-mounted depth camera according to claim 1, wherein: After obtaining the distance between the obstacle and the vehicle, it also includes: Pre-set multiple warning distance levels between obstacles and the vehicle body; Determine the specific alarm distance level that the distance between the obstacle and the vehicle body falls into, and issue a corresponding alarm message.

3. The method for detecting the distance to an obstacle while the vehicle is traveling based on a vehicle-mounted depth camera according to claim 1, wherein: A depth camera installed on the vehicle body is used to obtain RGB image information, depth point cloud information and 3D coordinate information of obstacles to be identified while the vehicle is moving.

4. The method for detecting the distance to an obstacle while the vehicle is traveling based on a vehicle-mounted depth camera according to claim 1, wherein: Extract the geometric features of key points in the RGB image information and identify obstacles, including: Transposed convolution is used to process RGB images to achieve supersampling; The super-sampling information of the RGB image is input into the graph neural network model to extract the geometric features of the key points. The residual block is introduced into the graph neural network model to solve the gradient vanishing and exploding problems in deep neural networks. The geometric features of the key points in the extracted RGB image information are input into the residual convolutional neural network model to identify and classify the RGB image information to obtain the obstacle recognition results.

5. The method for detecting the distance to an obstacle while the vehicle is traveling based on a vehicle-mounted depth camera according to claim 1, wherein: Align and integrate the 3D coordinate information and depth point cloud information according to the timestamp, and combine it with the identified obstacles to obtain the 3D coordinates of the obstacles with timestamps. Specifically, it includes: Convert 3D coordinate information and depth point cloud information to the same coordinate system, and use point cloud alignment technology to achieve timestamp alignment operation; Use the optical flow fusion algorithm to fuse 3D coordinate information and depth point cloud information, and perform filtering operations on the fused data; Based on the filtered data and combined with the specific location of the identified obstacle, the 3D coordinates of the obstacle with a timestamp are obtained.

6. The method for detecting the distance to an obstacle while the vehicle is traveling based on a vehicle-mounted depth camera according to claim 1, wherein: Also includes: Through the conversion between world coordinates, the obtained camera coordinates of the obstacle are converted into two-dimensional pixel coordinates, and the corresponding obstacle visual warning information is displayed on the two-dimensional RGB image.

7. The vehicle obstacle distance detection system based on the vehicle-mounted depth camera is characterized by: include: The information acquisition module is configured to: acquire RGB image information, depth point cloud information, and 3D coordinate information of obstacles to be identified while the vehicle is traveling; The obstacle recognition module is configured to: extract geometric features of key points in the RGB image information and identify obstacles therein; The integration module is configured to: align and integrate the 3D coordinate information and the depth point cloud information according to the timestamp, and combine it with the identified obstacles to obtain the 3D coordinates of the obstacles with the timestamp; The feature learning module is configured to learn the spatial features of the obstacle's 3D coordinates at different scales with timestamps, perform feature extraction, and obtain the distance between the obstacle and the vehicle body. The spatial features of the obstacle's 3D coordinates with timestamps at different scales are learned and extracted to obtain the distance between the obstacle and the vehicle body. Specifically, the following are performed: The 3D coordinates of the obstacles with timestamps are input into the temporal convolutional network model. Dilated Convolutions are first used to process the image time series to increase the receptive field. Residual modules are added between network layers to identify and classify RGB image information, enabling the network to learn deeper features. A bidirectional TCN is added in the middle of the network layer, and the past and future information of the sequence is simultaneously utilized to improve the model's predictive ability and obtain obstacle recognition results.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for detecting the distance to an obstacle when the vehicle is traveling based on a vehicle-mounted depth camera are implemented as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the vehicle obstacle distance detection method based on a vehicle-mounted depth camera are implemented as described in any one of claims 1 to 6.

Images