Obstacle detection method, electronic device, storage medium and program product

Abstract

The application provides an obstacle detection method, an electronic device, a storage medium and a program product, aiming to improve the accuracy and range of obstacle detection and reduce the cost. The method comprises: obtaining a plurality of images of different perspectives taken by a camera of a current vehicle; performing feature extraction on the plurality of images respectively to obtain a plurality of image features; converting the plurality of image features into a plurality of image features of the same perspective, obtaining a feature map based on the plurality of image features of the same perspective; and performing target detection based on the feature map to obtain a target detection result, wherein the target detection result comprises type and / or position information of an obstacle.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an obstacle detection method, electronic device, storage medium, and program product. Background Technology

[0002] In autonomous driving processes, such as driving or parking, ultrasonic sensors or 2D / 3D images are typically used for obstacle detection.

[0003] In related technologies, ultrasonic sensors have limited detection range and angle for obstacles, resulting in a small detection range for obstacle detection methods based on ultrasonic sensors; obstacle detection methods based on 2D images cannot capture the 3D contours of obstacles, resulting in low accuracy of the obstacle detection results; obstacle detection schemes based on 3D images require point clouds acquired by LiDAR as input, resulting in high implementation costs and making them unsuitable for mass production and widespread adoption.

[0004] Based on the above technical analysis, existing obstacle detection methods suffer from drawbacks such as small detection range, high implementation cost, and low detection accuracy. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide an obstacle detection method, electronic device, storage medium, and program product to overcome or at least partially solve the above problems.

[0006] A first aspect of this application provides an obstacle detection method, comprising:

[0007] Obtain multiple images from different perspectives captured by the vehicle's cameras;

[0008] Feature extraction is performed on the multiple images to obtain multiple image features;

[0009] The multiple image features are converted into multiple image features from the same viewpoint, and a feature map is obtained based on the multiple image features from the same viewpoint;

[0010] Target detection is performed based on the feature map to obtain target detection results, which include: obstacle type and / or location information.

[0011] Optionally, the method further includes:

[0012] Obtain point cloud data collected by the vehicle's onboard radar;

[0013] The point cloud data is extracted to obtain a point cloud feature map;

[0014] The target detection based on the feature map, to obtain the target detection result, includes:

[0015] The point cloud feature map is fused with the feature map to obtain a fused feature map;

[0016] The fused feature map is processed to obtain the target detection result.

[0017] Optionally, the method further includes:

[0018] The point cloud space corresponding to the point cloud data is divided into multiple spatial sub-blocks, and the point cloud data corresponding to each spatial sub-block is obtained;

[0019] The step of extracting point cloud data to obtain a point cloud feature map includes:

[0020] Feature extraction is performed on the point cloud data corresponding to each of the aforementioned spatial sub-blocks to obtain point cloud feature maps of multiple spatial sub-blocks;

[0021] The step of fusing the point cloud feature map with the feature map to obtain a fused feature map includes:

[0022] Based on the spatial mapping relationship between the multiple spatial sub-blocks and the viewpoint of the feature map, the point cloud feature map of each spatial sub-block is fused with the corresponding feature sub-map in the feature map to obtain a fused feature map.

[0023] Optionally, converting the plurality of image features into multiple image features from the same viewpoint includes:

[0024] The obstacle detection model converts the multiple image features into multiple image features from the same viewpoint.

[0025] The obstacle detection model is trained based on image samples at a single time point;

[0026] The image samples at each time point include image samples taken from multiple viewpoints, and the model parameters of the obstacle detection model include at least: model parameters for converting the image features of the image samples taken from multiple viewpoints into corresponding image features of the same viewpoint.

[0027] Optionally, the target detection result is obtained through the obstacle detection model and the feature map;

[0028] The obstacle detection model is also trained based on radar point cloud samples that are time-synchronized with the image samples;

[0029] The radar point cloud samples synchronized with the image samples are obtained through the following methods:

[0030] A radar point cloud map carrying obstacle tags is obtained, wherein the radar point cloud map is generated based on radar point cloud data of a sample vehicle at multiple times, and the obstacle tags are tags of obstacles appearing around the sample vehicle during the multiple times.

[0031] Based on the positioning data of the sample vehicle at the multiple times, the radar point cloud map carrying obstacle labels is mapped to the multiple times to obtain the obstacle labels for each of the multiple times. Alternatively, the obstacle labels for each of the multiple times are obtained from the radar point cloud map carrying obstacle labels. The obstacle label for each time is the label of the obstacle appearing around the sample vehicle at that time.

[0032] Based on the obstacle labels at each of the multiple time points, the radar point cloud data at each of the multiple time points are labeled to obtain radar point cloud samples carrying obstacle labels at the multiple time points.

[0033] Image samples captured by the vehicle-mounted camera of the sample vehicle at multiple times are obtained, and the image samples at multiple times are synchronized with radar point cloud samples carrying obstacle tags at multiple times.

[0034] Optionally, obtaining the radar point cloud map carrying obstacle tags includes:

[0035] The radar point cloud data of the sample vehicle at multiple times are stitched together to obtain an initial point cloud map;

[0036] Based on image samples captured by the camera of the sample vehicle at the multiple times, the initial point cloud map is filled with color to obtain a color-filled point cloud map, in which different obstacles are represented by different colors;

[0037] Obtain a radar point cloud map with obstacle labels after the user has marked obstacles on the color-filled point cloud map.

[0038] Optionally, for each of the plurality of time points, before synchronizing the image samples at the plurality of time points with the radar point cloud samples carrying obstacle tags at the plurality of time points, the method further includes:

[0039] Based on the radar point cloud sample carrying obstacle tags at that moment, determine the depth information of the obstacle corresponding to the obstacle tag from multiple viewpoints;

[0040] Based on the determined depth information, image samples taken at that moment from multiple viewpoints are labeled to obtain image samples with depth labels from multiple viewpoints.

[0041] The step of synchronizing the image samples at the multiple time points with the radar point cloud samples carrying obstacle tags at the multiple time points includes:

[0042] Synchronize the image samples carrying depth labels at multiple times and at multiple viewpoints with the radar point cloud samples carrying obstacle labels at multiple times in time;

[0043] The model parameters of the obstacle detection model are updated based on the depth label and the obstacle label.

[0044] Optionally, the multiple image features from the same viewpoint are multiple image features from the same top-down viewpoint.

[0045] Optionally, the location information includes: the location information of regularly shaped obstacles in the image, and / or the image area covered by irregularly shaped obstacles in the image.

[0046] A second aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the obstacle detection method disclosed in the embodiments of this application.

[0047] A third aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the obstacle detection method disclosed in the embodiments of this application.

[0048] A fourth aspect of this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the obstacle detection method disclosed in the embodiments of this application.

[0049] The embodiments of this application have the following advantages:

[0050] In this embodiment, obstacle detection is performed based on images captured by the camera from different perspectives, which can increase the scope of obstacle detection and reduce the implementation cost. Furthermore, image features corresponding to each perspective are obtained from each captured image. After converting each image feature into the same perspective, target detection is performed on the obstacle, enabling the target detection to capture the 3D contour of the obstacle, thereby improving the accuracy of obstacle detection. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of the steps of an obstacle detection method according to an embodiment of this application;

[0053] Figure 2 This is a flowchart illustrating the steps of a laser point cloud-visual combined obstacle detection method in an embodiment of this application;

[0054] Figure 3 This is a flowchart of the steps of an obstacle detection method for laser point cloud-assisted training in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of the structure of an obstacle detection device according to an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0057] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0059] Understandably, the performance of obstacle detection has a significant impact on the safe application of autonomous driving technology. With the widespread use of artificial intelligence, current obstacle detection primarily relies on ultrasonic sensors or 2D / 3D images.

[0060] Ultrasonic sensors emit ultrasonic beams within a certain range and then process the echoes to determine the distance to obstacles. The detection range is typically between 0.2m and 5m, and the detection coverage angle is limited, resulting in a small detection range for obstacle detection schemes based on ultrasonic sensors.

[0061] Obstacle detection schemes based on 2D images mainly detect the type and location of 2D targets in the images captured by the camera. Then, based on the assumption that the target is on the ground, the target is projected onto the world coordinate system using the camera's intrinsic and extrinsic parameters to obtain the actual location of the target. However, since 2D image-based obstacle detection schemes cannot capture the 3D contours of obstacles, they can only rely on ground assumptions and camera intrinsic and extrinsic parameters to roughly project the approximate location of the bottom of the obstacle onto the world coordinate system, resulting in low accuracy of the obstacle detection results.

[0062] Obstacle detection solutions based on 3D images require point clouds acquired by LiDAR as input. However, LiDAR is expensive, resulting in high implementation costs and making the solution unsuitable for mass production and widespread adoption.

[0063] This application provides an obstacle detection method, referring to... Figure 1 As shown, a flowchart illustrating the steps of an obstacle detection method according to an embodiment of this application is illustrated. Figure 1 As shown, the obstacle detection method includes the following steps:

[0064] Step S11: Obtain multiple images from different perspectives captured by the current vehicle's camera.

[0065] For example, multiple cameras are mounted around the vehicle body, and these multiple cameras are used to capture images from the vehicle's front view, side view, and rear view, respectively, to obtain multiple images captured by the cameras from different perspectives.

[0066] It is understood that the vehicle's camera can be one or more on-board cameras installed in the vehicle. These one or more on-board cameras capture images from different perspectives, obtaining images from multiple perspectives. Using these multiple images as input to the obstacle detection method provided in this application embodiment can avoid the limitation of small detection range caused by using ultrasonic sensors, and using a camera instead of LiDAR can greatly reduce costs. Therefore, the obstacle detection method based on camera images provided in this application embodiment can improve the obstacle detection range and reduce costs.

[0067] Step S12: Extract features from the multiple images to obtain multiple image features.

[0068] In this embodiment, common feature extraction methods such as wavelet transform and least squares can be used to extract image features such as color features, texture features and spatial relationship features of multiple images to obtain multiple image features; alternatively, trained models such as ResNet can be used to extract image features, and this embodiment does not limit this.

[0069] Step S13: Convert the multiple image features into multiple image features from the same viewpoint, and obtain a feature map based on the multiple image features from the same viewpoint.

[0070] For example, image features from different viewpoints can be mapped to a feature map of the same viewpoint. This involves using a viewpoint transformation method to convert image features from different viewpoints into features from the same viewpoint, and then fusing these features to obtain the aforementioned feature map. Alternatively, the obtained image features from the same viewpoint can be directly used as the feature map for subsequent target detection steps. This viewpoint transformation method can be implemented using a multilayer perceptron structure or a transformer structure, or it can be implemented using a depth estimation method combined with the transformation matrix corresponding to the intrinsic and extrinsic parameters of the sensor (such as a camera).

[0071] Optionally, the multiple image features from the same viewpoint are multiple image features from the same top-down viewpoint.

[0072] It is understandable that using a top-down view of the feature map is beneficial for subsequent capture and localization of the 3D contour of the obstacle (such as the obstacle height and the actual contact area between the obstacle and the ground), thereby further ensuring the detection accuracy of the obstacle detection method provided in the embodiments of this application.

[0073] Step S14: Perform target detection based on the feature map to obtain target detection results, which include: obstacle type and / or location information.

[0074] Optionally, the location information includes: the location information of regularly shaped obstacles in the image, and / or the image area covered by irregularly shaped obstacles in the image.

[0075] It should be noted that obstacles with regular shapes refer to obstacles with relatively fixed overall shapes, such as cars and pillars, while obstacles with irregular shapes refer to obstacles with more varied overall shapes, such as flower beds and walls.

[0076] The following explanation uses an obstacle detection scenario involving a car and flower beds as an example. Cameras installed on the front, left, right, and rear sides of the vehicle capture images of the scene around the vehicle, obtaining images from the vehicle's front-view, rear-view, left-view, and right-view perspectives. Feature extraction is performed on these four images to obtain their respective image features. All image features from these four images are then input into a trained viewpoint transformation model to obtain a feature map of the same top-down view. Object detection is performed on this top-down view to identify the obstacle types present, such as flower beds and pillars. The coverage area of ​​the flower beds in the image is marked using color or ID, and the location area of ​​the pillars in the image is marked using a bounding box (position box), resulting in a detection result that includes the location and type of the obstacles.

[0077] In the above embodiments, obstacles can be marked on images taken from multiple perspectives and converted to the same top-down perspective, or obstacles can be marked directly on multiple images taken from different perspectives.

[0078] As one possible implementation, the image taken from the same top-down viewpoint used for obstacle marking is a feature map.

[0079] In this embodiment, the feature map, which contains image features from multiple perspectives, is used to detect obstacles. This allows the target detection to capture the 3D contour of the obstacle. Based on this 3D contour, positional information such as the obstacle's height and the actual contact area with the ground can be obtained. Then, this positional information is transformed into coordinates (e.g., to the world coordinate system) to obtain the actual position of the obstacle with high precision. Therefore, compared to traditional 2D image obstacle detection schemes that can only capture 2D features of obstacles and coarsely project the approximate position of the obstacle's bottom onto world coordinates, the obstacle detection method provided in this embodiment can greatly improve the accuracy of detecting the location and type of obstacles.

[0080] The technical solution of this application embodiment is to perform obstacle detection based on images captured by the camera from different perspectives. This can increase the range of obstacle detection and reduce the implementation cost. Furthermore, by converting the image features corresponding to each perspective into the same perspective before performing target detection on the obstacle, the target detection can capture the 3D contour of the obstacle, thereby improving the accuracy of obstacle detection.

[0081] Although obstacle detection can be achieved based solely on visual images (i.e., camera images), considering the limited depth information reflected by camera images from different perspectives, obstacle detection can also be achieved by combining visual images with point cloud data.

[0082] As one possible implementation, the method further includes:

[0083] Obtain point cloud data collected by the vehicle's onboard radar;

[0084] The point cloud data is extracted to obtain a point cloud feature map;

[0085] The target detection based on the feature map, to obtain the target detection result, includes:

[0086] The point cloud feature map is fused with the feature map to obtain a fused feature map;

[0087] The fused feature map is processed to obtain the target detection result.

[0088] like Figure 2The diagram shows a flowchart of a laser point cloud-visual joint obstacle detection method provided in an embodiment of this application, including: sequentially extracting features and transforming feature perspectives from the input multi-view images to obtain feature maps; extracting features from the input laser point cloud data to obtain point cloud feature maps; and fusing visual and laser features based on the point cloud feature maps and feature maps to obtain a fused feature map for obstacle detection.

[0089] It is understood that all or part of the steps of the above-described laser point cloud-visual joint obstacle detection method can be implemented by the obstacle detection model provided in this application embodiment, and the remaining steps of the method can be implemented using existing model structures with corresponding functions. This embodiment does not limit the number or type of models.

[0090] In such Figure 2 In the illustrated embodiment, the obstacle detection model requires point cloud data collected by LiDAR as input during both the training and inference phases. That is, the point cloud data and camera image data are used together as the observation data for the obstacle detection method. The point cloud data and camera image data are fused to obtain a fused feature map. After the obstacle detection model is trained, the obstacle detection model uses the fused feature map to replace the feature map in steps S13 to S14 and executes the relevant steps in S11 to S14 to obtain the target detection result of the obstacle.

[0091] It should be noted that all or part of the steps involved in obtaining the fused feature map based on point cloud data and camera image data can be executed by the obstacle detection model or by a model with corresponding functions in the existing technology. This embodiment does not limit this.

[0092] Optionally, the method further includes:

[0093] The point cloud space corresponding to the point cloud data is divided into multiple spatial sub-blocks, and the point cloud data corresponding to each spatial sub-block is obtained;

[0094] The step of extracting point cloud data to obtain a point cloud feature map includes:

[0095] Feature extraction is performed on the point cloud data corresponding to each of the aforementioned spatial sub-blocks to obtain point cloud feature maps of multiple spatial sub-blocks;

[0096] The step of fusing the point cloud feature map with the feature map to obtain a fused feature map includes:

[0097] Based on the spatial mapping relationship between the multiple spatial sub-blocks and the viewpoint of the feature map, the point cloud feature map of each spatial sub-block is fused with the corresponding feature sub-map in the feature map to obtain a fused feature map.

[0098] It is understandable that the above spatial mapping relationship can be obtained based on the correspondence between spatial sub-blocks, feature sub-graphs and actual scene location points. That is, spatial sub-blocks and feature sub-graphs corresponding to the same location in the scene can be considered to have a spatial mapping relationship.

[0099] For example, a point cloud space is determined to describe the location of points in the point cloud data. This point cloud space is then divided into W*H*L sub-blocks (spatial sub-blocks), where W is the number of sub-blocks in the width range, H is the number of sub-blocks in the height range, and L is the number of sub-blocks in the length range. The point cloud data is then distributed among these sub-blocks to obtain the point cloud data for each sub-block. Feature extraction is then performed on the point cloud data of each sub-block to obtain point cloud feature maps for multiple spatial sub-blocks. The extracted point cloud feature maps are then fused with the feature maps obtained after viewpoint transformation, such as by direct addition or concatenation, to obtain a fused feature map.

[0100] Optionally, converting the plurality of image features into multiple image features from the same viewpoint includes:

[0101] The obstacle detection model converts the multiple image features into multiple image features from the same viewpoint.

[0102] The obstacle detection model is trained based on image samples at a single time point;

[0103] The image samples at each time point include image samples taken from multiple viewpoints, and the model parameters of the obstacle detection model include at least: model parameters for converting the image features of the image samples taken from multiple viewpoints into corresponding image features of the same viewpoint.

[0104] In this embodiment, model parameters for viewpoint switching are set for the obstacle detection model, enabling the obstacle detection model to have a learnable viewpoint switching function, thereby effectively reducing information errors or loss in the process of unifying multiple views.

[0105] Optionally, the target detection result is obtained through the obstacle detection model and the feature map;

[0106] The obstacle detection model is also trained based on radar point cloud samples that are time-synchronized with the image samples;

[0107] The radar point cloud samples synchronized with the image samples are obtained through the following methods:

[0108] A radar point cloud map carrying obstacle tags is obtained, wherein the radar point cloud map is generated based on radar point cloud data of a sample vehicle at multiple times, and the obstacle tags are tags of obstacles appearing around the sample vehicle during the multiple times.

[0109] Based on the positioning data of the sample vehicle at the multiple times, the radar point cloud map carrying obstacle labels is mapped to the multiple times to obtain the obstacle labels for each of the multiple times. Alternatively, the obstacle labels for each of the multiple times are obtained from the radar point cloud map carrying obstacle labels. The obstacle label for each time is the label of the obstacle appearing around the sample vehicle at that time.

[0110] Based on the obstacle labels at each of the multiple time points, the radar point cloud data at each of the multiple time points are labeled to obtain radar point cloud samples carrying obstacle labels at the multiple time points.

[0111] Image samples captured by the vehicle-mounted camera of the sample vehicle at multiple times are obtained, and the image samples at multiple times are synchronized with radar point cloud samples carrying obstacle tags at multiple times.

[0112] For example, all obstacle labels carried by the radar point cloud map, such as A1 and B1, can be processed through coordinate transformation and other methods to obtain obstacle labels such as A2 and B2 at a certain moment; alternatively, the radar point cloud data used to generate the radar point cloud map at that moment can be found, and based on the obstacle A described by the radar point cloud data, the obstacle label A1 corresponding to obstacle A can be found based on the radar point cloud map, and the obstacle label A2 at that moment can be obtained through coordinate transformation and other methods.

[0113] It should be noted that this embodiment can use a sample vehicle equipped with positioning and sensing devices (such as LiDAR and multiple vehicle-mounted cameras) to collect point cloud data, image data, and positioning data at various moments during the vehicle's operation. Then, based on the point cloud data collected at multiple moments, a high-precision point cloud map is reconstructed using SLAM technology. Obstacle-related information is then labeled on the point cloud map. After labeling, the obstacle-related information (such as location) is mapped back to the vehicle coordinate system as training data for the model.

[0114] Traditional single-frame perception information annotation, which involves annotating obstacle information separately for point cloud data at each time step, suffers from the drawback of requiring multiple annotations of the same obstacle-related information collected at different times. However, the obstacle-related information annotation method provided in this application maps the annotated point cloud map to different time steps based on the vehicle's pose, thereby generating a large amount of training data. For example, mapping a radar point cloud map carrying the world coordinate position information of obstacles to the vehicle's coordinate system at different times allows for obtaining obstacle position information at multiple time steps, thus avoiding repetitive annotation steps and improving the efficiency of training sample acquisition.

[0115] Optionally, obtaining the radar point cloud map carrying obstacle tags includes:

[0116] The radar point cloud data of the sample vehicle at multiple times are stitched together to obtain an initial point cloud map;

[0117] Based on image samples captured by the camera of the sample vehicle at the multiple times, the initial point cloud map is filled with color to obtain a color-filled point cloud map, in which different obstacles are represented by different colors;

[0118] Obtain a radar point cloud map with obstacle labels after the user has marked obstacles on the color-filled point cloud map.

[0119] For example, the sample vehicle collects data in three modalities: vehicle positioning information, LiDAR point cloud data, and onboard camera image data. Vehicle positioning information includes at least one of IMU (Inertial Measurement Unit) data and GNSS / INS integrated navigation data. LiDAR point cloud data is 3D point cloud data scanned by the onboard LiDAR. The acquisition time of each modality is recorded, and data from the three modalities with a time difference less than a threshold are matched (i.e., timestamped) to obtain time-synchronized data for the three modalities.

[0120] The point cloud map is then constructed using various methods. For example, GNSS / INS combined navigation data can be used directly as vehicle positioning information. In scenarios with poor GPS signal, such as underground parking garages, LiDAR and IMU fusion positioning can be used to determine vehicle positioning information. Based on this vehicle positioning information, the point cloud data at various times are stitched together. Then, using the extrinsic parameters of LiDAR and cameras, as well as the intrinsic parameters of each camera, the point cloud map is filled with color for subsequent labeling.

[0121] As one possible implementation, after the point cloud map is constructed, obstacle-related information can be manually labeled directly on the point cloud map. The labeling can use a 3D bounding box to surround the obstacle and mark the obstacle category.

[0122] As another possible implementation, obstacles can be detected in the constructed point cloud in advance by using an obstacle detection model. The detection results output by the model are then rendered on the point cloud map as pre-labeled results. The annotators only need to verify and adjust the pre-labeled results to obtain a radar point cloud map with obstacle labels, which greatly improves the annotation efficiency.

[0123] Optionally, for each of the plurality of time points, before synchronizing the image samples at the plurality of time points with the radar point cloud samples carrying obstacle tags at the plurality of time points, the method further includes:

[0124] Based on the radar point cloud sample carrying obstacle tags at that moment, determine the depth information of the obstacle corresponding to the obstacle tag from multiple viewpoints;

[0125] Based on the determined depth information, image samples taken at that moment from multiple viewpoints are labeled to obtain image samples with depth labels from multiple viewpoints.

[0126] The step of synchronizing the image samples at the multiple time points with the radar point cloud samples carrying obstacle tags at the multiple time points includes:

[0127] Synchronize the image samples carrying depth labels at multiple times and at multiple viewpoints with the radar point cloud samples carrying obstacle labels at multiple times in time;

[0128] The model parameters of the obstacle detection model are updated based on the depth label and the obstacle label.

[0129] like Figure 3 The diagram shows a flowchart of an obstacle detection method using laser point cloud-assisted training, provided in an embodiment of this application. In this method, the laser point cloud is only used for model training; point cloud data is not used during the model's inference phase. Obstacle detection is performed solely based on the acquired visual images. Figure 3 In the illustrated embodiment, the obstacle detection model uses point cloud data collected by LiDAR as supervisory information (depth supervision) during the training phase. This supervisory information is used to provide depth information to the 2D images captured from various viewpoints. In other words, the obstacle detection model iteratively updates its parameters based on the aforementioned depth labels during the training phase, enabling it to predict the depth distribution of images captured from various viewpoints, thereby improving the model's perception accuracy. The trained obstacle detection model is then used to execute the relevant steps S11 to S14 (including the steps provided in the above embodiment that enable feature extraction and feature viewpoint transformation) to obtain the obstacle detection result.

[0130] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0131] Figure 4 This is a schematic diagram of the structure of an obstacle detection device according to an embodiment of this application. The obstacle detection device includes an acquisition module, a first processing module, a second processing module, and a detection module, wherein:

[0132] The acquisition module is used to obtain multiple images from different perspectives captured by the current vehicle's camera;

[0133] The first processing module is used to extract features from the multiple images respectively to obtain multiple image features;

[0134] The second processing module is used to convert the multiple image features into multiple image features from the same viewpoint, and to obtain a feature map based on the multiple image features from the same viewpoint.

[0135] The detection module is used to perform target detection based on the feature map and obtain target detection results, which include: obstacle type and / or location information.

[0136] Optionally, the acquisition module is further configured to acquire point cloud data collected by the vehicle-mounted radar of the current vehicle;

[0137] The first processing module is further configured to extract the point cloud data to obtain a point cloud feature map;

[0138] The second processing module is further configured to fuse the point cloud feature map with the feature map to obtain a fused feature map;

[0139] The detection module is also used to process the fused feature map to obtain the target detection result.

[0140] Optionally, a third processing module may also be included;

[0141] The third processing module is used to divide the point cloud space corresponding to the point cloud data into multiple spatial sub-blocks and obtain the point cloud data corresponding to each spatial sub-block.

[0142] The first processing module is further configured to extract features from the point cloud data corresponding to each of the spatial sub-blocks to obtain point cloud feature maps of multiple spatial sub-blocks.

[0143] The second processing module is further configured to fuse the point cloud feature map of each spatial sub-block with the corresponding feature sub-map in the feature map based on the spatial mapping relationship between the multiple spatial sub-blocks and the viewpoint of the feature map, to obtain a fused feature map.

[0144] Optionally, the second processing module is further configured to convert the multiple image features into multiple image features from the same viewpoint using an obstacle detection model;

[0145] The obstacle detection model is trained based on image samples at a single time point;

[0146] The image samples at each time point include image samples taken from multiple viewpoints, and the model parameters of the obstacle detection model include at least: model parameters for converting the image features of the image samples taken from multiple viewpoints into corresponding image features of the same viewpoint.

[0147] Optionally, the target detection result is obtained through the obstacle detection model and the feature map;

[0148] The obstacle detection model is also trained based on radar point cloud samples that are time-synchronized with the image samples;

[0149] It also includes a fourth processing module;

[0150] The fourth processing module is used to obtain a radar point cloud map carrying obstacle tags. The radar point cloud map is generated based on radar point cloud data of the sample vehicle at multiple times. The obstacle tags are the tags of obstacles that appear around the sample vehicle during the multiple times.

[0151] Based on the positioning data of the sample vehicle at the multiple times, the radar point cloud map carrying obstacle labels is mapped to the multiple times to obtain the obstacle labels for each of the multiple times. Alternatively, the obstacle labels for each of the multiple times are obtained from the radar point cloud map carrying obstacle labels. The obstacle label for each time is the label of the obstacle appearing around the sample vehicle at that time.

[0152] Based on the obstacle labels at each of the multiple time points, the radar point cloud data at each of the multiple time points are labeled to obtain radar point cloud samples carrying obstacle labels at the multiple time points.

[0153] Image samples captured by the vehicle-mounted camera of the sample vehicle at multiple times are obtained, and the image samples at multiple times are synchronized with radar point cloud samples carrying obstacle tags at multiple times.

[0154] Optionally, the fourth processing module is specifically used to stitch together the radar point cloud data of the sample vehicle at multiple times to obtain an initial point cloud map.

[0155] Based on image samples captured by the camera of the sample vehicle at the multiple times, the initial point cloud map is filled with color to obtain a color-filled point cloud map, in which different obstacles are represented by different colors;

[0156] Obtain a radar point cloud map with obstacle labels after the user has marked obstacles on the color-filled point cloud map.

[0157] Optionally, the fourth processing module is further configured to, for each of the plurality of times, mark the radar point cloud samples taken at that time from multiple perspectives according to the determined depth information, so as to obtain image samples with depth labels from multiple perspectives.

[0158] The step of synchronizing the image samples at the multiple time points with the radar point cloud samples carrying obstacle tags at the multiple time points includes:

[0159] Synchronize the image samples carrying depth labels at multiple times and at multiple viewpoints with the radar point cloud samples carrying obstacle labels at multiple times in time;

[0160] The model parameters of the obstacle detection model are updated based on the depth label and the obstacle label.

[0161] Optionally, the multiple image features from the same viewpoint are multiple image features from the same top-down viewpoint.

[0162] Optionally, the location information includes: the location information of regularly shaped obstacles in the image, and / or the image area covered by irregularly shaped obstacles in the image.

[0163] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the method embodiments.

[0164] This application also provides an electronic device, see embodiments thereof. Figure 5 , Figure 5 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 5 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the obstacle detection method disclosed in the embodiments of this application.

[0165] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the obstacle detection method disclosed in this application.

[0166] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the obstacle detection method disclosed in this application.

[0167] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0173] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0174] The foregoing has provided a detailed description of the obstacle detection method, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An obstacle detection method characterized by, The method comprises: obtaining a plurality of images of different perspectives captured by a camera of a current vehicle; performing feature extraction on the plurality of images respectively to obtain a plurality of image features; converting the plurality of image features into a plurality of image features of the same perspective, and obtaining a feature map based on the plurality of image features of the same perspective; performing target detection based on the feature map to obtain a target detection result, the target detection result comprising: type and / or position information of an obstacle; the target detection result is obtained through an obstacle detection model and the feature map; the obstacle detection model is trained based on radar point cloud samples that are time-synchronized with image samples; the radar point cloud samples that are time-synchronized with the image samples are obtained by: obtaining a radar point cloud map carrying obstacle labels, the radar point cloud map being generated according to radar point cloud data of a sample vehicle at a plurality of time points, the obstacle labels being labels of obstacles appearing around the sample vehicle during the plurality of time points; mapping the radar point cloud map carrying the obstacle labels to the plurality of time points based on positioning data of the sample vehicle at the plurality of time points to obtain respective obstacle labels of the plurality of time points, or obtaining respective obstacle labels of the plurality of time points from the radar point cloud map carrying the obstacle labels, the obstacle label of each time point being a label of an obstacle appearing around the sample vehicle at the time point; labeling respective radar point cloud data of the plurality of time points according to the respective obstacle labels of the plurality of time points to obtain radar point cloud samples carrying obstacle labels of the plurality of time points; obtaining image samples captured by a vehicle-mounted camera of the sample vehicle at the plurality of time points, and time-synchronizing the image samples of the plurality of time points with the radar point cloud samples carrying obstacle labels of the plurality of time points.

2. The method of claim 1, wherein, The method further comprises: obtaining point cloud data collected by a vehicle-mounted radar of the current vehicle; performing extraction on the point cloud data to obtain a point cloud feature map; the target detection based on the feature map to obtain a target detection result comprises: fusing the point cloud feature map and the feature map to obtain a fused feature map; processing the fused feature map to obtain the target detection result.

3. The method of claim 2, wherein, The method further comprises: dividing a point cloud space corresponding to the point cloud data into a plurality of spatial sub-blocks to obtain point cloud data corresponding to each spatial sub-block; the extraction on the point cloud data to obtain a point cloud feature map comprises: performing feature extraction on point cloud data corresponding to each spatial sub-block respectively to obtain a point cloud feature map of each spatial sub-block; the fusion of the point cloud feature map and the feature map to obtain a fused feature map comprises: based on a spatial mapping relationship between the plurality of spatial sub-blocks and the perspective of the feature map, fusing the point cloud feature map of each spatial sub-block and a feature sub-map corresponding thereto in the feature map to obtain a fused feature map.

4. The method according to any of claims 1 to 3, characterized in that, the conversion of the plurality of image features into a plurality of image features of the same perspective comprises: converting the plurality of image features into a plurality of image features of the same perspective through an obstacle detection model; The obstacle detection model is also trained based on image samples at a single time point; The image samples at each time point include image samples taken at multiple perspectives, and the model parameters of the obstacle detection model at least include model parameters for converting image features of the image samples taken at the multiple perspectives into corresponding image features at the same perspective.

5. The method of claim 1, wherein, The radar point cloud map carrying the obstacle label is obtained, including: The radar point cloud data of the sample vehicle at multiple time points is spliced to obtain an initial point cloud map; According to the image samples taken by the camera of the sample vehicle at the multiple time points, the initial point cloud map is color-filled to obtain a color-filled point cloud map, and different obstacles in the color-filled point cloud map are represented by different colors; The radar point cloud map carrying the obstacle label is obtained, including:

6. The method of claim 1, wherein, For the radar point cloud sample at each time point of the multiple time points, before the image samples at the multiple time points and the radar point cloud samples carrying the obstacle label at the multiple time points are time-synchronized, the method further includes: According to the radar point cloud sample carrying the obstacle label at the time point, depth information of the obstacle corresponding to the obstacle label at multiple perspectives is determined; According to the determined depth information, the image samples taken at the multiple perspectives at the time point are respectively marked to obtain image samples carrying depth labels at the multiple perspectives; The radar point cloud samples carrying the obstacle label at the multiple time points and the image samples carrying the depth labels at the multiple perspectives at the multiple time points are time-synchronized. The radar point cloud samples carrying the obstacle label at the multiple time points and the image samples carrying the depth labels at the multiple perspectives at the multiple time points are time-synchronized. The model parameters of the obstacle detection model are updated according to the depth labels and the obstacle labels.

7. The method of claim 1, wherein, The multiple image features at the same perspective are multiple image features at the same top view perspective.

8. The method of claim 1, wherein, The position information includes position information of regular-shaped obstacles in the image and / or image regions covered by irregular-shaped obstacles in the image.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-8. The processor executes the computer program to implement the obstacle detection method of any one of claims 1-8.

10. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the obstacle detection method of any one of claims 1-8.

11. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the obstacle detection method of any one of claims 1-8.

Images