Obstacle identification method and electronic equipment

Abstract

The invention discloses an obstacle identification method and electronic equipment, and the method comprises the steps: extracting a target image from a synchronization frame, carrying out the obstacle identification based on the target image, obtaining the information of a target obstacle, extracting a corresponding target point cloud from the synchronization frame based on the information of the target obstacle, and carrying out the obstacle identification based on the target point cloud. And obtaining a target obstacle in the synchronization frame. Therefore, according to the method and the device, obstacle detection is performed on the target image, the target point cloud is determined from the point cloud data according to the detection result, and finally the target obstacle in the synchronization frame can be identified according to the target point cloud, so that the obstacle in the point cloud data can be determined without a point cloud detection model; under the condition of avoiding the adoption of a point cloud detection model, the purpose of identifying the obstacle in the point cloud data can be achieved by consuming less computing resources.

Description

Technical Field

[0001] The present application belongs to the field of point cloud processing technology, and in particular relates to a method and electronic device for identifying obstacles. Background Art

[0002] With the rapid development of image acquisition technology, sensors such as lidar and millimeter-wave radar have unique advantages in environmental perception tasks, especially when combined with vision (camera) systems, which can significantly improve the robustness of the system. This makes these sensors widely used in image acquisition scenarios in the field of autonomous driving technology.

[0003] In the field of autonomous driving technology, the recognition of obstacles in images is crucial. Generally, obstacle recognition is performed on point cloud data collected by sensors such as lidar and millimeter-wave radar. Point cloud detection models are usually used to identify obstacles in point cloud data, but the use of point cloud detection models requires large computing resources.

[0004] Therefore, a method is proposed to achieve the purpose of identifying obstacles in point cloud data with less computing resources, which has great application significance and practical value. Summary of the Invention

[0005] The embodiments of the present application provide a method and electronic device for identifying obstacles, which can achieve the purpose of identifying obstacles in point cloud data while consuming less computing resources.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying an obstacle, the method comprising:

[0007] Acquiring a synchronized frame based on data from a plurality of sensors, the plurality of sensors including at least one first target camera and a target radar, the synchronized frame including an image from the first target camera and a point cloud from the target radar, wherein the point cloud from the target radar and the image from the first target camera are synchronized in time and correspond to the same scene;

[0008] Extracting a target image from the synchronization frame;

[0009] performing obstacle recognition based on the target image to obtain target obstacle information;

[0010] Extracting a corresponding target point cloud in the synchronous frame based on the target obstacle information, where the target point cloud is at least a portion of the point cloud in the synchronous frame;

[0011] Obstacle identification is performed based on the target point cloud to obtain the target obstacle in the synchronous frame.

[0012] In some embodiments, the above-mentioned acquisition of a synchronization frame based on data from multiple sensors includes: aligning the acquisition time of the data of the at least one first target camera with the acquisition time of the point cloud as a reference to obtain a synchronization frame; projecting the point cloud onto the image plane captured by the second target camera according to the external parameters of the target radar and the second target camera and the internal parameters of the second target camera to obtain a point cloud image; and placing the point cloud image into a synchronization frame to obtain the synchronization frame of the fused point cloud.

[0013] In some embodiments, the target image includes a first image captured by a pinhole camera, and the performing obstacle recognition based on the target image to obtain target obstacle information includes: performing obstacle recognition based on the first image to obtain a two-dimensional frame of the first obstacle, and the target obstacle information includes the two-dimensional frame of the first obstacle; extracting a corresponding target point cloud from the synchronous frame based on the target obstacle information includes: extracting a corresponding first point cloud from the synchronous frame based on the two-dimensional frame of the first obstacle; and performing obstacle recognition based on the target point cloud to obtain the target obstacle in the synchronous frame includes: performing obstacle recognition based on the first point cloud to obtain the first obstacle in the synchronous frame, and the target obstacle includes the first obstacle.

[0014] In some embodiments, the above-mentioned obstacle recognition based on the first image to obtain the two-dimensional frame of the first obstacle includes: inputting the first image into a preset first neural network model to obtain the two-dimensional frame of the first obstacle predicted and output by the first neural network model.

[0015] In some embodiments, the above-mentioned two-dimensional frame based on the first obstacle, extracting the corresponding first point cloud in the synchronous frame, includes: based on the two-dimensional frame of the first obstacle, determining the obstacle area corresponding to the two-dimensional frame of the first obstacle from the point cloud image in the synchronous frame; obtaining the point cloud in the obstacle area to obtain the first point cloud.

[0016] In some embodiments, the target image includes a second image captured by a fisheye camera, and the performing obstacle identification based on the target image to obtain target obstacle information includes: performing obstacle identification based on the second image to obtain coordinate information of a second obstacle, wherein the target obstacle information includes the coordinate information of the second obstacle; extracting a corresponding target point cloud from the synchronous frame based on the target obstacle information includes: extracting a corresponding second point cloud from the synchronous frame based on the coordinate information of the second obstacle; and performing obstacle identification based on the target point cloud to obtain a target obstacle in the synchronous frame includes: performing obstacle identification based on the second point cloud to obtain a second obstacle in the synchronous frame, wherein the target obstacle includes the second obstacle.

[0017] In some embodiments, the above-mentioned obstacle recognition based on the second image to obtain the coordinate information of the second obstacle includes: inputting the second image into a preset second neural network model to obtain a semantic mask image output by the second neural network model; performing instance segmentation on the semantic mask image to obtain the coordinate information of the instance in the semantic mask image, and the coordinate information of the instance is the coordinate information of the second obstacle.

[0018] In some embodiments, the above-mentioned extraction of the corresponding second point cloud in the synchronous frame based on the coordinate information of the second obstacle includes: based on the coordinate information of the second obstacle, obtaining the point cloud under the coordinate information from the point cloud image of the synchronous frame to obtain the second point cloud.

[0019] In some embodiments, the target obstacle includes a first obstacle and a second obstacle, the first obstacle and the second obstacle being obstacles identified based on different target images. After performing obstacle identification based on the target point cloud to obtain the target obstacle in the synchronous frame, the method further includes: fusing information about the first obstacle and information about the second obstacle to obtain fused obstacle information; based on the fused obstacle information, removing the target obstacle point cloud corresponding to the fused obstacle information from the point cloud to obtain a remaining point cloud after removing the target obstacle point cloud; performing three-dimensional meshing processing on the remaining point cloud to obtain a three-dimensional mesh; performing point cloud clustering processing on the three-dimensional mesh to obtain an obstacle point cloud; and performing obstacle identification on the obstacle point cloud to obtain a third obstacle.

[0020] In a second aspect, an embodiment of the present application provides a device for identifying obstacles, the device comprising:

[0021] A data acquisition module is configured to acquire synchronized frames based on data from a plurality of sensors, the plurality of sensors including at least one first target camera and a target radar, the synchronized frames including an image of the first target camera and a point cloud of the target radar, wherein the point cloud of the target radar is synchronized with the image of the first target camera in time, and the point cloud of the target radar and the image of the first target camera correspond to the same scene.

[0022] The image extraction module is used to extract the target image from the synchronization frame.

[0023] The image recognition module is used to identify obstacles based on the target image and obtain target obstacle information.

[0024] The point cloud extraction module is used to extract the corresponding target point cloud in the synchronous frame based on the target obstacle information, and the target point cloud is at least a part of the point cloud in the synchronous frame.

[0025] The obstacle recognition module is used to identify obstacles based on the target point cloud and obtain the target obstacles in the synchronous frame.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for identifying obstacles as described in the first aspect is implemented.

[0027] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for identifying obstacles as described in the first aspect is implemented.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for identifying obstacles as described in the first aspect.

[0029] The obstacle identification method, apparatus, device, computer storage medium, and computer program product of the embodiments of the present application can extract a target image from a synchronous frame, perform obstacle identification based on the target image, obtain target obstacle information, extract a corresponding target point cloud from the synchronous frame based on the target obstacle information, perform obstacle identification based on the target point cloud, and obtain the target obstacle in the synchronous frame. In this way, the present application performs obstacle detection on the target image, determines the obstacle point cloud (target point cloud) from the point cloud data based on the detection results, and finally identifies the target obstacle in the synchronous frame based on the target point cloud, so as to achieve the purpose of determining obstacles in the point cloud data without using a point cloud detection model. In the case of avoiding the use of a point cloud detection model, less computing resources can be consumed to achieve the purpose of identifying obstacles in the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 1 is a flow chart of a method for identifying obstacles provided in an embodiment of the present application;

[0032] Figure 2 This is a flow chart of a multi-sensor fusion perception system provided by an embodiment of the present application;

[0033] Figure 3 Schematic diagram of the structure of the device for identifying obstacles provided in an embodiment of the present application;

[0034] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0036] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0037] To solve the related technical problems, the embodiments of the present application provide a method, apparatus, device, computer storage medium, and computer program product for identifying obstacles. The following first introduces the method for identifying obstacles provided by the embodiments of the present application.

[0038] Figure 1 FIG. 1 is a flow chart of a method for identifying obstacles provided by an embodiment of the present application. Figure 1 As shown, the method specifically includes steps S101 to S105.

[0039] Step S101: Acquire a synchronization frame based on data from multiple sensors.

[0040] The above-mentioned sensor is a sensor that can be used for imaging, and the above-mentioned multiple sensors include at least one first target camera and a target radar.

[0041] The first target camera may be an infrared / thermal imaging camera, a stereo / binocular camera, a global shutter camera, a pinhole camera (Pin camera) or a fisheye camera, etc.

[0042] The above-mentioned target radar can be a laser radar or a millimeter wave radar, etc.

[0043] The above-mentioned synchronized frame includes the image of the first target camera and the point cloud of the target radar, wherein the point cloud of the target radar and the image of the first target camera are synchronized in time, and the point cloud of the target radar and the image of the first target camera correspond to the same scene.

[0044] The point cloud of the target radar and the image of the first target camera are synchronized in time, which can be understood as the acquisition time of the point cloud of the target radar and the image of the first target camera is the same moment.

[0045] The point cloud of the target radar and the image of the first target camera correspond to the same scene. It can be understood that the point cloud of the target radar and the image of the first target camera capture the same scene.

[0046] The data from the above-mentioned multiple sensors may be data collected by the multiple sensors for the same scene at the same time, or data collected by the multiple sensors for the corresponding same scene at multiple time points, wherein each time point may correspond to a scene.

[0047] If the data of the above-mentioned multiple sensors are data collected by multiple sensors for the same scene at multiple times, then the above-mentioned synchronous frame obtained based on the data of multiple sensors can be the images and point clouds of the same scene at the same time obtained from the data of multiple sensors, and the synchronous frame is obtained by fusing the images and point clouds of the same scene at the same time obtained.

[0048] In one embodiment, the synchronization frame obtained by acquiring data based on multiple sensors can also be obtained by aligning the data acquisition time of the at least one first target camera with reference to the acquisition time of the point cloud to obtain a synchronization frame. Then, the point cloud is projected onto the plane of the image taken by the second target camera through the external parameters of the target radar and the second target camera and the internal parameters of the second target camera to obtain a point cloud image, and the point cloud image is placed in the synchronization frame to obtain a synchronization frame of the fused point cloud.

[0049] In the above point cloud image, each point cloud corresponds to a pixel of the point cloud image.

[0050] Step S102: extracting a target image from the synchronization frame.

[0051] The above-mentioned target image can be at least one image among the images corresponding to at least one first target camera in the above-mentioned synchronization frame, that is, the above-mentioned target image can include n target images, n is a positive integer greater than 0, and n can be equal to or less than the number of images contained in the above-mentioned synchronization frame.

[0052] The target image may also be any one of the images corresponding to at least one first target camera in the synchronous frame.

[0053] That is, the target image may be a single target image or may include multiple target images.

[0054] The target image may be extracted from the synchronized frames using a deep learning model, such as a target detection model, a semantic segmentation model, or an instance segmentation model.

[0055] The above-mentioned target detection model can be a YOLO series model, an SSD model, etc., the above-mentioned semantic segmentation model can be a Transformer-based model, a DeepLab series model, etc., and the above-mentioned instance segmentation model can be a SOLOv2 model, a Mask R-CNN model, etc.

[0056] Step S103: Obstacle recognition is performed based on the target image to obtain target obstacle information.

[0057] The target obstacle information may be information used to locate the position of the obstacle, such as a two-dimensional frame, coordinate information, and the like.

[0058] Step S104: extracting the corresponding target point cloud in the synchronous frame based on the target obstacle information.

[0059] The target point cloud is at least a portion of the point cloud in the synchronization frame, and the target point cloud is a point cloud determined based on target obstacle information.

[0060] The above-mentioned target obstacle information can be information used to locate the obstacle position. Then, based on the target obstacle information, the corresponding target point cloud is extracted in the synchronous frame, which can be the point cloud of the area where the target obstacle information is located extracted in the synchronous frame to obtain the target point cloud.

[0061] Step S105 : performing obstacle recognition based on the target point cloud to obtain the target obstacle in the synchronous frame.

[0062] The above-mentioned obstacle recognition based on the target point cloud to obtain the target obstacle in the synchronous frame may be performed by determining geometric feature information representing the target obstacle based on the target point cloud to obtain the target obstacle in the synchronous frame.

[0063] The above-mentioned geometric feature information may be the shape, size information, center position, etc. of the target obstacle.

[0064] In an embodiment of the present application, a target image is extracted from a synchronous frame, obstacle identification is performed based on the target image, target obstacle information is obtained, a corresponding target point cloud is extracted from the synchronous frame based on the target obstacle information, obstacle identification is performed based on the target point cloud, and the target obstacle in the synchronous frame is obtained. In this way, the present application performs obstacle detection on image data (target image) other than point cloud data, determines the obstacle point cloud (target point cloud) from the point cloud data based on the detection results, and ultimately identifies the target obstacle in the synchronous frame based on the target point cloud, thereby achieving the goal of determining obstacles in the point cloud data without the need for a point cloud detection model. By avoiding the use of a point cloud detection model, less computing resources can be consumed to achieve the purpose of identifying obstacles in the point cloud data.

[0065] In some embodiments, the target image may include a first image of a pinhole camera, and the obstacle recognition based on the target image to obtain target obstacle information may be performed based on the first image to obtain a two-dimensional frame of the first obstacle.

[0066] The first image is an image captured by a pinhole camera (Pin camera), and the target obstacle information includes a two-dimensional frame of the first obstacle.

[0067] The obstacle recognition based on the first image may be performed by inputting the first image into a preset first neural network model to obtain a two-dimensional frame of the first obstacle predicted and output by the first neural network model.

[0068] The above-mentioned first neural network model can be a target detection model, and the above-mentioned target detection model can be a model trained using a data set. The above-mentioned data set can include multiple images and multiple corresponding obstacle information. The above-mentioned data set can be data obtained based on experiments or actual historical data.

[0069] In one embodiment, the first neural network model can also predict the category of the output first obstacle.

[0070] In one embodiment, after obtaining the two-dimensional box and category of the first obstacle predicted and output by the first neural network model, the first obstacle can also be tracked based on the two-dimensional box and category of the first obstacle to obtain an identifier of the first obstacle, and the above identifier can be an ID.

[0071] In some embodiments, when the target obstacle information includes a two-dimensional frame of the first obstacle, the above-mentioned extraction of the corresponding target point cloud in the synchronous frame based on the target obstacle information may be based on the two-dimensional frame of the first obstacle, and the extraction of the corresponding first point cloud in the synchronous frame.

[0072] The above-mentioned two-dimensional frame based on the first obstacle extracts the corresponding first point cloud in the synchronous frame. It can be that based on the two-dimensional frame of the first obstacle, the obstacle area corresponding to the two-dimensional frame of the first obstacle is determined from the point cloud image in the synchronous frame, and the point cloud in the obstacle area is obtained to obtain the first point cloud.

[0073] In some embodiments, when the target point cloud includes the first point cloud, the above obstacle identification based on the target point cloud to obtain the target obstacle in the synchronous frame can also be performed based on the first point cloud to obtain the first obstacle in the synchronous frame.

[0074] The target obstacle mentioned above includes a first obstacle.

[0075] In this embodiment, image recognition using the Pin camera accurately captures the outline of obstacles in the image and generates a two-dimensional frame around the first obstacle. Based on this visual framework, the target point cloud region can be intelligently targeted, cleverly avoiding the reliance on complex point cloud detection models. Furthermore, integrating the Pin camera's high-definition image recognition capabilities with the depth information of point cloud data not only simplifies the detection process but also significantly improves obstacle detection accuracy and overall system reliability.

[0076] In some embodiments, the target image may include a second image from a fisheye camera. In this case, the obstacle identification based on the target image to obtain target obstacle information may also be performed based on the second image to obtain coordinate information of the second obstacle.

[0077] The target obstacle information includes coordinate information of the second obstacle.

[0078] The above-mentioned obstacle recognition based on the second image to obtain the coordinate information of the second obstacle can be performed by inputting the second image into a preset second neural network model to obtain a semantic mask image output by the second neural network model, performing instance segmentation on the semantic mask image to obtain the coordinate information of the instance in the semantic mask image, and the coordinate information of the instance is the coordinate information of the second obstacle.

[0079] The second neural network model mentioned above can be a semantic segmentation model.

[0080] Each pixel value in the above semantic mask image can represent the ID of a category.

[0081] The above-mentioned instance segmentation is performed on the semantic mask image to obtain the coordinate information of the instance in the semantic mask image, which may be input into the instance segmentation model to obtain the coordinate information of the instance output by the instance segmentation model.

[0082] The above examples can be understood as single object entities with independent semantics and spatial positions in the image, such as people, cars, dogs, etc.

[0083] In some embodiments, when the target obstacle information includes the coordinate information of the second obstacle, the above-mentioned extraction of the corresponding target point cloud in the synchronous frame based on the target obstacle information can also be based on the coordinate information of the second obstacle to extract the corresponding second point cloud in the synchronous frame.

[0084] The above-mentioned extraction of the corresponding second point cloud in the synchronous frame based on the coordinate information of the second obstacle can be performed by obtaining the point cloud under the coordinate information from the point cloud image of the synchronous frame based on the coordinate information of the second obstacle to obtain the second point cloud.

[0085] In some embodiments, when the target point cloud includes a second point cloud, the obstacle identification based on the target point cloud to obtain the target obstacle in the synchronous frame may also be performed based on the second point cloud to obtain the second obstacle in the synchronous frame.

[0086] The target obstacle includes a second obstacle.

[0087] In this embodiment, by leveraging the fisheye camera's wide field of view and employing sophisticated semantic segmentation techniques, the precise outline of the second obstacle is delineated in the image, and its key coordinate information is extracted. This coordinate information guides the targeting of the corresponding target point cloud region within the point cloud data, thereby bypassing the complex computations of traditional point cloud detection models and enabling more efficient obstacle identification in three-dimensional space. Furthermore, this method cleverly combines the semantic segmentation capabilities of the fisheye camera with the depth perception advantages of the point cloud: image segmentation provides precise two-dimensional positioning, while the point cloud imparts a three-dimensional spatial dimension. This combination not only enhances the robustness of obstacle detection but also makes the system's perception more acute and reliable.

[0088] In some embodiments, when the target image may include n target images, steps S102 to S105 are performed for each target image to obtain a target obstacle determined according to the multiple target images.

[0089] When a target obstacle is determined based on multiple target images, the method may further include the following steps:

[0090] The information of the target obstacle determined according to the multiple target images is fused to obtain fused obstacle information.

[0091] In one embodiment, the above-mentioned fusing of the target obstacle information determined based on the multiple target images to obtain the fused obstacle information may be performing deduplication processing on the target obstacle information determined based on the multiple target images to obtain the fused obstacle information that does not contain duplicate obstacle information.

[0092] In one embodiment, after the fused obstacle information is acquired, the final target obstacle in the synchronization frame may be determined according to the fused obstacle information.

[0093] Based on the fused obstacle information, the target obstacle point cloud corresponding to the fused obstacle information is removed from the point cloud to obtain the remaining point cloud after removing the target obstacle point cloud.

[0094] The remaining point clouds are processed into three-dimensional grids to obtain three-dimensional grids.

[0095] The above-mentioned three-dimensional meshing processing of the remaining point cloud to obtain the three-dimensional mesh may be performed by using a preset three-dimensional meshing method to perform three-dimensional meshing processing on the remaining point cloud to obtain the three-dimensional mesh.

[0096] The three-dimensional meshing method may be based on a Poisson Reconstruction method, a Delaunay triangulation (Alpha Shapes) method, or the like.

[0097] In one embodiment, the above-mentioned three-dimensional meshing process of the remaining point cloud to obtain the three-dimensional mesh may also be to voxelize the remaining point cloud using a preset voxelization method to obtain the three-dimensional voxel mesh.

[0098] The aforementioned preset voxelization method may be a direct voxelization (Binary Occupancy Grid) method, a voxel generation method based on deep learning (such as 3D CNN), and the like.

[0099] Perform point cloud clustering on the three-dimensional grid to obtain the obstacle point cloud.

[0100] The above-mentioned point cloud clustering process is performed on the three-dimensional grid to obtain the obstacle point cloud. This process may be performed on the three-dimensional grid using a preset point cloud clustering method to obtain the obstacle point cloud.

[0101] The point cloud clustering method may be a Euclidean distance clustering method (DBSCAN / Euclidean Clustering), a K-Means / K-Medoids clustering method, a normal vector / curvature clustering method, or the like.

[0102] In this embodiment, by fusing target obstacle information detected based on multiple different target images to obtain the final target obstacle, the final target obstacle can be determined by combining multiple target images, thereby improving the effectiveness of identifying obstacles in synchronous frames.

[0103] In a case where the target image includes a first image from a pinhole camera and a second image from a fisheye camera, the target obstacle includes a first obstacle and a second obstacle.

[0104] The first obstacle and the second obstacle are obstacles identified based on different target images. The first obstacle is an obstacle identified by performing obstacle identification on the first image of the pinhole camera, and the second obstacle is an obstacle identified by performing obstacle identification on the second image of the fisheye camera.

[0105] That is, the target obstacle information determined based on the plurality of target images includes the first obstacle information and the second obstacle information.

[0106] After obstacle identification is performed based on the target point cloud to obtain the target obstacle in the synchronized frame, the information of the target obstacle determined based on the multiple target images is fused to obtain the fused obstacle information. This may be the fusion of the information of the first obstacle and the information of the second obstacle to obtain the fused obstacle information.

[0107] In this embodiment, by fusing the information based on the first obstacle and the information based on the second obstacle to obtain the final target obstacle, it is possible to achieve dual recognition based on the image detection of the Pin camera and the semantic segmentation detection of the fisheye camera, combined with the depth information of the point cloud, to jointly identify the final target obstacle, thereby effectively improving the accuracy of obstacle detection.

[0108] In order to better implement the above method, the embodiment of the present application provides a multi-sensor fusion perception system, referring to Figure 2 , Figure 2 This is a flow chart of the multi-sensor fusion perception system provided in an embodiment of the present application. The multi-sensor fusion perception system includes a data acquisition module, a data preprocessing module, a Pin camera detection and tracking module, a fisheye camera segmentation and tracking module, a fusion module, and a point cloud detection and tracking module.

[0109] The data acquisition module is used to acquire data collected by multiple sensors and send the acquired data to the data preprocessing module.

[0110] The above-mentioned data preprocessing module is used to first cache the data of all sensors into a queue, then align the acquisition time of all first target cameras with the acquisition time of the point cloud of the target radar as a reference to obtain a multi-sensor synchronization frame (synchronization frame), and then project the laser point cloud (point cloud) onto the plane of the image taken by the second target camera through the external parameters of the target radar and the second target camera and the internal parameters of the second target camera to obtain a point cloud image, where each point cloud corresponds to a pixel of the point cloud image, put the point cloud image into the synchronization frame, and send the synchronization frame to the Pin camera detection and tracking module and the fisheye camera segmentation and tracking module.

[0111] The Pin camera detection and tracking module is configured to receive synchronized frame data, extract the Pin image (first image) from the Pin camera, and feed it into a neural network detection model (target detection model) for processing. After detecting and outputting the 2D bounding boxes and categories of all obstacles (first obstacles) in the Pin image, the module then outputs the tracked obstacle ID, 2D bounding box, and category based on the 2D bounding box and category. Finally, the module extracts the obstacle point cloud (first point cloud) from the point cloud image using the obstacle 2D bounding box, filters the obstacle point cloud (first point cloud) to obtain the final obstacle point cloud (first point cloud), and then calculates obstacle information (first obstacle information) such as polygons, size information, and center position based on the obstacle point cloud (first point cloud).

[0112] The fisheye camera segmentation and tracking module is used to receive data from synchronized frames, extract the fisheye image (second image) from the fisheye camera, and feed it into a neural network segmentation model (semantic segmentation model) for processing. After segmentation, a semantic mask image is output, in which each pixel value represents the ID of a category. The semantic mask image is then fed to the fisheye camera segmentation and tracking module for processing, and instance segmentation is performed to obtain the obstacle coordinates of each instance (coordinate information of the second obstacle). Finally, a segmented obstacle point cloud (second point cloud) is extracted from the point cloud image, and segmented obstacle information (second obstacle information) such as polygons, size information, and center position is calculated based on the segmented obstacle point cloud (second point cloud).

[0113] The above-mentioned fusion module is used to fuse the obstacles (first obstacle and second obstacle) detected by the Pin camera detection and tracking module and the fisheye camera segmentation and tracking module based on the obstacle information (first obstacle information and second obstacle information) to obtain image obstacle information (fused obstacle information), then remove the obstacle point cloud (target obstacle point cloud) to obtain a non-image obstacle point cloud (remaining point cloud), and finally provide the non-image obstacle point cloud (remaining point cloud) to the point cloud detection and tracking module for processing, and output image obstacle information (fused obstacle information).

[0114] The point cloud detection and tracking module is used to output the detected unknown obstacle point cloud based on the non-obstacle point cloud (remaining point cloud), finally track the obstacle to obtain the obstacle ID, and then calculate the point cloud obstacle information (information of the third obstacle) such as polygon, size information, and center position based on the unknown obstacle point cloud.

[0115] In order to better implement the above method, the embodiment of the present application also provides a device for identifying obstacles, referring to Figure 3 , Figure 3 A schematic diagram of the structure of the obstacle recognition device provided in the embodiment of the present application is shown as follows: Figure 3 As shown, the obstacle recognition device 30 specifically includes:

[0116] The data acquisition module 301 is configured to acquire synchronized frames based on data from multiple sensors, where the multiple sensors include at least one first target camera and a target radar. The synchronized frames include an image of the first target camera and a point cloud of the target radar, wherein the point cloud of the target radar is synchronized with the image of the first target camera in time, and the point cloud of the target radar and the image of the first target camera correspond to the same scene.

[0117] The image extraction module 302 is used to extract the target image from the synchronous frame.

[0118] The image recognition module 303 is used to identify obstacles based on the target image and obtain target obstacle information.

[0119] The point cloud extraction module 304 is configured to extract a corresponding target point cloud in the synchronous frame based on the target obstacle information, where the target point cloud is at least a portion of the point cloud in the synchronous frame.

[0120] The obstacle recognition module 305 is used to perform obstacle recognition based on the target point cloud to obtain the target obstacle in the synchronous frame.

[0121] In some embodiments, the above-mentioned data acquisition module 301 is specifically used to: align the data acquisition time of at least one first target camera with the acquisition time of the point cloud as a reference to obtain a synchronous frame; project the point cloud onto the image plane captured by the second target camera according to the external parameters of the target radar and the second target camera and the internal parameters of the second target camera to obtain a point cloud image; and place the point cloud image into the synchronous frame to obtain a synchronous frame of the fused point cloud.

[0122] In some embodiments, the target image includes a first image from a pinhole camera. The image recognition module 303 is specifically configured to perform obstacle recognition based on the first image to obtain a two-dimensional bounding box of the first obstacle, with the target obstacle information including the two-dimensional bounding box of the first obstacle. The image recognition module 303 is further configured to input the first image into a preset first neural network model to obtain a two-dimensional bounding box of the first obstacle as predicted and output by the first neural network model. The point cloud extraction module 304 is further configured to extract a corresponding first point cloud from the synchronized frame based on the two-dimensional bounding box of the first obstacle. The point cloud extraction module 304 is further configured to determine, from the point cloud image in the synchronized frame, an obstacle region corresponding to the two-dimensional bounding box of the first obstacle, and obtain a point cloud in the obstacle region to obtain a first point cloud. The obstacle recognition module 305 is further configured to perform obstacle recognition based on the first point cloud to obtain the first obstacle in the synchronized frame, with the target obstacle including the first obstacle.

[0123] In some embodiments, the target image includes a second image from a fisheye camera. The image recognition module 303 is specifically configured to perform obstacle recognition based on the second image to obtain coordinate information of the second obstacle, where the target obstacle information includes the coordinate information of the second obstacle. The image recognition module 303 is further configured to input the second image into a preset second neural network model to obtain a semantic mask image output by the second neural network model; and perform instance segmentation on the semantic mask image to obtain coordinate information of instances in the semantic mask image, where the coordinate information of the instances is the coordinate information of the second obstacle. The point cloud extraction module 304 is specifically configured to extract a corresponding second point cloud from the synchronized frame based on the coordinate information of the second obstacle. The point cloud extraction module 304 is further configured to obtain the point cloud corresponding to the coordinate information from the point cloud image of the synchronized frame based on the coordinate information of the second obstacle, thereby obtaining a second point cloud. The obstacle recognition module 305 is further configured to perform obstacle recognition based on the second point cloud to obtain the second obstacle in the synchronized frame, where the target obstacle includes the second obstacle.

[0124] In some embodiments, the target obstacle includes a first obstacle and a second obstacle, and the first obstacle and the second obstacle are obstacles identified based on different target images. The obstacle identification device 30 further includes a fusion identification module for fusing information about the first obstacle and information about the second obstacle to obtain fused obstacle information; based on the fused obstacle information, removing the target obstacle point cloud corresponding to the fused obstacle information from the point cloud to obtain a residual point cloud after removing the target obstacle point cloud; performing three-dimensional meshing processing on the residual point cloud to obtain a three-dimensional mesh; performing point cloud clustering processing on the three-dimensional mesh to obtain an obstacle point cloud; and performing obstacle identification on the obstacle point cloud to obtain a third obstacle.

[0125] Based on the above device, it is possible to determine obstacles in point cloud data without using a point cloud detection model. By avoiding the use of a point cloud detection model, less computing resources can be consumed to achieve the purpose of identifying obstacles in point cloud data.

[0126] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0127] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0128] Specifically, the processor 401 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0129] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0130] In some embodiments, the memory 402 may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0131] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any one of the obstacle identification methods in the above embodiments.

[0132] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401 , the memory 402 , and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0133] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0134] Bus 410 comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 410 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0135] The electronic device can execute the method for identifying obstacles in the embodiment of the present application, thereby realizing the combination Figure 1 and Figure 3 A method and apparatus for identifying obstacles are described.

[0136] In addition, in conjunction with the obstacle identification method in the above embodiments, embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the obstacle identification methods in the above embodiments is implemented.

[0137] In combination with the method for identifying obstacles in the above embodiment, an embodiment of the present application further provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device implements the method for identifying obstacles in the above embodiment.

[0138] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0139] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0140] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0141] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0142] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A method for identifying obstacles, characterized in that: The method comprises: Acquiring a synchronized frame based on data from a plurality of sensors, the plurality of sensors including at least one first target camera and a target radar, the synchronized frame including an image from the first target camera and a point cloud from the target radar, wherein the point cloud from the target radar and the image from the first target camera are synchronized in time and correspond to the same scene; Extracting a target image from the synchronization frame; performing obstacle recognition based on the target image to obtain target obstacle information; Extracting a corresponding target point cloud in the synchronous frame based on the target obstacle information, where the target point cloud is at least a portion of the point cloud in the synchronous frame; Obstacle identification is performed based on the target point cloud to obtain the target obstacle in the synchronous frame.

2. The method according to claim 1, characterized in that The acquiring of a synchronization frame based on data from a plurality of sensors includes: Aligning the acquisition time of the data of the at least one first target camera with the acquisition time of the point cloud as a reference to obtain a synchronized frame; Projecting the point cloud onto an image plane captured by the second target camera according to the external parameters of the target radar and the second target camera and the internal parameters of the second target camera to obtain a point cloud image; The point cloud image is placed into a synchronization frame to obtain the synchronization frame of the fused point cloud.

3. The method according to claim 1, characterized in that The target image includes a first image of a pinhole camera, The performing obstacle recognition based on the target image to obtain target obstacle information includes: performing obstacle recognition based on the first image to obtain a two-dimensional frame of a first obstacle, wherein the target obstacle information includes the two-dimensional frame of the first obstacle; The extracting the corresponding target point cloud in the synchronization frame based on the target obstacle information includes: Based on the two-dimensional frame of the first obstacle, extracting a corresponding first point cloud in the synchronous frame; The performing obstacle identification based on the target point cloud to obtain the target obstacle in the synchronous frame includes: Obstacle identification is performed based on the first point cloud to obtain a first obstacle in the synchronization frame, where the target obstacle includes the first obstacle.

4. The method according to claim 3, characterized in that The performing obstacle recognition based on the first image to obtain a two-dimensional frame of the first obstacle includes: The first image is input into a preset first neural network model to obtain a two-dimensional frame of the first obstacle predicted and output by the first neural network model.

5. The method according to claim 3, characterized in that The two-dimensional frame based on the first obstacle, extracting a corresponding first point cloud in the synchronous frame, includes: Based on the two-dimensional frame of the first obstacle, determining an obstacle area corresponding to the two-dimensional frame of the first obstacle from the point cloud image in the synchronized frame; Acquire a point cloud in the obstacle area to obtain the first point cloud.

6. The method according to claim 1, wherein The target image includes a second image of a fisheye camera, The performing obstacle recognition based on the target image to obtain target obstacle information includes: performing obstacle recognition based on the second image to obtain coordinate information of the second obstacle, wherein the target obstacle information includes the coordinate information of the second obstacle; The extracting the corresponding target point cloud in the synchronization frame based on the target obstacle information includes: Extracting a corresponding second point cloud in the synchronous frame based on the coordinate information of the second obstacle; The performing obstacle identification based on the target point cloud to obtain the target obstacle in the synchronous frame includes: Obstacle identification is performed based on the second point cloud to obtain a second obstacle in the synchronization frame, where the target obstacle includes the second obstacle.

7. The method according to claim 6, characterized in that The performing obstacle recognition based on the second image to obtain coordinate information of the second obstacle includes: Inputting the second image into a preset second neural network model to obtain a semantic mask image output by the second neural network model; Instance segmentation is performed on the semantic mask image to obtain coordinate information of instances in the semantic mask image, where the coordinate information of the instances is the coordinate information of the second obstacle.

8. The method according to claim 6, characterized in that The extracting, in the synchronization frame, a corresponding second point cloud based on the coordinate information of the second obstacle includes: Based on the coordinate information of the second obstacle, a point cloud under the coordinate information is acquired from the point cloud image of the synchronous frame to obtain the second point cloud.

9. The method according to claim 1, characterized in that The target obstacle includes a first obstacle and a second obstacle, wherein the first obstacle and the second obstacle are obstacles identified based on different target images. After performing obstacle recognition based on the target point cloud to obtain the target obstacle in the synchronization frame, the method further includes: fusing the first obstacle information and the second obstacle information to obtain fused obstacle information; Based on the fused obstacle information, removing the target obstacle point cloud corresponding to the fused obstacle information from the point cloud to obtain a remaining point cloud after removing the target obstacle point cloud; Performing three-dimensional meshing on the remaining point cloud to obtain a three-dimensional mesh; Performing point cloud clustering processing on the three-dimensional grid to obtain an obstacle point cloud; Obstacle recognition is performed on the obstacle point cloud to obtain a third obstacle.

10. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for identifying an obstacle according to any one of claims 1 to 9 is implemented.

Images