Space passable area detection method and device, detection equipment and storage medium

Abstract

The application relates to the technical field of target detection, in particular to a space passable area detection method and device, a detection equipment and a storage medium, wherein the method comprises the following steps: obtaining three-dimensional point cloud data and an environment image of a target area; performing plane projection and view cone projection on the three-dimensional point cloud data respectively to obtain multiple surface features and view cone projection features; fusing the multiple surface features and the view cone projection features to obtain a space feature map; fusing the space feature map and the environment image under the same time and space to obtain a fused feature map; inputting the fused feature map into a trained detection model; and outputting a space passable area of the target area. Thus, the related art is generally used for detecting and identifying grounded obstacles, pedestrians, vehicles and the like, and the resolution and generalization are not strong, the spatial position of the obstacle cannot be accurately determined, and the problems of avoiding collision and causing accidents and the like are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular relates to a space passable area detection method and device, a detection equipment and a storage medium. BACKGROUND

[0002] With the development of transportation and technology, the intensity and density of highway traffic are rapidly increasing, leading to frequent occurrence of various traffic accidents. However, if an effective warning can be given to the driver 1 second in advance, more than 90% of traffic accidents can be avoided. Therefore, autonomous driver assistance systems (ADAS) have emerged. One of the most important research directions in autonomous driver assistance systems is vehicle intelligent obstacle detection. Vehicle obstacle detection refers to the ability of a vehicle to perceive the surrounding environment with the aid of sensors, to obtain real-time surrounding environment information, to determine whether there is an obstacle, and to avoid collisions and accidents. The obstacles in front of the vehicle are divided into static obstacles (such as roadblocks, cone barrels, etc.) and dynamic obstacles (such as vehicles, pedestrians, etc.). Obstacle detection is a research hotspot in the field of intelligent vehicles, and how to efficiently and accurately identify various static and dynamic obstacles is the key technology.

[0003] The fusion detection means in the related art is mostly concentrated in the use of traditional 3D millimeter wave radar, but only the distance, speed, and azimuth angle information of the target can be obtained, and the road traffic information cannot be reflected in the real three-dimensional space, and then the 4D millimeter wave radar can additionally obtain the height information of the target, that is, the spatial coordinates and speed of the target around the vehicle can be captured. The detection device using 4D millimeter wave radar and camera fusion in the related art is usually for the detection and identification of grounded obstacles, pedestrians, vehicles, etc., and gives the ground passable area of the vehicle. Due to the identification algorithm, the occupied area and target size of the ground obstacle cannot be given. SUMMARY

[0004] The present application provides a space passable area detection method, device, detection equipment and storage medium to solve the problems that the related art is usually for the detection and identification of grounded obstacles, pedestrians, vehicles, etc., the resolution and generalization are not strong, and the accurate spatial position of the obstacle cannot be accurately determined, so as to avoid collisions and accidents.

[0005] The first aspect of the present application provides a space passable area detection method, comprising the following steps: obtaining three-dimensional point cloud data and an environment image of a target area; performing plane projection and view cone projection on the three-dimensional point cloud data respectively to obtain a plurality of surface features and view cone projection features, and fusing the plurality of surface features and the view cone projection features to obtain a space feature map; fusing the space feature map and the environment image under the same time and space to obtain a fused feature map; and inputting the fused feature map into a trained detection model to output a space passable area of the target area.

[0006] According to the above technical means, the three-dimensional point cloud data can be respectively projected on a plane and a view cone to form an abstract feature map, and network mapping fusion can be performed to form a relatively complete space feature map matched with the image, thereby providing comprehensive feature information for a space fusion recognition network, and a space point cloud feature matched with the image representation can be fused under the same time and space, so that the space passable area is obtained through the network, which can not only give the space position information of the identifiable target object, but also give the space position information of other position objects, thereby providing a safe passable space detection result for a vehicle.

[0007] Optionally, in an embodiment of the present application, the detection model comprises a plurality of deep network modules, wherein the plurality of deep network modules perform network mapping on the fused feature map, divide the target area in the fused feature map into a two-class space of a space passable area and a space impassable area based on a mapping result, and identify the two-class space to obtain the space passable area of the target area.

[0008] According to the above technical means, the two-class space division of the passable area and the impassable area in the space can be parsed by using the fully connected neural network of the plurality of deep network modules, and the detection of the space passable area can be realized.

[0009] Optionally, in an embodiment of the present application, the fusing the space feature map and the environment image under the same time and space to obtain the fused feature map comprises: segmenting a foreground and a background of the environment image under the same time and space to obtain a segmented feature map; and mapping the space feature map onto the segmented feature map to obtain the fused feature map.

[0010] According to the above technical means, the segmented feature image and the space point cloud feature can be fused under the same time and space, so that the feature map has not only space position information, but also provides more comprehensive information for the recognition network.

[0011] Optionally, in an embodiment of the present application, the fusing the plurality of surface features and the frustum projection features to obtain a spatial feature map comprises: performing network mapping fusion on the plurality of surface features and the frustum projection features by using a mapping fully connected network to obtain the spatial feature map.

[0012] According to the above technical means, the plurality of surface features and the frustum features are network mapping fused to form a more complete spatial feature map matching the image, and comprehensive feature information is provided for the spatial fusion recognition network.

[0013] Optionally, in an embodiment of the present application, the projecting the three-dimensional point cloud data respectively by plane projection and frustum projection to obtain a plurality of surface features and frustum projection features comprises: projecting the three-dimensional point cloud data to different planes to obtain a plurality of surface features; dividing a grid frustum in an image plane of the environment image, dividing each grid frustum into different spatial bodies according to a distance between each grid frustum and a projection origin, clustering the three-dimensional point cloud data in each spatial body to obtain the frustum projection features.

[0014] According to the above technical means, the three-dimensional point cloud data is projected to different planes to obtain a plurality of surface features, and the spatial point cloud is modularly clustered in the form of a frustum, and the image region is projected to form an abstract feature map.

[0015] Optionally, in an embodiment of the present application, before the projecting the three-dimensional point cloud data respectively by plane projection and frustum projection, the method further comprises: jointly calibrating the three-dimensional point cloud data and the environment image, so that the three-dimensional point cloud data and the environment image are in the same coordinate system.

[0016] According to the above technical means, the three-dimensional point cloud data and the environment image are jointly calibrated, the spatial perception information is mapped to a unified coordinate system, and the method is not dependent on the installation position of the acquisition device, and has higher practicability.

[0017] The second aspect embodiment of the present application provides a spatial passable area detection device, comprising: an acquisition module configured to acquire three-dimensional point cloud data and an environment image of a target area; a projection module configured to project the three-dimensional point cloud data respectively by plane projection and frustum projection to obtain a plurality of surface features and frustum projection features, and fuse the plurality of surface features and the frustum projection features to obtain a spatial feature map; and a fusion module configured to fuse the spatial feature map and the environment image under the same time and space to obtain a fused feature map, input the fused feature map into a trained detection model, and output a spatial passable area of the target area.

[0018] Optionally, in an embodiment of the present application, the detection model comprises a plurality of deep network modules, wherein the plurality of deep network modules perform network mapping on the fusion feature map, divide the target region in the fusion feature map into a binary classification space of a spatially passable region and a spatially impassable region based on a mapping result, and identify the binary classification space to obtain the spatially passable region of the target region.

[0019] Optionally, in an embodiment of the present application, the fusion module is further configured to segment a foreground and a background of the environment image under the same time and space to obtain a segmentation feature map; and map the spatial feature map onto the segmentation feature map to obtain the fusion feature map.

[0020] Optionally, in an embodiment of the present application, the projection module is further configured to perform network mapping fusion on the plurality of face features and the cone projection features by using a mapping fully connected network to obtain the spatial feature map.

[0021] Optionally, in an embodiment of the present application, the projection module is further configured to project the three-dimensional point cloud data onto different planes to obtain a plurality of face features; divide a grid of an image plane of the environment image to obtain a grid cone, divide each grid cone into different space bodies according to a distance between the each grid cone and a projection origin, and cluster the three-dimensional point cloud data in each space body to obtain the cone projection features.

[0022] Optionally, in an embodiment of the present application, further comprising a calibration module configured to jointly calibrate the three-dimensional point cloud data and the environment image before performing the plane projection and the cone projection on the three-dimensional point cloud data respectively, so that the three-dimensional point cloud data and the environment image are in the same coordinate system.

[0023] An embodiment of the third aspect of the present application provides a detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the spatially passable region detection method as described in the above embodiments.

[0024] An embodiment of the fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the spatially passable region detection method as described in the above embodiments.

[0025] Therefore, the present application has at least the following beneficial effects:

[0026] 1、The embodiment of the present application can perform plane projection and view cone projection on three-dimensional point cloud data respectively to form abstract feature maps, and perform network mapping fusion to form a relatively complete spatial feature map matching the image, thereby providing comprehensive feature information for the spatial fusion recognition network, and forming a spatial point cloud feature matching the image representation under the same time and space. The spatial passable area is obtained through the network based on the feature, which can not only give the spatial position information of the identifiable target object, but also give the spatial position information of other position objects, thereby providing a safe passable space detection result for the vehicle.

[0027] 2、The embodiment of the present application can use the full connection neural network of multiple deep network modules to analyze the binary classification space division of the passable area and the impassable area in the space, thereby realizing the detection of the spatial passable area.

[0028] 3、The embodiment of the present application can fuse the segmented feature image and the spatial point cloud feature under the same time and space, so that the feature map has spatial position information and provides more comprehensive information for the recognition network.

[0029] 4、The embodiment of the present application can perform network mapping fusion on multiple plane features and view cone features to form a relatively complete spatial feature map matching the image, thereby providing comprehensive feature information for the spatial fusion recognition network.

[0030] 5、The embodiment of the present application can project three-dimensional point cloud data to different planes to obtain multiple plane features, and use the view cone form to modularize clustering of the spatial point cloud, and perform image area projection to form abstract feature maps.

[0031] 6、The embodiment of the present application can jointly calibrate three-dimensional point cloud data and environmental images, map the spatial perception information to a unified coordinate system, and does not depend on the installation position of the collection device, thereby having higher practicability.

[0032] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A flowchart of a spatial passable area detection method according to an embodiment of the present application is shown in FIG. 1;

[0035] Figure 2 A point cloud data imaging schematic diagram according to an embodiment of the present application is shown in FIG. 2;

[0036] Figure 3 A multi-plane point cloud feature mapping extraction schematic diagram provided according to an embodiment of the application;

[0037] Figure 4 A view frustum point cloud feature mapping extraction schematic diagram provided according to an embodiment of the application;

[0038] Figure 5 An image semantic segmentation schematic diagram provided according to an embodiment of the application;

[0039] Figure 6 A three-dimensional point cloud data and environment image joint calibration schematic diagram provided according to an embodiment of the application;

[0040] Figure 7 A flowchart of a space passable area detection method provided according to an embodiment of the application;

[0041] Figure 8 A block diagram of a space passable area detection device provided according to an embodiment of the application;

[0042] Figure 9 A structural schematic diagram of a test device provided according to an embodiment of the application.

[0043] Legend: acquisition module-100, projection module-200, fusion module-300, memory-901, processor-902, communication interface-903. DETAILED DESCRIPTION

[0044] Embodiments of the application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the application, and cannot be understood as a limitation of the application.

[0045] A spatial passable region detection method, device, detection equipment and storage medium are described below with reference to the accompanying drawings. In view of the problems mentioned in the background art, the present application provides a spatial passable region detection method. In the method, three-dimensional point cloud data and environment images of a target region are obtained, the three-dimensional point cloud data is projected on a plane and a view cone respectively, a plurality of surface features and view cone projection features are obtained, an abstract feature map is formed, network mapping fusion is performed, a complete spatial feature map matching the image is formed, comprehensive feature information is provided for a spatial fusion recognition network, and a spatial point cloud feature matching the image representation is formed under the same time and space. The spatial passable region is obtained by the feature through the network, which can not only give the spatial position information of the identifiable target object, but also give the spatial position information of other position objects, and provide a safe passable space detection result for the vehicle. Thus, the problems that the related art is generally used for detecting and identifying ground obstacles, pedestrians, vehicles and the like, the resolution and generalization are not strong, and the spatial position of the obstacle cannot be accurately determined, so as to avoid accidents caused by collision are solved.

[0046] Specifically, Figure 1 A flowchart of a spatial passable region detection method provided by the present application is shown.

[0047] As Figure 1 shown, the spatial passable region detection method includes the following steps:

[0048] In step S101, three-dimensional point cloud data and environment images of a target region are obtained.

[0049] It can be understood that vehicle obstacle detection refers to that a vehicle can perceive the surrounding environment by means of a sensor, obtain surrounding environment information in real time, and determine whether there is an obstacle, so as to avoid accidents caused by collision. The obstacle detection method mainly includes machine vision detection and radar detection. The machine vision detection method mainly includes motion feature detection, shape feature detection and feature and template matching method. The millimeter wave radar detection method obtains the distance, speed and angle information of the target by emitting electromagnetic waves and processing the received echo signals. The present application can collect external information by means of a 4D millimeter wave radar and a camera, so as to obtain three-dimensional point cloud data and environment images of a target region, greatly improve the reliability and stability of the system, expand the application range, and at the same time, not bring very high cost.

[0050] In step S102, the three-dimensional point cloud data is projected on a plane and a view cone respectively, a plurality of surface features and view cone projection features are obtained, and the spatial feature map is obtained by fusing the plurality of surface features and view cone projection features.

[0051] As Figure 2As shown, the three-dimensional point cloud data of the embodiment of the present application contains distance, orientation, speed, and height information of the reflection target object, but it is very difficult to identify the target object category and reflect the size of the target object contour because the imaging is very sparse and contains a large amount of noise. Therefore, the embodiment of the present application can extract point cloud features by a multi-plane feature extraction method and a view cone feature extraction method to form an abstract feature map, and fuse the image features to finally map the output space drivable area.

[0052] In an embodiment of the present application, the three-dimensional point cloud data is respectively projected on a plane and a view cone to obtain a plurality of plane features and view cone projection features, including: projecting the three-dimensional point cloud data on different planes to obtain a plurality of plane features; dividing a grid of an image plane of an environment image to obtain a grid view cone, dividing each grid view cone into different space bodies according to the distance between each grid view cone and a projection origin, and clustering the three-dimensional point cloud data in each space body to obtain the view cone projection features.

[0053] First, for simplicity, the embodiment of the present application can project the space point cloud on three planes (x-y, x-z, z-y) in a three-dimensional coordinate system, but is not limited to these three planes, and can be projected on any plane according to actual needs. The projection mode in the embodiment of the present application is to obtain the obstacle distribution of the ground plane, the obstacle distribution in the horizontal direction, and the obstacle distribution in the vertical direction. In the actual execution process, for example, Figure 3 As shown, the embodiment of the present application can project the space point cloud on the x-y plane, the x-z plane, and the y-z plane to obtain the space occupation information in three dimensions. Thus, three H*W*(dis, ori, vel, alt) multi-dimensional sparse feature maps can be obtained. Wherein, H and W are the width and height of the feature map, dis is the distance of the echo point, ori is the orientation of the echo point, vel is the speed of the echo point, and alt is the height information of the echo point.

[0054] Secondly, in order to match the image, the embodiment of the present application can modularize and cluster the space point cloud in the form of a view cone, and project on the image area. Specifically, the projection origin and the image plane form a view cone, and the space point cloud is contained in the entire view cone. The corresponding grid view cone is obtained by dividing the image plane into corresponding grids. Each grid view cone is divided into different space bodies by distance, and the point cloud information is clustered and information is accumulated in each space body. The space body information is projected on the image plane, and each grid plane will obtain the point cloud features containing longitudinal and lateral information. In the actual execution process, for example, Figure 4As shown, the embodiment of the present application can project the spatial point cloud to the imaging plane by using the principle of the frustum. First, in the frustum block, the spatial information points are clustered by using the feature clustering algorithm, so as to filter out part of the noise points. Then, the point cloud in the frustum block is mapped to the image frustum plane by using the 3D convolution network. Finally, the feature map on the image plane: U*V*F is formed by splicing. U and V are the width and height of the image, and F is the feature dimension.

[0055] In an embodiment of the present application, the spatial feature map is obtained by fusing multiple face features and frustum projection features, comprising: network mapping fusion of the multiple face features and the frustum projection features by using a mapping fully connected network to obtain the spatial feature map.

[0056] In the embodiment of the present application, the face features extract the point cloud distribution features of three faces in space respectively; the spatial features also extract the features of three directions based on the image face, which can better match the image features. Since the face features are physical projection, the embodiment of the present application can be considered as lossless features, while the frustum projection features lose part of the point cloud feature information in order to match the image. Therefore, in order to extract lossless features, the embodiment of the present application designs a mapping fully connected network for feature fusion, which network maps and fuses the multiple face features and the frustum features to form a more complete spatial feature map that matches the image, so as to provide comprehensive feature information for the spatial fusion recognition network.

[0057] In step S103, the spatial feature map and the environment image are fused under the same time and space to obtain a fused feature map. The fused feature map is input into the trained detection model, and the spatial passable area of the target region is output.

[0058] It can be understood that the embodiment of the present application can fuse the environment image and the spatial feature map, and finally analyze the spatial passable area by using the trained detection model after the spatio-temporal consistency verification. Not only can the spatial position information of the identifiable target object be given, but also the spatial position information of other objects can be given, so as to provide a safe passable space detection result for the vehicle and change the driving collision caused by only having the ground passable area information and the spatial obstacles in the past.

[0059] In an embodiment of the present application, the spatial feature map and the environment image are fused under the same time and space to obtain a fused feature map, comprising: segmenting the foreground and background of the environment image under the same time and space to obtain a segmented feature map; and mapping the spatial feature map to the segmented feature map to obtain the fused feature map.

[0060] Specifically, as shown in FIG. 6, the embodiment of the present application can fuse the spatial feature map and the environment image under the same time and space to obtain a fused feature map. Figure 5As shown, the embodiments of the present application can utilize a general algorithm to perform semantic segmentation on the collected environment image to form a region segmentation image. A general motion compensation method is adopted to match and adjust the spatial positions of the three-dimensional point cloud data through the whole vehicle motion information to ensure the consistency with the imaging time of the image and ensure the consistency of the extracted external information features. After the spatio-temporal consistency is verified, the image can be segmented by using any segmentation algorithm to separate the foreground and background to facilitate the point cloud feature mapping, and finally the segmented feature image and the spatial point cloud feature are fused to make the feature map not only have spatial position information but also have color information of the image, thereby providing a more comprehensive information for the recognition network.

[0061] In an embodiment of the present application, the detection model includes a plurality of deep network modules, wherein the plurality of deep network modules perform network mapping on the fusion feature map, divide the target region in the fusion feature map into a binary classification space of a spatial passable region and a spatial impassable region based on the mapping result, and identify the binary classification space to obtain the spatial passable region of the target region.

[0062] In the embodiments of the present application, the fully connected neural network of the plurality of deep network modules implements the detection network of the spatial passable region, but is not limited to the use of this network, and any combination network can complete the detection of the spatial passable region of the embodiments of the present application through the training of the true value data.

[0063] Specifically, after obtaining the fusion feature map, the embodiments of the present application can utilize the fully connected neural network of the plurality of deep network modules to implement the detection network of the spatial passable region. In order to learn the parameters of the neural network, random sampling points are considered in the three-dimensional boundary body of the object, K points are sampled for the i th sample, and then the small batch loss L b is evaluated for these positions, and the expression is as follows:

[0064]

[0065] where x i is the i th observation of B batches, o ij is the true position of the point cloud, and L is the cross-entropy loss. Finally, the network will output the binary classification space division of the passable region and the impassable region in the output space according to the network mapping.

[0066] In an embodiment of the present application, before the three-dimensional point cloud data is respectively projected on a plane and projected in a view cone, it further includes jointly calibrating the three-dimensional point cloud data and the environment image so that the three-dimensional point cloud data and the environment image are in the same coordinate system.

[0067] The embodiments of the present application can first adopt a general calibration algorithm to jointly calibrate the three-dimensional point cloud data and the environment image, and map the spatial perception information to a unified coordinate system. For example, the embodiments of the present application can adopt a general calibration algorithm to jointly calibrate the three-dimensional point cloud data and the environment image, and map the spatial perception information to a unified coordinate system.Figure 6 As shown, after joint calibration, the detected object will be unified in the coordinate, without depending on the installation position of the external information collection device.

[0068] Specifically, the embodiment of the application can be based on the external information collected by the 4D millimeter wave radar and the camera, the normal vector of the radar detection surface and the optical axis of the camera are respectively parallel to the longitudinal symmetry plane of the vehicle body after the 4D millimeter wave radar and the camera are calibrated separately, and on this basis, the radar projection coordinate system O rw -X rw Y rw Z rw , the origin is the projection point of the radar detection center point on the ground, the X rw axis points to the right side of the vehicle body, the Y rw axis points to the forward direction of the vehicle, and the Z rw axis is the vertical direction; the origin of the camera projection coordinate system O cw -X cw Y cw Z cw is the projection point of the camera optical center on the ground, the X cw axis points to the right side of the vehicle body, the Y cw axis points to the forward direction of the vehicle, and the Z cw axis is the vertical direction. The radar projection coordinate system and the camera projection coordinate system are two coordinate systems in space, and their spatial relative relationship is as shown in Figure 6

[0069] According to the pinhole imaging model, and according to the translation and rotation relationship between the camera coordinate system and the camera projection coordinate system, and the mutual conversion relationship between the radar projection coordinate system and the camera projection coordinate system, the conversion relationship of any point in the radar projection coordinate system to the image coordinate system can be obtained:

[0070] X p =C x +(X rw -L x )f x Hsinθ+(Y rw +L y )cosθ

[0071] Y p =C y +[Hcosθ-(Y rw +L y )sinθ]f y Hsinθ+(Y rw +L y )cosθ

[0072] Wherein, C x , C y ​is the optical axis offset; f x , f y is the focal length, and is the camera intrinsic parameter, which is obtained by calibration; L x , L y are the radar projection coordinate system and the camera projection coordinate system, respectively, and the distance between the x-axes and the distance between the y-axes can be obtained by measurement.

[0073] It should be noted that the space passable area detection method of the embodiments of the present application can be based on the collected external information to realize the judgment of the space passable area of the vehicle, which does not depend on the recognition of the target, and can judge the position of the space obstacle. Therefore, the embodiments of the present application can be applied to various devices, such as the passable area judgment of vehicles, aircrafts and the like.

[0074] The space passable area detection method of the embodiments of the present application will be described in detail in combination with a specific implementation mode. As shown in Figure 7 , the embodiments of the present application can detect the 3D passable area by camera target detection and fusion of sparse and noisy 4D millimeter wave radar point cloud. Since the sampling periods of the 4D millimeter wave radar and the image are different: the millimeter wave radar is 50ms, and the camera is within 40ms. Therefore, the embodiments of the present application can use the vehicle IMU (Inertial Measurement Unit) information to perform motion compensation on the space point cloud, correct the spatial position information of the space point cloud to the time of image exposure, and use a general calibration algorithm to calibrate the camera and the 4D millimeter wave radar to a unified coordinate system. Further, the multi-plane feature extraction method and the view cone feature extraction method are combined to extract the 4D millimeter wave radar point cloud features, to form an abstract feature map, and the image features are fused to obtain the fusion feature map of the 4D millimeter wave radar and the image features. The all-obstacle detection in the space is realized by using the full connection neural network of multiple deep network modules, and the space passable area is outputted. The embodiments of the present application do not depend on the recognition of the target, and all the targets that hinder the passable area can be detected and positioned.

[0075] According to the space passable area detection method provided in the embodiments of the present application, by obtaining the three-dimensional point cloud data and the environment image of a target area, the three-dimensional point cloud data is respectively subjected to plane projection and view cone projection, a plurality of face features and view cone projection features are obtained to form an abstract feature map, and network mapping fusion is performed to form a relatively complete space feature map matched with the image, comprehensive feature information is provided for a space fusion recognition network, and a space point cloud feature matched with the image representation is fused under the same time and space, and the space passable area is obtained by the network based on the feature, which can not only provide the space position information of the identifiable target object, but also provide the space position information of other position objects, and provide a safe passable space detection result for a vehicle. Thus, the problems that the related art is generally used for detecting and recognizing the ground obstacles, pedestrians, vehicles and the like, the resolution and generalization are not strong, and the space position of the obstacle cannot be accurately determined, so that the collision and accidents are avoided.

[0076] Secondly, a space passable area detection device according to the embodiments of the present application is described with reference to the accompanying drawings.

[0077] Figure 8 is a block schematic diagram of a space passable area detection device according to the embodiments of the present application.

[0078] As shown in Figure 8 , the space passable area detection device 10 comprises an acquisition module 100, a projection module 200 and a fusion module 300.

[0079] The acquisition module 100 is configured to acquire three-dimensional point cloud data and an environment image of a target area; the projection module 200 is configured to respectively perform plane projection and view cone projection on the three-dimensional point cloud data, obtain a plurality of face features and view cone projection features, and fuse the plurality of face features and view cone projection features to obtain a space feature map; and the fusion module 300 is configured to fuse the space feature map and the environment image under the same time and space to obtain a fused feature map, input the fused feature map into a detection model trained, and output a space passable area of the target area.

[0080] In an embodiment of the present application, the detection model comprises a plurality of deep network modules, the plurality of deep network modules perform network mapping on the fused feature map, the target area in the fused feature map is divided into a two-class space of a space passable area and a space impassable area based on a mapping result, and the space passable area of the target area is identified by recognizing the two-class space.

[0081] In an embodiment of the present application, the fusion module 300 is further configured to segment a foreground and a background of the environment image under the same time and space to obtain a segmented feature map; and map the space feature map onto the segmented feature map to obtain the fused feature map.

[0082] In an embodiment of the present application, the projection module 200 is further configured to perform network mapping fusion on the plurality of surface features and the cone projection features by using a mapping fully connected network to obtain a spatial feature map.

[0083] In an embodiment of the present application, the projection module 200 is further configured to project the three-dimensional point cloud data to different planes to obtain a plurality of surface features, divide a grid of an image surface of the environment image to obtain a grid cone, divide each grid cone into different spatial bodies according to a distance between each grid cone and a projection origin, and cluster the three-dimensional point cloud data in each spatial body to obtain cone projection features.

[0084] In an embodiment of the present application, the device 10 of the embodiment of the present application further includes a calibration module.

[0085] The calibration module is configured to jointly calibrate the three-dimensional point cloud data and the environment image before the three-dimensional point cloud data is respectively projected on a plane and a cone, so that the three-dimensional point cloud data and the environment image are in the same coordinate system.

[0086] It should be noted that the foregoing explanation and description of the space passable region detection method embodiments are also applicable to the space passable region detection device of this embodiment, which will not be described here again.

[0087] According to the space passable region detection device provided in the embodiment of the present application, the three-dimensional point cloud data and the environment image of a target region are obtained, the three-dimensional point cloud data is respectively projected on a plane and a cone to obtain a plurality of surface features and cone projection features, an abstract feature map is formed, network mapping fusion is performed, a complete spatial feature map matching the image is formed, comprehensive feature information is provided for a spatial fusion recognition network, and a space point cloud feature matching the image representation is fused under the same time and space. The space passable region is obtained by using the feature through the network, which can not only give the spatial position information of the identifiable target object, but also give the spatial position information of other position objects, and provide a safe passable space detection result for a vehicle. Thus, the problem that the related art is usually used for detecting and identifying ground obstacles, pedestrians, vehicles and the like, and has poor resolution and generalization, and cannot accurately determine the spatial position of the obstacle, thereby avoiding accidents caused by collision and the like is solved.

[0088] Figure 9 The structure diagram of the detection device provided in the embodiment of the present application is shown. The detection device can include:

[0089] The memory 901, the processor 902, and the computer program stored in the memory 901 and executable on the processor 902.

[0090] The processor 902 implements the space passable region detection method provided in the above embodiments when executing the program.

[0091] Further, the detection device further comprises:

[0092] A communication interface 903 is configured to communicate between the memory 901 and the processor 902.

[0093] The memory 901 is configured to store a computer program executable by the processor 902.

[0094] The memory 901 can include a high-speed RAM (Random Access Memory) memory, and can further include a nonvolatile memory, such as at least one disk memory.

[0095] If the memory 901, the processor 902 and the communication interface 903 are implemented independently, the communication interface 903, the memory 901 and the processor 902 can be connected to each other through a bus and complete communication between each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 9 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0096] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can complete communication between each other through an internal interface.

[0097] The processor 902 can be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0098] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the above space passable area detection method.

[0099] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the description of the application, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the description and the features of the different embodiments or examples, without contradiction.

[0100] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0101] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the various embodiments of the application contemplate that the modules, segments, or portions of code may, in some embodiments, be implemented by hardware, software, firmware, or any combination thereof. The various embodiments of the application contemplate that the order of the steps in the processes described herein can be altered, that other steps can be added, and that some steps can be eliminated, without departing from the scope of the application.

[0102] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if implemented in hardware, any of the following technologies known in the art or their combinations can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array, field programmable gate array, etc.

[0103] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium. The program, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0104] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A method for detecting passable areas in space, characterized in that, Includes the following steps: Acquire 3D point cloud data and environmental images of the target area; The three-dimensional point cloud data is subjected to planar projection and frusto projection respectively to obtain multiple surface features and frusto projection features. The multiple surface features and the frusto projection features are then fused to obtain a spatial feature map. The spatial feature map and the environmental image are fused under the same time and space to obtain a fused feature map. The fused feature map is input into the trained detection model to output the spatial passable area of ​​the target region. The process of fusing the multiple surface features and the view frustum projection features to obtain a spatial feature map includes: The spatial feature map is obtained by using a fully connected mapping network to perform network mapping and fusion on the multiple surface features and the view frustum projection features; The three-dimensional point cloud data is subjected to planar projection and frustum projection respectively to obtain multiple surface features and frustum projection features, including: The three-dimensional point cloud data is projected onto different planes to obtain multiple surface features; The image surface of the environment image is divided into grids to obtain grid frustums. Each grid frustum is divided into different spatial volumes according to the distance between each grid frustum and the projection origin. The three-dimensional point cloud data is clustered within each spatial volume to obtain the projection features of the frustum.

2. The method according to claim 1, characterized in that, The detection model includes multiple deep network modules, wherein the multiple deep network modules perform network mapping on the fused feature map, and based on the mapping result, divide the target region in the fused feature map into a binary space of spatially passable regions and spatially inaccessible regions, and identify the spatially passable regions of the target region by identifying the binary space.

3. The method according to claim 1, characterized in that, The process of fusing the spatial feature map and the environmental image in the same time and space to obtain a fused feature map includes: The foreground and background of the environmental image are segmented in the same time and space to obtain a segmentation feature map; The spatial feature map is mapped onto the segmentation feature map to obtain the fused feature map.

4. The method according to any one of claims 1-3, characterized in that, Before performing planar projection and frustum projection on the three-dimensional point cloud data, the method further includes: The 3D point cloud data and the environmental image are jointly calibrated so that the 3D point cloud data and the environmental image are in the same coordinate system.

5. A device for detecting passable areas in space, characterized in that, include: The acquisition module is used to acquire 3D point cloud data and environmental images of the target area; The projection module is used to perform planar projection and frustum projection on the three-dimensional point cloud data to obtain multiple surface features and frustum projection features, and to fuse the multiple surface features and the frustum projection features to obtain a spatial feature map. The fusion module is used to fuse the spatial feature map and the environmental image in the same time and space to obtain a fused feature map, input the fused feature map into the trained detection model, and output the spatially passable area of ​​the target region. The projection module is further used for: The spatial feature map is obtained by using a fully connected mapping network to perform network mapping and fusion on the multiple surface features and the view frustum projection features; The projection module is further used for: The three-dimensional point cloud data is projected onto different planes to obtain multiple surface features; The image surface of the environment image is divided into grids to obtain grid frustums. Each grid frustum is divided into different spatial volumes according to the distance between each grid frustum and the projection origin. The three-dimensional point cloud data is clustered within each spatial volume to obtain the projection features of the frustum.

6. The apparatus according to claim 5, characterized in that, The detection model includes multiple deep network modules, wherein the multiple deep network modules perform network mapping on the fused feature map, and based on the mapping result, divide the target region in the fused feature map into a binary space of spatially passable regions and spatially inaccessible regions, and identify the spatially passable regions of the target region by identifying the binary space.

7. The apparatus according to claim 5, characterized in that, The fusion module is further used for: The foreground and background of the environmental image are segmented in the same time and space to obtain a segmentation feature map; The spatial feature map is mapped onto the segmentation feature map to obtain the fused feature map.

8. The apparatus according to any one of claims 5-7, characterized in that, Also includes: The calibration module is used to jointly calibrate the three-dimensional point cloud data and the environmental image before performing planar projection and frustum projection on the three-dimensional point cloud data respectively, so that the three-dimensional point cloud data and the environmental image are in the same coordinate system.

9. A testing device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the spatially accessible area detection method as described in any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the spatially accessible area detection method as described in any one of claims 1-4.

Images