Multi-mode obstacle detection method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-mode obstacle detection method and device, electronic equipment and a storage medium, relates to the technical field of computers, in particular to the fields of computer vision, data processing and the like, and can be applied to application scenes such as obstacle detection and automatic driving. According to the specific scheme, the method comprises the following steps: extracting a point feature and a laser point representation feature corresponding to any feature point in a laser point cloud according to the obtained laser point cloud; projecting the laser point cloud of the current frame to a visual image, and determining image point representation features corresponding to feature points in combination with image features extracted from the visual image; obtaining a multi-modal fusion feature according to the point feature, the laser point representation feature and the image point representation feature; generating a reserved feature point set and a millimeter wave detection frame based on the multi-modal fusion feature and the millimeter wave point cloud, and projecting the reserved feature point set and the millimeter wave detection frame to the grid map; and generating an obstacle detection frame according to the reserved feature point set and the millimeter wave detection frame. According to the scheme, the detection precision of heterogeneous dynamic obstacles can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the fields of computer vision and data processing, and can be used in application scenarios such as obstacle detection and autonomous driving, and specifically to a multimodal obstacle detection method, device, electronic device and storage medium. Background Art

[0002] In the field of autonomous driving technology, the current data-driven three-dimensional (3D) target detection algorithms can only detect trained obstacles, but cannot accurately detect dynamic obstacles with abnormal shapes or features. Summary of the invention

[0003] The present disclosure provides a multimodal obstacle detection method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a multimodal obstacle detection method is provided, comprising: extracting point features and laser point representation features corresponding to any feature point in the laser point cloud according to an acquired laser point cloud; the laser point cloud includes a current frame laser point cloud; projecting the current frame laser point cloud onto a visual image, and determining image point representation features corresponding to the feature points in combination with image features extracted from the visual image; obtaining multimodal fusion features according to the point features, the laser point representation features and the image point representation features; generating a retained feature point set and a millimeter wave detection frame based on the multimodal fusion features and the millimeter wave point cloud, and projecting the retained feature point set and the millimeter wave detection frame onto a grid map; generating an obstacle detection frame according to the retained feature point set and the millimeter wave detection frame on the grid map.

[0005] According to a second aspect of the present disclosure, a multimodal obstacle detection device is provided, comprising: a laser feature extraction module; for extracting point features and laser point representation features corresponding to any feature point in the laser point cloud according to the acquired laser point cloud; the laser point cloud includes a current frame laser point cloud; an image feature extraction module, for projecting the current frame laser point cloud to a visual image, and determining the image point representation features corresponding to the feature points in combination with the image features extracted from the visual image; a multimodal feature extraction module, for obtaining multimodal fusion features according to the point features, the laser point representation features and the image point representation features; a grid projection module, for generating a reserved feature point set and a millimeter wave detection frame based on the multimodal fusion features and the millimeter wave point cloud, and projecting the reserved feature point set and the millimeter wave detection frame to a grid map; an obstacle detection module, for generating an obstacle detection frame according to the reserved feature point set and the millimeter wave detection frame on the grid map.

[0006] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0007] at least one processor; and

[0008] a memory communicatively connected to the at least one processor; wherein,

[0009] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0010] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0011] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0012] According to a sixth aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the electronic device as described in the third aspect.

[0013] By adopting the solution disclosed in the present invention, various types of dynamic obstacles can be detected, and the detection accuracy of heterogeneous dynamic obstacles can be improved.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0016] Figure 1 is a flow chart of a multimodal obstacle detection method according to an embodiment of the present disclosure;

[0017] Figure 2 is a schematic diagram of a process of obtaining multimodal fusion features based on laser point cloud and visual image according to an embodiment of the present disclosure;

[0018] Figure 3 is a schematic diagram of a process of obtaining an obstacle detection frame based on multimodal fusion features and millimeter wave point cloud according to an embodiment of the present disclosure;

[0019] Figure 4 is a schematic structural diagram of a multimodal obstacle detection device according to an embodiment of the present disclosure;

[0020] Figure 5 is a schematic diagram of a scenario of multimodal obstacle detection according to an embodiment of the present disclosure;

[0021] Figure 6 Schematic diagram of the structure of an electronic device used to implement the multimodal obstacle detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0024] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0025] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0026] Multi-sensor fusion: Data from multiple sensors are processed comprehensively to obtain more accurate and reliable information. By fusing data from different sensors, the limitations of a single sensor can be overcome and the robustness and accuracy of the system can be improved. For example, in autonomous driving, the fusion of data from lidar, cameras, and millimeter-wave radar can more accurately perceive the environment around the vehicle and improve driving safety.

[0027] Obstacle detection: Use sensors and algorithms to identify and locate objects in the environment to avoid collisions. Obstacle detection systems typically use multiple sensors, such as cameras, lidar, ultrasonic sensors, and millimeter-wave radars, to acquire environmental data. The data is then analyzed through image processing and machine learning algorithms to identify potential obstacles and take appropriate avoidance measures. This helps improve the safety and efficiency of the system.

[0028] In the related technology, with the popularization and development of deep learning and LiDAR, it has gradually become possible to use deep learning methods to perform point cloud target detection. However, for some dynamic obstacles with abnormal shapes or features on the road, such as tree-pulling carts, special-shaped non-motorized tricycles, etc., the current 3D target detection algorithms often have limitations and cannot detect such abnormal dynamic obstacles. The existing data-driven 3D target detection algorithms can only detect whitelist obstacles in the training set (such as cars, trucks, pedestrians, etc.), but cannot detect uncommon non-whitelist obstacles (such as special-shaped cars, fallen pedestrians, pedestrians pushing carts, etc.), and cannot accurately detect dynamic obstacles with abnormal shapes or features, and have the defect of low detection accuracy.

[0029] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a multimodal obstacle detection method, which can fully utilize the advantages of various sensors, effectively alleviate the collision risk of autonomous driving vehicles due to missed detection, and realize the detection of non-whitelisted dynamic obstacles by calculating the speed of each grid.

[0030] The present disclosure provides a multi-modal obstacle detection method. Figure 1 1 is a flow chart of a multimodal obstacle detection method according to an embodiment of the present disclosure. The multimodal obstacle detection method can be applied to a multimodal obstacle detection device. The multimodal obstacle detection device is located in an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, a fixed device includes but is not limited to a server, and the server can be a cloud server or an ordinary server. For example, a mobile device includes but is not limited to: a mobile phone, a tablet computer, and a vehicle-mounted terminal. In some possible implementations, the multimodal obstacle detection method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the multimodal obstacle detection method includes:

[0031] S101, extracting point features and laser point representation features corresponding to any feature point in the laser point cloud according to the acquired laser point cloud; the laser point cloud includes the laser point cloud of the current frame;

[0032] S102, projecting the laser point cloud of the current frame onto a visual image, and combining the image features extracted from the visual image to determine the image point representation features corresponding to the feature points;

[0033] S103, obtaining a multimodal fusion feature according to the point feature, the laser point representation feature and the image point representation feature;

[0034] S104, generating a retained feature point set and a millimeter wave detection frame based on the multimodal fusion features and the millimeter wave point cloud, and projecting the retained feature point set and the millimeter wave detection frame to a grid map;

[0035] S105 . Generate an obstacle detection frame according to the set of retained feature points on the grid image and the millimeter wave detection frame.

[0036] In the disclosed embodiments, a laser point cloud is a three-dimensional data set obtained by laser scanning technology. It consists of a large number of points, each of which has its precise coordinates in space. These points together depict the three-dimensional shape and structure of an object or environment. Laser point clouds are usually generated by laser radar devices, which calculate distances by emitting laser pulses and measuring reflection time, thereby acquiring spatial information.

[0037] In the disclosed embodiments, feature points in the laser point cloud refer to unique or significant points extracted from the laser point cloud data, and these feature points exhibit certain special geometric or intensity characteristics in space. For example, feature point detection algorithms such as the Intrinsic Shape Signature (ISS) based on point cloud curvature and normal vectors, the 3D key point detection method based on the geometric characteristics of the local surface (Harris3D), etc. can be used to determine feature points from the laser point cloud, and other methods in the prior art can also be used to determine feature points. The above is only an exemplary description and is not intended to limit all possible situations for determining feature points, but it is not exhaustive here.

[0038] In the disclosed embodiments, point features refer to features extracted from laser point cloud data for describing local or global properties of feature points, and are basic properties directly extracted from each feature point in the laser point cloud data. These features may include geometric features such as curvature and normal vectors and / or intensity features such as reflection intensity, as well as other information that can help identify and classify points. The above is only an exemplary description and is not intended to limit all possible situations of point features, but it is not exhaustive here.

[0039] In the disclosed embodiments, the laser point representation feature refers to a feature vector or descriptor used to represent and encode feature points in the laser point cloud data, which is generated by extracting features from the laser point cloud data and then performing point representation transformation on the features. In particular, the representation of each point in the laser point representation feature contains not only its own features, but also global and multi-scale information.

[0040] In the disclosed embodiment, the current frame laser point cloud refers to the laser point cloud data at a specific moment. In the dynamic scene of autonomous driving, the on-board laser radar device will continuously scan the environment and generate a series of laser point cloud frames. Each frame represents the environmental information collected at different times, and the "current frame" emphasizes the frame that is being processed or analyzed at the current moment in a series of continuously collected point cloud data.

[0041] In the disclosed embodiment, the visual image is a two-dimensional image captured by an optical device such as a camera, which represents the scene within the field of view of the device. In the dynamic scene of autonomous driving, the on-board camera device will continuously capture the environment within the field of view and generate a series of visual images. The visual image in the disclosed embodiment refers to the current frame visual image, that is, the frame being processed or analyzed at the current moment in a series of continuously captured visual images.

[0042] In the embodiments of the present disclosure, image features refer to specific attributes or information extracted from a visual image that can describe its content. They can be extracted using deep learning methods such as image encoders, or they can be extracted using methods such as manual feature extraction or feature descriptors in the prior art. The above is only an exemplary description and is not intended to limit all methods of extracting image features, but they are not listed exhaustively here.

[0043] In the disclosed embodiments, the image point representation feature refers to a feature vector or descriptor used to represent and encode feature points in a visual image, which can be used to better represent and distinguish different feature points in a high-dimensional space. The image point representation feature can be a set of image features of all feature points on the visual image after the laser point cloud is projected onto the visual image. This set contains the feature information corresponding to each feature point in the image, combining geometric and visual information for further analysis and processing.

[0044] In the disclosed embodiments, multimodal fusion features refer to features information from different sensors or data sources combined to form a more comprehensive and accurate representation, which can integrate data from different modalities, give full play to their respective advantages, and make up for the shortcomings of a single modality. Multimodal fusion features fuse point features, laser point representation features, and image point representation features, utilizing their respective advantages, combining the three-dimensional information of point features and laser point representation features with the two-dimensional information of image point representation features to form a more comprehensive feature representation, thereby enabling the system to understand the environment more comprehensively, combining the advantages of lidar and cameras, and improving accuracy and robustness. Exemplarily, multimodal fusion features may include: speed information and semantic categories.

[0045] In the disclosed embodiment, the millimeter wave point cloud is a three-dimensional point cloud data generated by the millimeter wave radar. The millimeter wave radar emits millimeter-level electromagnetic waves to detect objects and receive reflected signals to obtain the distance, speed and angle information of the object. Then, this information is processed into point cloud data, that is, a series of spatial coordinate points, which constitute a three-dimensional model of the target object. In particular, the electromagnetic waves emitted and received by the millimeter wave radar can be used to detect dynamic objects. By analyzing the frequency changes of the reflected signals, the millimeter wave radar can use the Doppler effect to measure the speed of the object. Moving objects will cause the frequency of the reflected signal to change. An increase in frequency means that the object is approaching, and a decrease in frequency means that the object is moving away. In the field of autonomous driving, the millimeter wave point cloud generated by the millimeter wave radar can accurately detect and track the position and speed of dynamic objects, and can more accurately detect all moving objects in the environment, ensuring the comprehensiveness of obstacle detection.

[0046] In the disclosed embodiment, the reserved feature point set refers to the set of feature points selected and retained during the feature point screening process, which can reduce the complexity of the data while retaining the contributing feature information. The reserved feature point set is obtained by screening all feature points based on multimodal fusion features, which can retain all required feature points, remove unnecessary feature points, reduce the amount of data, and improve processing efficiency.

[0047] In the disclosed embodiment, the millimeter wave detection frame refers to a three-dimensional bounding box created by using the point cloud data generated by the millimeter wave radar to represent and track objects. Since the millimeter wave point cloud is a collection of a series of discrete points of an object in space, the point cloud data is first analyzed by an algorithm to identify the point set belonging to the same object, and then a minimum three-dimensional bounding box is generated based on the identified point set to surround these points, thereby obtaining the millimeter wave detection frame.

[0048] In the disclosed embodiment, the process of projecting the retained feature point set and the millimeter wave detection frame onto the grid map is to project the three-dimensional feature points and the detection frame onto the two-dimensional grid map. Among them, the grid map is a plane composed of grids, and each grid unit represents a fixed spatial area. On the grid map, the information of the feature points and the detection frame is mapped to the corresponding grid cells to form a two-dimensional representation, which can simplify the three-dimensional information into a two-dimensional grid map, reducing the computational complexity, and in the field of autonomous driving, the grid map helps with path planning and navigation, and can provide clear obstacle information.

[0049] In the disclosed embodiment, the process of generating an obstacle detection frame involves the process of using information in a two-dimensional grid map to identify and describe obstacles. Combining the information of the feature point set and the millimeter wave detection frame, an algorithm is used to more accurately identify the position and shape of the obstacle, and then based on the fused information, a more accurate obstacle detection frame is generated on the grid map, and the obstacle detection frame can represent the boundary of the detected obstacle. In the field of autonomous driving, generating an obstacle detection frame can improve the accuracy of obstacle detection.

[0050] In the disclosed embodiment, laser point cloud data obtained by scanning by the vehicle-mounted laser radar device is obtained, and relevant point features are extracted from the point cloud based on the selected feature points, and then the laser point representation features are extracted by nonlinear transformation and other methods. Through the above steps, rich feature information can be extracted from the laser point cloud, including point features and laser point representation features.

[0051] Furthermore, after extracting image features from the visual image using a feature extraction algorithm or a neural network model, the known relative position and posture between the camera and the laser radar are used to perform coordinate transformation, thereby projecting the three-dimensional point cloud data collected by the laser radar onto the two-dimensional visual image. Then, for each feature point projected onto the image, the corresponding feature value is obtained from the image feature using a quadratic linear interpolation method. Through the above steps, each feature point in the laser point cloud is assigned a corresponding image feature, and the image features of all feature points are the image point representation features.

[0052] Furthermore, the semantic category and speed information of each point in the current frame are predicted. The speed estimation task can output the two-dimensional (2D) speed of each point, which is the 2D speed in the X and Y directions in the current coordinate system. The semantic segmentation task outputs the semantic segmentation category of each point in the point cloud of the current frame, including movable objects, ground, curbs, fences, green plants, etc.

[0053] Furthermore, based on the multimodal fusion features, the feature points are screened according to the preset rules, and the feature points that meet the requirements are used as the reserved feature points. All the reserved feature points constitute the reserved feature point set. Then, the millimeter wave point cloud data obtained by the vehicle-mounted millimeter wave radar scanning is obtained, and the point set belonging to the same object is identified according to the millimeter wave point cloud. Then, based on the identified point set, a minimum three-dimensional bounding box is generated to surround these points, and the millimeter wave detection frame can be obtained. Next, the x and y coordinates of each feature point in the reserved feature point set are mapped to the corresponding position of the grid map, and the z coordinate is ignored. According to the position of the point, the corresponding grid unit is marked in the grid map, and the reserved feature point set can be projected to the grid map. Similarly, the bottom surface of the millimeter wave detection frame is projected onto the grid map. The boundary of the detection frame is calculated, and the grid units covered by the frame are marked in the raster map, and the millimeter wave detection frame can be projected to the grid map.

[0054] Finally, the retained feature point set is identified and analyzed on the grid map, and the millimeter wave detection frame is used to determine the initial position and range of potential obstacles. Subsequently, a more accurate obstacle detection frame is generated on the grid map by combining the information of the retained feature point set and the millimeter wave detection frame.

[0055] The technical solution of the disclosed embodiment can generate a set of retained feature points and a millimeter wave detection frame, and finally generate an obstacle detection frame by extracting point features, laser point representation features and image point representation features based on the acquired laser point cloud, millimeter wave point cloud and visual image. The disclosed embodiment makes full use of the advantages of various sensors, can effectively alleviate the collision risk caused by missed detection of autonomous driving vehicles, and at the same time avoids the risk of sudden braking caused by false detection of noise through various algorithm designs, can improve the versatility of obstacle detection, realize accurate detection of dynamic obstacles, and improve the detection accuracy of heterogeneous dynamic obstacles.

[0056] In some embodiments, the laser point cloud includes not only the laser point cloud of the current frame, but also the laser point cloud of the historical frame. According to the acquired laser point cloud, point features and laser point representation features corresponding to any feature point in the laser point cloud are extracted, including: extracting point features corresponding to the feature points after dimensional expansion of the laser point cloud of the current frame; projecting the laser point cloud of the current frame and the laser point cloud of the historical frame onto a bird's-eye view plane image, and extracting laser point representation features.

[0057] In the embodiment of the present disclosure, the historical frame laser point cloud refers to the previous frame laser point cloud of the current frame laser point cloud.

[0058] In the disclosed embodiment, dimensional expansion refers to adding information of other dimensions, such as color, intensity, normal vector, etc., based on the original three-dimensional coordinates to enrich the features of point cloud data. Dimension expansion of laser point cloud can more accurately identify and describe feature points through the expanded dimensional information, thereby improving the analysis and processing capabilities of point cloud data.

[0059] In the disclosed embodiment, projecting the current frame laser point cloud and the historical frame laser point cloud onto a bird's-eye view plane image means projecting the three-dimensional point cloud data in the current frame laser point cloud and the historical frame laser point cloud onto a horizontal two-dimensional plane, which can convert the three-dimensional data into a two-dimensional plane, thereby simplifying the data processing and analysis process.

[0060] In this way, by expanding the dimension of the current frame laser point cloud, feature points can be more accurately identified and described, thereby improving the analysis and processing capabilities of point cloud data. By projecting the current frame laser point cloud and the historical frame laser point cloud onto a bird's-eye view plane image, three-dimensional data can be converted to a two-dimensional plane, thereby simplifying the process of data processing and analysis.

[0061] In some embodiments, after the current frame laser point cloud is dimensionally expanded, point features corresponding to the feature points are extracted, including: based on the current frame laser point cloud, using a neural network to perform dimensional expansion to obtain enhanced point cloud data; based on the enhanced point cloud data, extracting point features corresponding to the feature points.

[0062] In the disclosed embodiment, the input point cloud data is processed through a trained neural network model to add additional dimensional information. After the neural network processing, the enhanced point cloud data obtained contains not only the original spatial coordinates, but also the expanded feature information. Exemplarily, the expanded feature information may include the color, reflection intensity, normal vector and other features of each point, and other features may also be expanded according to actual conditions. The above is only an exemplary description and is not intended to limit all possible situations of dimensional expansion, but it is not exhaustive here.

[0063] Exemplarily, the process of using a neural network to perform dimensional expansion based on the laser point cloud of the current frame to obtain enhanced point cloud data may include: preprocessing the laser point cloud data of the current frame, which may include denoising, normalization, and coordinate conversion, etc., to ensure that the data is suitable for input into the neural network; selecting a neural network architecture suitable for processing laser point cloud data, such as a point cloud neural network (PointNet), a deep graph convolutional neural network (Deep Graph Convolutional Neura1Network, DGCNN), etc.; training the neural network with annotated data sets to enable it to learn the ability to extract and expand features from the laser point cloud. During the training process, the network will optimize its weights to minimize the prediction error; input the processed point cloud data into the trained neural network, and the network will output enhanced point cloud data, which includes the coordinates of the original points and the expanded feature dimensions. The above is only an exemplary description, and is not intended to limit all possible situations in which the enhanced point cloud data can be obtained by using a neural network to perform dimensional expansion based on the laser point cloud of the current frame, but it is not exhaustive here.

[0064] In the disclosed embodiment, extracting point features corresponding to feature points means identifying and extracting feature information related to feature points from enhanced point cloud data, and the feature information may include the spatial position, normal vector, color information, etc. of the feature points. By extracting these features, the environment or objects can be better described and analyzed, and the performance in subsequent tasks can be improved.

[0065] Exemplarily, the process of extracting point features corresponding to feature points based on enhanced point cloud data may include: local neighborhood analysis, for each feature point, defining a local neighborhood of a spherical or cubic range, collecting nearby point data within this range; feature descriptor calculation, calculating point features within the local neighborhood. The above is only an exemplary description and is not intended to limit all possible situations for extracting point features corresponding to feature points based on enhanced point cloud data, but it is not exhaustive here.

[0066] In this way, by expanding the dimension of the laser point cloud, the originally sparse and discrete point cloud data can be made richer and more useful, thereby improving the effect of analysis and application. By extracting point features based on the enhanced point cloud data, the environment or object can be better described and analyzed, improving the performance in subsequent tasks.

[0067] In some embodiments, the laser point cloud of the current frame and the laser point cloud of the historical frame are projected onto a bird's-eye view plane image to extract laser point representation features, including: projecting the laser point cloud of the current frame and the laser point cloud of the historical frame onto a bird's-eye view plane image to obtain a current frame bird's-eye view projection and a historical frame bird's-eye view projection; determining the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features respectively according to the current frame bird's-eye view projection and the historical frame bird's-eye view projection; performing feature interaction and feature fusion on the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features to obtain bird's-eye features; interpolating the bird's-eye view features to extract laser point representation features.

[0068] In the disclosed embodiments, manual features refer to features designed and extracted by human experts based on domain knowledge and experience in data processing. These features are used to help machine learning models understand and process data. Manual features are usually some statistics or descriptive indicators extracted from the original data, which can better represent the key attributes of the data. Exemplarily, in the field of autonomous driving, manual features include but are not limited to maximum height, minimum height and average intensity, and other manual features can also be selected according to actual conditions. The above is only an exemplary explanation and is not intended to limit all possible situations of manual features, but it is not exhaustive here.

[0069] In the disclosed embodiment, since the three-dimensional laser point cloud data of the current frame and the historical frame are projected onto the two-dimensional bird's-eye view plane respectively, and this projection focuses on the information of the ground plane and ignores details such as height and intensity, it is necessary to use manual features to supplement the corresponding details. For example, the maximum height can be obtained by finding the maximum height value of all points in the bird's-eye view projection, and using this value as the maximum height; similarly, the minimum height can be obtained by finding the minimum height value of all points in the bird's-eye view projection, and using this value as the minimum height; similarly, the average intensity can be obtained by calculating the average intensity of all points in the bird's-eye view projection, and using this value as the average intensity. In particular, the height value and intensity value of each point exist in the data of the corresponding point in the laser point cloud.

[0070] In the disclosed embodiment, the bird's-eye view features obtained after feature interaction and feature fusion of the bird's-eye view manual features of the current frame and the historical frame can provide an environment description that combines time and space information. Exemplarily, feature interaction can identify the relationship and changes between the current frame and the historical frame by comparing and analyzing the manual features of the current frame and the historical frame. In the field of autonomous driving, it can help capture the movement of dynamic objects or changes in the environment. Further, feature fusion can include integrating the features of the current frame and the historical frame to form a unified feature representation. The feature fusion process can be fused by weighted averaging, feature splicing, or using a deep learning model. The above is only an exemplary description and is not intended to limit all possible situations for feature interaction and feature fusion of manual features, but it is not exhaustive here.

[0071] In the disclosed embodiment, since the bird's-eye view features are usually extracted from a sparse laser point cloud, there may be spatial discontinuities. Interpolation processing is used to fill these gaps and generate a more continuous and complete feature representation, which can be achieved through a variety of methods, such as linear interpolation, bilinear interpolation, or more complex algorithms. Subsequently, the interpolated bird's-eye view features can be mapped back to the original laser point cloud, and each laser point can obtain a feature representation, and the feature representations of all points are combined to generate the laser point representation feature.

[0072] In the disclosed embodiment, the point clouds of the current frame and the historical frame can be projected onto the bird's-eye view plane to calculate manual features such as the maximum height, minimum height, and average intensity, respectively, to introduce spatial information and historical information into the model, and then a two-dimensional U-shaped network (2D-UNet) structure is used to perform sufficient cross-scale feature interaction and cross-scale feature fusion to obtain the bird's-eye view features. Then, quadratic linear interpolation is used to transform the bird's-eye view features into point representations, and the point representations of all points constitute the laser point representation features.

[0073] In this way, by projecting the laser point cloud of the current and historical frames onto a bird's-eye view plane image, the dynamic changes of the vehicle's surrounding environment can be more intuitively understood. Feature interaction and fusion utilize time series information to more accurately identify and track dynamic objects. Through interpolation processing and feature extraction, sparse data can be better processed to reduce the probability of false detection and missed detection.

[0074] In some embodiments, the laser point cloud of the current frame is projected onto the visual image, and the image point representation features corresponding to the feature points are determined in combination with the image features extracted from the visual image, including: extracting the image features based on the visual image; projecting the laser point cloud of the current frame onto the visual image to obtain a point cloud fused image; and determining the image point representation features based on the point cloud fused image and the image features.

[0075] In the disclosed embodiment, the laser point cloud of the current frame is projected onto the visual image, and the three-dimensional point cloud data from the laser radar can be combined with the two-dimensional image data of the visual camera, and the generated point cloud fusion image can provide rich environmental information. Exemplarily, the process of projecting the laser point cloud of the current frame onto the visual image to obtain the point cloud fusion image can include: ensuring that the coordinate systems of the laser radar and the camera are aligned through a calibration process; projecting the three-dimensional laser point cloud data onto the two-dimensional camera image plane, and using the camera's focal length, principal point and other internal parameters and rotation matrix, translation vector and other external parameters for geometric transformation; combining the projected point cloud data with the visual image to generate a point cloud fusion image, which contains both the depth information of the laser radar and the color information of the camera.

[0076] In the disclosed embodiment, the process of determining the features represented by image points based on point cloud fusion images and image features is obtained by combining the three-dimensional depth information of each feature point extracted from the point cloud fusion image with the image information, which can better describe objects and scenes in the environment.

[0077] In the disclosed embodiment, an image encoder may be first used to extract image features, and then the laser point cloud may be projected onto the image by calculating the pose transformation between the camera and the lidar. The image features corresponding to each point may then be calculated by quadratic linear interpolation, and the image features corresponding to each point may be combined into image point representation features.

[0078] In this way, by combining the depth information of the laser point cloud with the color and texture information of the visual image, the system can perceive the surrounding environment more accurately. This fusion provides a more comprehensive scene representation, which helps to identify and classify objects. Combining multimodal data can reduce errors caused by the limitations of a single sensor. The depth information provided by the lidar can compensate for the shortcomings of the camera in the case of changing lighting and lack of texture. The point cloud fusion image obtained by fusing image and point cloud features can more accurately detect and track dynamic and static objects, improving the performance of the autonomous driving system in complex traffic environments.

[0079] Figure 2 The process of obtaining multimodal fusion features based on laser point cloud and visual image is shown, as Figure 2 As shown, firstly, laser feature extraction is performed. On the one hand, a multi-layer perceptron is used to expand the dimension of the laser point cloud of the current frame to extract point features; on the other hand, the laser point cloud of the current frame and the laser point cloud of the historical frame are projected to the bird's-eye view (Bird's-Eye-View, BEV) plane to calculate manual features such as maximum height, minimum height and average intensity, and then the features are interacted and fused through the 2D-UNet structure to obtain the bird's-eye view features, and then the bird's-eye view features are transformed into BEV point representation features using quadratic linear interpolation, that is, the laser point representation features in the embodiment of the present disclosure. Then, image feature extraction is performed, and image features are extracted using an image encoder, and then the point cloud is projected to the image by calculating the pose transformation between the camera and the radar, and then the image features corresponding to each point are calculated by quadratic linear interpolation, and finally the image point representation features are obtained. Then, multimodal feature fusion is performed to splice point features, laser point representation features, and image point representation features. Then, a multi-layer perceptron is used to fully fuse the multimodal features to obtain multimodal fusion features. Finally, the multimodal fusion features are sent to the neural network to perform semantic segmentation and speed estimation on each point in the current frame to predict its semantic category and speed information. The speed estimation task can output the 2D speed of each point, including the 2D speed in the X and Y directions in the current coordinate system, while the semantic segmentation task can output the semantic segmentation category of each point in the point cloud of the current frame, including movable objects, ground, curbs, fences, green plants, etc.

[0080] In some embodiments, based on multimodal fusion features and millimeter wave point clouds, a set of retained feature points and a millimeter wave detection frame are generated, and the retained feature point set and the millimeter wave detection frame are projected onto a grid map, including: filtering feature points according to multimodal fusion features, generating a set of retained feature points according to a first filtering result, and projecting the retained feature point set onto a grid map; filtering the millimeter wave point cloud, generating a millimeter wave detection frame according to a second filtering result, and projecting the millimeter wave detection frame onto a grid map.

[0081] In the disclosed embodiment, the process of screening feature points according to the multimodal fusion features is to select useful feature points from all feature points according to the multimodal fusion features and preset rules, and to remove noise or irrelevant information to retain the most representative feature points. Subsequently, the coordinates of the retained feature points are converted into the coordinates of the grid map, and then the feature points are marked on the grid map according to the converted grid coordinates.

[0082] In the disclosed embodiment, the process of screening the millimeter wave point cloud is to select useful points from all millimeter wave points according to preset rules, and to remove noise or irrelevant information to retain the most representative feature points. Subsequently, the three-dimensional coordinates of the millimeter wave detection frame are converted into two-dimensional coordinates of the grid map, and the projection boundary of the detection frame on the grid map is calculated according to the size and position of the detection frame, and finally the area of ​​the detection frame is marked on the grid map.

[0083] In this way, through multi-level screening and fusion of feature points and millimeter wave point clouds, false alarms and missed alarms can be effectively reduced, and the overall performance of the autonomous driving system can be improved.

[0084] In some embodiments, feature points are screened according to multimodal fusion features, a set of retained feature points is generated according to a first screening result, and the set of retained feature points is projected to a grid map, including: determining the semantic category of any feature point according to the multimodal fusion features; screening according to the semantic category, extracting all feature points that meet the preset semantic category, and generating a set of retained feature points; projecting all retained feature points in the set of retained feature points to the grid map.

[0085] In the disclosed embodiment, the multimodal fusion features may include semantic categories. The process of screening feature points based on the multimodal fusion features may be to screen according to the semantic categories of the feature points, eliminate feature points corresponding to fixed objects such as road surface points, fence points, curb points and green plant points, and only retain feature points corresponding to movable objects. The feature points corresponding to the movable objects are used as the first screening results, and then a set of retained feature points is generated.

[0086] In the disclosed embodiment, all feature points are traversed, and road surface points, fence points, curb points, and green plant points are filtered according to the semantic categories of the feature points, and only movable object categories are retained. The feature points under the movable object category are used as the first screening result, and then a set of retained feature points is generated.

[0087] In this way, through semantic screening, we can better understand the types of objects in the surrounding environment, more effectively detect and track important objects, optimize computing resources, reduce unnecessary data processing, and thus reduce the risk of accidents and improve driving safety.

[0088] In some embodiments, the millimeter wave point cloud is screened, a millimeter wave detection frame is generated based on a second screening result, and the millimeter wave detection frame is projected onto a grid map, including: screening the target points in the millimeter wave point cloud, extracting all target points that meet preset dynamic and static categories, and obtaining a dynamic target point set; clustering all dynamic target points in the dynamic target point set to obtain a clustering result; generating a millimeter wave detection frame based on the clustering result, and projecting the millimeter wave detection frame onto the grid map.

[0089] In the disclosed embodiment, the information of each millimeter wave point in the millimeter wave point cloud includes speed information measured based on the Doppler effect, which indicates the speed of the object relative to the radar and helps identify moving objects. Further, based on the speed information, the millimeter wave points are divided into dynamic categories and static categories. Based on the classification of the millimeter wave points, all millimeter wave points classified into the dynamic category are used as target points. All millimeter wave points are traversed, and a dynamic target point set is obtained based on all dynamic target points.

[0090] In the disclosed embodiments, a clustering algorithm can be used to extract features used for clustering, such as spatial coordinates, velocity vectors, etc., and then points belonging to the same object can be classified into one category. By converting the set of points into objects, the motion state and trajectory of the object can be more accurately estimated, which helps to more accurately understand and predict the environment, which helps to identify and track different moving objects.

[0091] In this way, by screening and clustering dynamic target points, moving objects such as pedestrians and other vehicles can be more accurately identified and distinguished, which can improve the accuracy of target detection and reduce false positives and false negatives. The clustering of dynamic target points and the generation of detection frames help the autonomous driving system better understand the dynamic changes of the surrounding environment and identify potential traffic participants and obstacles.

[0092] In some embodiments, an obstacle detection frame is generated based on a set of retained feature points and a millimeter wave detection frame on a grid map, including: determining the millimeter wave existence of each grid based on the millimeter wave detection frame on the grid map; determining the obstacle existence in each grid based on the set of retained feature points on the grid map; traversing all grids in the grid map based on the millimeter wave existence and obstacle existence to obtain an obstacle grid set; clustering the obstacle grid set, and generating an obstacle detection frame based on the clustering results.

[0093] In the embodiment of the present disclosure, a grid map is a two-dimensional grid structure for representing spatial information. The basic component unit of the grid map is a grid, and each grid represents a small area in the space.

[0094] In the disclosed embodiment, based on the millimeter wave detection box on the grid map, it can be determined whether there is a millimeter wave in each grid. When there is a millimeter wave in the grid, it means that there is a dynamic object in the grid, and there may be an obstacle.

[0095] In the disclosed embodiment, based on the set of retained feature points on the grid map, it is possible to determine whether there is a movable object in the grid. When there is a movable object in the grid, it means that there may be an obstacle in the grid.

[0096] In the embodiment of the present disclosure, the process of traversing all grids in the grid map to obtain the obstacle grid set refers to screening out all grids where obstacles may exist, and then obtaining the obstacle grid set.

[0097] In the disclosed embodiment, clustering the obstacle grid set and generating the obstacle detection frame according to the clustering result means grouping the grids marked as obstacles in the grid map to identify and distinguish different obstacles. Clustering algorithms such as density-based spatial clustering of applications with noise (DBSCAN) and K-means clustering algorithm (K-means) are used to group these obstacle grids. The clustering algorithm classifies adjacent or close grids into a group according to the spatial position of the grid. Different obstacle groups can be identified through the clustering results, and each group represents an independent obstacle.

[0098] In this way, by combining the results of the two detection methods in the grid map, the existence of obstacles in each grid can be judged more accurately, thereby reducing false positives and missed positives. The obstacle grid set is quickly processed by the clustering algorithm, and the detection box is generated, which can update the information of the vehicle's surrounding environment in real time.

[0099] In some embodiments, the millimeter wave existence of each grid is determined according to the millimeter wave detection frame on the grid map, including: judging whether there is an overlapping area between any grid and the millimeter wave detection frame according to the millimeter wave detection frame on the grid map; when the corresponding grid and the millimeter wave detection frame have an overlapping area, the millimeter wave existence of the corresponding grid is set to 1, otherwise it is set to 0.

[0100] In the disclosed embodiment, the detection frame provided by the millimeter wave radar is a rectangular area, indicating the spatial range where the radar detects that an object may exist; the grid map is a gridded representation of the environment, and each grid represents a fixed spatial area. By calculation, it is determined whether each grid overlaps with the millimeter wave detection frame. If any part of a grid overlaps with the detection frame, it is considered that the grid may contain an obstacle.

[0101] In the disclosed embodiment, if any part of a grid overlaps with the detection frame, the millimeter wave existence of the corresponding grid is set to 1, indicating that the grid may contain an obstacle, otherwise it means that the grid does not contain an obstacle.

[0102] In this way, by judging the overlap between the grid and the millimeter wave detection frame, the location of obstacles can be identified more accurately, reducing false positives and missed positives. Through simple 0 or 1 marking, large amounts of data can be processed quickly, improving real-time performance and computing efficiency.

[0103] In some embodiments, the existence of obstacles in each grid is determined based on the set of retained feature points on the grid map, including: judging whether there are retained feature points in any grid based on the set of retained feature points on the grid map; determining the speed information of any retained feature point in the corresponding grid based on the multimodal fusion features; determining the average speed information of all retained feature points in the corresponding grid based on the speed information; determining the average speed information of all retained feature points in the corresponding grid based on the speed information; when there are retained feature points in the corresponding grid and the average speed information is greater than a preset average speed threshold, the obstacle existence of the corresponding grid is set to 1, otherwise it is set to 0.

[0104] In the disclosed embodiment, it is checked whether each grid contains any retained feature points. If a grid contains feature points, it means that the grid may contain obstacles or important environmental information.

[0105] In the disclosed embodiment, the multimodal fusion feature contains speed information. Based on the speed information in the multimodal fusion feature, the speed is calculated for each feature point in the grid, and then the average speed of the entire grid is calculated. The average speed is compared with a preset threshold. If there is a feature point in a grid and its average speed exceeds the threshold, the grid is marked as possibly containing an obstacle; otherwise, it is marked as not containing an obstacle.

[0106] In the disclosed embodiment, if a feature point exists in a grid and its average speed exceeds a threshold, the obstacle existence of the corresponding grid is set to 1, indicating that the grid may contain an obstacle, otherwise it means that the grid does not contain an obstacle.

[0107] In this way, by quickly calculating the speed information and average speed of feature points, it is possible to respond in time and adapt to complex traffic environments. By setting the speed threshold, it is possible to effectively distinguish between stationary objects and moving obstacles and reduce the false alarm rate.

[0108] In some embodiments, based on millimeter wave existence and obstacle existence, all grids in the grid map are traversed to obtain an obstacle grid set, including: traversing all grids in the grid map, extracting grids whose millimeter wave existence is 1 and obstacle existence is 1, and generating an obstacle grid set.

[0109] In the disclosed embodiment, each grid in the entire grid map is checked, and during the traversal process, grids that satisfy the millimeter wave existence of 1 and the obstacle existence of 1 are found, and all grids that satisfy the above conditions are collected to form a new set, called the obstacle grid set.

[0110] In the embodiment of the present disclosure, the millimeter wave existence is 1 and the obstacle existence is 1, which means that at the grid position, not only the existence of a movable object is detected, but also the object has been confirmed to be a moving obstacle.

[0111] In this way, double confirmation improves the accuracy and reliability of obstacle detection, thereby enhancing the safety and decision-making ability of autonomous driving.

[0112] Figure 3 The process of obtaining the obstacle detection frame based on multimodal fusion features and millimeter wave point cloud is shown as follows: Figure 3 As shown in the figure, first traverse the feature points, filter the road surface points, fence points, curb points and green plant points according to the semantic segmentation results of the laser point cloud, and combine the speed estimation results to retain only the movable object category. Then project the point to the BEV grid map, and calculate the average speed, height and number of points of each grid. Then preprocess the millimeter wave point cloud, remove the static points, and retain only the dynamic points. After clustering the dynamic points to generate the detection frame, the detection frame is projected to the BEV grid map, and the millimeter wave existence of each grid is calculated. Finally, according to the grid map, all obstacle detection frames of the current frame are output, and each grid of the grid map is traversed. If the number of points in the current grid is 0 and there is no millimeter wave, it is set to an empty state. If the number of points in the current grid is greater than 0 but the average speed is less than 0.3 and there is no millimeter wave, it is set to static. If the above two states are not met, it is set to dynamic. Finally, all dynamic grids in the grid map are clustered, and obstacle detection frames are generated according to the clustering information.

[0113] It should be understood that Figures 2 to 3 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figures 2 to 3 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0114] The present disclosure provides a multi-modal obstacle detection device, such as Figure 4 As shown, the device may include: a laser feature extraction module 401, which is used to extract point features and laser point representation features corresponding to any feature point in the laser point cloud according to the acquired laser point cloud; the laser point cloud includes the current frame laser point cloud; an image feature extraction module 402, which is used to project the current frame laser point cloud to the visual image, and determine the image point representation features corresponding to the feature points in combination with the image features extracted from the visual image; a multimodal feature extraction module 403, which is used to obtain multimodal fusion features according to the point features, the laser point representation features and the image point representation features; a grid projection module 404, which is used to generate a reserved feature point set and a millimeter wave detection frame based on the multimodal fusion features and the millimeter wave point cloud, and project the reserved feature point set and the millimeter wave detection frame to the grid map; an obstacle detection module 405, which is used to generate an obstacle detection frame according to the reserved feature point set and the millimeter wave detection frame on the grid map.

[0115] In some embodiments, the laser point cloud also includes a historical frame laser point cloud; the laser feature extraction module 401 includes: a first laser feature extraction submodule, used to extract point features corresponding to feature points after dimensional expansion of the current frame laser point cloud; a second laser feature extraction submodule, used to project the current frame laser point cloud and the historical frame laser point cloud onto a bird's-eye view plane image to extract features represented by the laser points.

[0116] In some embodiments, the first laser feature extraction submodule is used to: perform dimensional expansion using a neural network based on the laser point cloud of the current frame to obtain enhanced point cloud data; and extract point features corresponding to the feature points based on the enhanced point cloud data.

[0117] In some embodiments, the second laser feature extraction submodule is used to: project the current frame laser point cloud and the historical frame laser point cloud onto a bird's-eye view plane image to obtain a current frame bird's-eye view projection and a historical frame bird's-eye view projection; determine the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features respectively according to the current frame bird's-eye view projection and the historical frame bird's-eye view projection; perform feature interaction and feature fusion on the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features to obtain a bird's-eye view feature; interpolate the bird's-eye view feature to extract the laser point representation feature.

[0118] In some embodiments, the image feature extraction module 402 includes: an image feature extraction submodule, which is used to extract image features based on the visual image; a point cloud fusion submodule, which is used to project the current frame laser point cloud onto the visual image to obtain a point cloud fusion image; and an image feature determination submodule, which is used to determine the image point representation features based on the point cloud fusion image and the image features.

[0119] In some embodiments, the grid projection module 404 includes: a point projection submodule, which is used to filter feature points according to multimodal fusion features, generate a set of retained feature points according to a first filtering result, and project the set of retained feature points to a grid map; a frame projection submodule, which is used to filter the millimeter wave point cloud, generate a millimeter wave detection frame according to a second filtering result, and project the millimeter wave detection frame to a grid map.

[0120] In some embodiments, the point projection submodule is used to: determine the semantic category of any feature point based on the multimodal fusion features; filter according to the semantic category, extract all feature points that meet the preset semantic category, and generate a set of retained feature points; project all retained feature points in the set of retained feature points to a grid map.

[0121] In some embodiments, the frame projection submodule is used to: screen the target points in the millimeter wave point cloud, extract all target points that meet the preset dynamic and static categories, and obtain a dynamic target point set; cluster all dynamic target points in the dynamic target point set to obtain a clustering result; generate a millimeter wave detection frame based on the clustering result, and project the millimeter wave detection frame to a grid map.

[0122] In some embodiments, the obstacle detection module 405 includes: a millimeter wave existence determination submodule, which is used to determine the millimeter wave existence of each grid according to the millimeter wave detection frame on the grid map; an obstacle existence determination submodule, which is used to determine the obstacle existence in each grid according to the set of retained feature points on the grid map; a grid traversal submodule, which is used to traverse all grids in the grid map based on the millimeter wave existence and the obstacle existence to obtain an obstacle grid set; a detection frame generation submodule, which is used to cluster the obstacle grid set and generate an obstacle detection frame according to the clustering result.

[0123] In some embodiments, the millimeter wave existence determination submodule is used to: determine whether there is an overlapping area between any grid and the millimeter wave detection frame based on the millimeter wave detection frame on the grid map; when the corresponding grid and the millimeter wave detection frame have an overlapping area, the millimeter wave existence of the corresponding grid is set to 1, otherwise it is set to 0.

[0124] In some embodiments, an obstacle existence determination submodule is used to: determine whether there are retained feature points in any grid based on the set of retained feature points on the grid map; determine the speed information of any retained feature point in the corresponding grid based on the multimodal fusion features; determine the average speed information of all retained feature points in the corresponding grid based on the speed information; when there are retained feature points in the corresponding grid and the average speed information is greater than a preset average speed threshold, set the obstacle existence of the corresponding grid to 1, otherwise it is set to 0.

[0125] In some embodiments, the grid traversal submodule is used to traverse all grids in the grid map, extract grids whose millimeter wave existence is 1 and obstacle existence is 1, and generate an obstacle grid set.

[0126] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0127] The multimodal obstacle detection device of the disclosed embodiment can detect various types of dynamic obstacles and improve the detection accuracy of heterogeneous dynamic obstacles.

[0128] The present disclosure provides a schematic diagram of a multi-modal obstacle detection scenario. Figure 5 shown.

[0129] As mentioned above, the multimodal obstacle detection method provided by the embodiment of the present disclosure is applied to an electronic device. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can be installed on an autonomous driving vehicle or connected to an autonomous driving vehicle.

[0130] Specifically, the electronic device can perform the following operations:

[0131] According to the acquired laser point cloud, extract the point feature and the laser point representation feature corresponding to any feature point in the laser point cloud;

[0132] Project the laser point cloud of the current frame onto the visual image, and determine the image point representation feature corresponding to the feature point by combining the image features extracted from the visual image;

[0133] According to the point features, laser point representation features and image point representation features, a multimodal fusion feature is obtained;

[0134] Based on the multimodal fusion features and the millimeter wave point cloud, a set of retained feature points and a millimeter wave detection frame are generated, and the set of retained feature points and the millimeter wave detection frame are projected onto a grid map;

[0135] Generate an obstacle detection frame based on the set of retained feature points on the grid map and the millimeter wave detection frame.

[0136] It should be understood that Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0137] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0138] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle.

[0139] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0140] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0141] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0142] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the multimodal obstacle detection method. For example, in some embodiments, the multimodal obstacle detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the multimodal obstacle detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the multimodal obstacle detection method in any other appropriate manner (eg, by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0148] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0149] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0150] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal obstacle detection method, comprising: Extracting point features and laser point representation features corresponding to any feature point in the laser point cloud according to the acquired laser point cloud; the laser point cloud includes the laser point cloud of the current frame; Projecting the laser point cloud of the current frame onto a visual image, and combining the image features extracted from the visual image to determine the image point representation features corresponding to the feature points; Obtaining a multimodal fusion feature according to the point feature, the laser point representation feature and the image point representation feature; Based on the multimodal fusion features and the millimeter wave point cloud, generating a set of retained feature points and a millimeter wave detection frame, and projecting the set of retained feature points and the millimeter wave detection frame onto a grid map; An obstacle detection frame is generated according to the set of retained feature points on the grid map and the millimeter wave detection frame.

2. The method according to claim 1, wherein: The laser point cloud also includes a historical frame laser point cloud; The step of extracting point features and laser point representation features corresponding to any feature point in the laser point cloud according to the acquired laser point cloud comprises: After dimensional expansion of the laser point cloud of the current frame, extracting the point features corresponding to the feature points; The current frame laser point cloud and the historical frame laser point cloud are projected onto a bird's-eye view plane image, and the laser point representation features are extracted.

3. The method according to claim 2, wherein: After the dimension expansion of the laser point cloud of the current frame is performed, the point features corresponding to the feature points are extracted, including: According to the laser point cloud of the current frame, a neural network is used to perform dimension expansion to obtain enhanced point cloud data; The point features corresponding to the feature points are extracted according to the enhanced point cloud data.

4. The method according to claim 2, wherein: The step of projecting the current frame laser point cloud and the historical frame laser point cloud onto a bird's-eye view plane image and extracting the laser point representation features includes: Projecting the current frame laser point cloud and the historical frame laser point cloud onto the bird's-eye view plane image to obtain a current frame bird's-eye view projection and a historical frame bird's-eye view projection; Determine the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features respectively according to the current frame bird's-eye view projection and the historical frame bird's-eye view projection; Performing feature interaction and feature fusion on the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features to obtain bird's-eye view features; The bird's-eye view features are interpolated to extract the laser point representation features.

5. The method according to claim 1, wherein: The projecting the laser point cloud of the current frame onto a visual image and combining the image features extracted from the visual image to determine the image point representation features corresponding to the feature points comprises: Extracting the image features according to the visual image; Projecting the laser point cloud of the current frame onto the visual image to obtain a point cloud fusion image; Based on the point cloud fusion image and the image features, the image point representation features are determined.

6. The method according to claim 1, wherein: The generating a reserved feature point set and a millimeter wave detection frame based on the multimodal fusion feature and the millimeter wave point cloud, and projecting the reserved feature point set and the millimeter wave detection frame to a grid map includes: Screening the feature points according to the multimodal fusion features, generating the set of retained feature points according to a first screening result, and projecting the set of retained feature points onto the grid map; The millimeter wave point cloud is screened, the millimeter wave detection frame is generated according to a second screening result, and the millimeter wave detection frame is projected onto the grid map.

7. The method according to claim 6, wherein: The step of screening the feature points according to the multimodal fusion features, generating the set of retained feature points according to a first screening result, and projecting the set of retained feature points to the grid map includes: Determining the semantic category of any of the feature points according to the multimodal fusion features; Screening according to the semantic category, extracting all the feature points that meet the preset semantic category, and generating the set of retained feature points; All the retained feature points in the retained feature point set are projected onto the grid map.

8. The method according to claim 6, wherein: The screening of the millimeter wave point cloud, generating the millimeter wave detection frame according to the second screening result, and projecting the millimeter wave detection frame onto the grid map includes: Screening the target points in the millimeter wave point cloud, extracting all the target points that meet the preset dynamic and static categories, and obtaining a dynamic target point set; Clustering all dynamic target points in the dynamic target point set to obtain a clustering result; The millimeter wave detection frame is generated according to the clustering result, and the millimeter wave detection frame is projected onto the grid map.

9. The method according to claim 1, wherein: The generating an obstacle detection frame according to the set of retained feature points on the grid map and the millimeter wave detection frame comprises: Determining the millimeter wave existence of each grid according to the millimeter wave detection frame on the grid map; Determining the existence of obstacles in each grid according to the set of retained feature points on the grid map; Based on the existence of the millimeter wave and the existence of the obstacle, traverse all grids in the grid map to obtain an obstacle grid set; The obstacle grid set is clustered, and the obstacle detection frame is generated according to the clustering result.

10. The method according to claim 9, wherein: The determining the existence of the millimeter wave of each grid according to the millimeter wave detection frame on the grid map includes: According to the millimeter wave detection frame on the grid map, determining whether there is an overlapping area between any of the grids and the millimeter wave detection frame; When there is an overlapping area between the corresponding grid and the millimeter wave detection frame, the millimeter wave existence corresponding to the grid is set to 1, otherwise it is set to 0.

11. The method according to claim 9, wherein: The determining the existence of obstacles in each grid according to the set of retained feature points on the grid map includes: According to the set of retained feature points on the grid map, determining whether there is a retained feature point in any of the grids; Determining, according to the multimodal fusion features, speed information corresponding to any of the retained feature points in the grid; Determine, according to the speed information, the average speed information of all the retained feature points in the grid; When the reserved feature point exists in the corresponding grid and the average speed information is greater than a preset average speed threshold, the obstacle existence of the corresponding grid is set to 1, otherwise it is set to 0.

12. The method according to any one of claims 10 or 11, wherein: The step of traversing all grids in the grid map based on the existence of the millimeter wave and the existence of the obstacle to obtain an obstacle grid set includes: All grids in the grid map are traversed, grids whose millimeter wave existence is 1 and obstacle existence is 1 are extracted, and the obstacle grid set is generated.

13. A multimodal obstacle detection device, comprising: Laser feature extraction module; Used to extract point features and laser point representation features corresponding to any feature point in the laser point cloud according to the acquired laser point cloud; the laser point cloud includes the laser point cloud of the current frame; An image feature extraction module, used to project the laser point cloud of the current frame onto a visual image, and determine the image point representation feature corresponding to the feature point in combination with the image features extracted from the visual image; A multimodal feature extraction module, used for obtaining a multimodal fusion feature according to the point feature, the laser point representation feature and the image point representation feature; A grid projection module, used to generate a set of retained feature points and a millimeter wave detection frame based on the multimodal fusion features and the millimeter wave point cloud, and project the set of retained feature points and the millimeter wave detection frame to a grid map; The obstacle detection module is used to generate an obstacle detection frame according to the set of retained feature points on the grid map and the millimeter wave detection frame.

14. The device according to claim 13, wherein: The laser point cloud also includes a historical frame laser point cloud; The image feature extraction module comprises: A first laser feature extraction submodule, configured to extract the point features corresponding to the feature points after dimensional expansion of the laser point cloud of the current frame; The second laser feature extraction submodule is used to project the current frame laser point cloud and the historical frame laser point cloud onto a bird's-eye view plane image to extract the laser point representation features.

15. The device according to claim 14, wherein: The first laser feature extraction submodule is used to: According to the laser point cloud of the current frame, a neural network is used to perform dimension expansion to obtain enhanced point cloud data; The point features corresponding to the feature points are extracted according to the enhanced point cloud data.

16. The device according to claim 14, wherein: The second laser feature extraction submodule is used to: Projecting the current frame laser point cloud and the historical frame laser point cloud onto the bird's-eye view plane image to obtain a current frame bird's-eye view projection and a historical frame bird's-eye view projection; Determine the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features respectively according to the current frame bird's-eye view projection and the historical frame bird's-eye view projection; Performing feature interaction and feature fusion on the current frame bird's-eye view manual features and the historical frame bird's-eye view manual features to obtain bird's-eye view features; The bird's-eye view features are interpolated to extract the laser point representation features.

17. The device according to claim 13, wherein: The image feature extraction module comprises: An image feature extraction submodule, used for extracting the image features according to the visual image; A point cloud fusion submodule, used for projecting the laser point cloud of the current frame onto the visual image to obtain a point cloud fusion image; The image feature determination submodule is used to determine the image point representation feature based on the point cloud fusion image and the image feature.

18. The device according to claim 13, wherein: The grid projection module comprises: A point projection submodule, configured to screen the feature points according to the multimodal fusion features, generate the set of retained feature points according to a first screening result, and project the set of retained feature points to the grid map; The frame projection submodule is used to screen the millimeter wave point cloud, generate the millimeter wave detection frame according to the second screening result, and project the millimeter wave detection frame onto the grid map.

19. The device according to claim 18, wherein: The point projection submodule is used for: Determining the semantic category of any of the feature points according to the multimodal fusion features; Screening according to the semantic category, extracting all the feature points that meet the preset semantic category, and generating the set of retained feature points; All the retained feature points in the retained feature point set are projected onto the grid map.

20. The device according to claim 18, wherein The frame projection submodule is used to: Screening the target points in the millimeter wave point cloud, extracting all the target points that meet the preset dynamic and static categories, and obtaining a dynamic target point set; Clustering all dynamic target points in the dynamic target point set to obtain a clustering result; The millimeter wave detection frame is generated according to the clustering result, and the millimeter wave detection frame is projected onto the grid map.

21. The device according to claim 13, wherein: The obstacle detection module comprises: A millimeter wave existence determination submodule, used to determine the millimeter wave existence of each grid according to the millimeter wave detection frame on the grid map; An obstacle existence determination submodule, used to determine the existence of obstacles in each grid according to the set of retained feature points on the grid map; A grid traversal submodule, used to traverse all grids in the grid map based on the existence of the millimeter wave and the existence of the obstacle to obtain an obstacle grid set; The detection frame generation submodule is used to cluster the obstacle grid set and generate the obstacle detection frame according to the clustering result.

22. The device according to claim 21, wherein The millimeter wave existence determination submodule is used to: According to the millimeter wave detection frame on the grid map, determining whether there is an overlapping area between any of the grids and the millimeter wave detection frame; When there is an overlapping area between the corresponding grid and the millimeter wave detection frame, the millimeter wave existence corresponding to the grid is set to 1, otherwise it is set to 0.

23. The device according to claim 21, wherein The obstacle existence determination submodule is used to: According to the set of retained feature points on the grid map, determining whether there is a retained feature point in any of the grids; Determining, according to the multimodal fusion features, speed information corresponding to any of the retained feature points in the grid; Determine, according to the speed information, the average speed information of all the retained feature points in the grid; When the reserved feature point exists in the corresponding grid and the average speed information is greater than a preset average speed threshold, the obstacle existence of the corresponding grid is set to 1, otherwise it is set to 0.

24. The device according to any one of claims 22 or 23, wherein: The grid traversal submodule is used to traverse all grids in the grid map, extract grids whose millimeter wave existence is 1 and whose obstacle existence is 1, and generate the obstacle grid set.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause a determination machine to perform a method according to any one of claims 1-12.

27. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 12 when executed by a processor.

28. An autonomous driving vehicle comprising the electronic device as claimed in claim 25.

Images