Obstacle detection method, device, apparatus and storage medium

Abstract

The application discloses an obstacle detection method, device and equipment and a storage medium. The method extracts a feature tensor of an image from image features by using a setting ray and a three-dimensional sampling point, and detects obstacles based on the feature tensor. Compared with the prior art, the method does not need to comprehensively identify and label the collected image, reduces the feature analysis amount, improves the analysis efficiency, accurately positions and identifies effective features based on the three-dimensional sampling point, improves the identification accuracy, further avoids the interference of label noise, and focuses on stable detection of obstacles.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an obstacle detection method, apparatus, device, and storage medium. Background Technology

[0002] Nearest neighbor obstacles refer to the nearest neighbor obstacles that appear outwards from the vehicle in any direction in a top-down view during autonomous driving perception tasks. These obstacles are unaffected by any occlusion factors and are the obstacles most relevant to the vehicle's interaction. Ensuring that these objects are always accurately detected is crucial for the safety of autonomous driving.

[0003] Currently, in the nearest neighbor obstacle detection methods of autonomous driving control technology, all objects in the space are uniformly labeled. Deep learning object detection networks need to detect all objects in the space simultaneously. However, the impact of object occlusion makes it difficult to confirm the existence of many objects, and their outlines are hard to determine, resulting in labeling noise. At the same time, a large number of objects that are unrelated to the vehicle's movement (such as a large number of vehicles and non-motorized vehicles parked in off-street parking lots) are also labeled and required to be detected. This also increases the burden on network training, and focusing too much on some irrelevant objects may actually lead to a reduction in the detection performance of important objects. Summary of the Invention

[0004] The main purpose of this application is to solve the problem of low detection efficiency and accuracy in existing nearest neighbor obstacle detection methods, which rely on comprehensive identification and labeling.

[0005] The first aspect of the present invention provides an obstacle detection method, the obstacle detection method comprising:

[0006] Acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data;

[0007] Taking the main vehicle as the origin, multiple three-dimensional sampling points are selected by extending outward in different directions;

[0008] Based on the image acquisition parameters of the main vehicle, the image coordinates of the three-dimensional sampling point in the image features are determined, and the feature tensor of the corresponding three-dimensional sampling point is extracted based on the image coordinates;

[0009] Obstacle prediction is performed based on the aforementioned feature tensors, and obstacles in the driving scenario are determined based on the prediction results.

[0010] In a first implementation of the first aspect of the present invention, the step of acquiring image data of the driving scene where the main vehicle is located and extracting image features from the image data includes:

[0011] Using an image acquisition device installed in the main vehicle, at least one image is captured from different directions with the main vehicle as the center, and image data of the driving scene where the main vehicle is located is generated based on the images from each direction;

[0012] Using a pre-defined convolutional neural network and feature pyramid network, features are extracted from the image data at different scales to obtain image features.

[0013] In a second implementation of the first aspect of the present invention, the step of selecting multiple three-dimensional sampling points extending outward in different directions from the main vehicle as the origin includes:

[0014] With the main vehicle as the origin, place rays at different directional positions and extend the rays outward to construct a three-dimensional ray model with the main vehicle as the coordinate origin;

[0015] K three-dimensional sampling points are selected from near to far on each ray.

[0016] In a third implementation of the first aspect of the present invention, the step of determining the image coordinates of the three-dimensional sampling point in the image features based on the image acquisition parameters of the main vehicle, and extracting the feature tensor of the corresponding three-dimensional sampling point based on the image coordinates, includes:

[0017] Based on the intrinsic and extrinsic parameters of the image acquisition device, the correspondence between the three-dimensional sampling points and the pixels on the image features is determined, and the image coordinates of each three-dimensional sampling point in the image features are calculated based on the correspondence.

[0018] Based on the image coordinates, extract the positional features of the same three-dimensional sampling point in the image features, and generate a feature tensor with the image coordinates.

[0019] In a fourth implementation of the first aspect of the present invention, the step of extracting positional features belonging to the same three-dimensional sampling point from the image features based on the image coordinates, and generating a feature tensor with the image coordinates, includes:

[0020] Based on the image coordinates, the features at the same three-dimensional sampling point in the image features are collected using a collection operator to obtain the corresponding content features;

[0021] The image coordinates of each 3D sampling point are superimposed onto the corresponding content features to generate a multidimensional feature tensor.

[0022] In a fifth implementation of the first aspect of the present invention, the step of predicting obstacles based on each of the feature tensors and determining obstacles in the driving scenario based on the prediction results includes:

[0023] Using a preset feature fusion algorithm, the feature tensors corresponding to each three-dimensional sampling point are fused to obtain a fused feature tensor.

[0024] Extract the parameters from each of the fused feature tensors, and identify obstacles in the driving scene based on the parameters.

[0025] In a sixth implementation of the first aspect of the present invention, the step of using a preset feature fusion algorithm to fuse the feature tensors corresponding to each three-dimensional sampling point to obtain a fused feature tensor includes:

[0026] Convolve all three-dimensional sampling points according to the ray adjacency and point adjacency relationships of the corresponding tensor features to obtain the first fused feature tensor.

[0027] The first fused feature tensors are fused and calculated using a self-attention mechanism according to different frames and different ray directions to obtain the fused feature tensor.

[0028] A second aspect of the present invention provides an obstacle detection device, the obstacle detection device comprising:

[0029] The feature extraction module is used to acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data;

[0030] The selection module is used to select multiple three-dimensional sampling points by extending outward in different directions with the main vehicle as the origin.

[0031] The tensor extraction module is used to determine the image coordinates of the three-dimensional sampling points in the image features based on the image acquisition parameters of the main vehicle, and to extract the feature tensors of the corresponding three-dimensional sampling points based on the image coordinates.

[0032] An obstacle detection module is used to predict obstacles based on the aforementioned feature tensors, and to determine the obstacles in the driving scenario based on the prediction results.

[0033] In a first implementation of the second aspect of the present invention, the feature extraction module includes:

[0034] The acquisition unit is used to capture at least one image from different directions with the main vehicle as the center, using an image acquisition device installed in the main vehicle, and to generate image data of the driving scene where the main vehicle is located based on the images from each direction;

[0035] The feature extraction unit is used to extract features from the image data at different scales using a preset convolutional neural network and feature pyramid network to obtain image features.

[0036] In a second implementation of the second aspect of the present invention, the selection module includes:

[0037] A construction unit is used to place rays at different directional positions with the main vehicle as the origin, and extend the rays outward to construct a three-dimensional ray model with the main vehicle as the coordinate origin;

[0038] The selection unit is used to select K three-dimensional sampling points from near to far on each ray.

[0039] In a third implementation of the second aspect of the present invention, the tensor extraction module includes:

[0040] The coordinate calculation unit is used to determine the correspondence between the three-dimensional sampling points and the pixels on the image features based on the intrinsic and extrinsic parameters of the image acquisition device, and to calculate the image coordinates of each three-dimensional sampling point in the image features based on the correspondence.

[0041] The tensor generation unit is used to extract the positional features of the same three-dimensional sampling point in the image features based on the image coordinates, and generate a feature tensor with the image coordinates.

[0042] In a fourth implementation of the second aspect of the present invention, the tensor generation unit is specifically used for:

[0043] Based on the image coordinates, the features at the same three-dimensional sampling point in the image features are collected using a collection operator to obtain the corresponding content features;

[0044] The image coordinates of each 3D sampling point are superimposed onto the corresponding content features to generate a multidimensional feature tensor.

[0045] In a fifth implementation of the second aspect of the present invention, the obstacle detection module includes:

[0046] The tensor fusion unit is used to fuse the feature tensors corresponding to each three-dimensional sampling point using a preset feature fusion algorithm to obtain a fused feature tensor.

[0047] An obstacle detection unit is used to extract parameters from each of the fused feature tensors and identify obstacles in the driving scenario based on the parameters.

[0048] In a sixth implementation of the second aspect of the present invention, the tensor fusion unit is specifically used for:

[0049] Convolution operations are performed on the corresponding tensor features of all three-dimensional sampling points according to the ray adjacency and point adjacency relationships to obtain the first fused feature tensor.

[0050] The first fused feature tensors are fused and calculated using a self-attention mechanism according to different frames and different ray directions to obtain the fused feature tensor.

[0051] A third aspect of the present invention provides an obstacle detection device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the obstacle detection device to perform the steps of the obstacle detection method described above.

[0052] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the obstacle detection method described above.

[0053] Beneficial effects:

[0054] The technical solution provided by this invention acquires image data of the driving scene where the main vehicle is located, extracts image features from the image data, selects multiple three-dimensional sampling points extending outward in different directions with the main vehicle as the origin, determines the image coordinates of the three-dimensional sampling points in the image features based on the image acquisition parameters of the main vehicle, and extracts the feature tensors of the corresponding three-dimensional sampling points based on the image coordinates. Obstacle prediction is performed based on each feature tensor, and the nearest neighbor obstacle in the driving scene is determined based on the prediction results. By setting rays and three-dimensional sampling points to extract the image feature tensors from the image features, the nearest neighbor obstacle is detected based on the feature tensors. Compared with the prior art, it does not require comprehensive recognition and annotation of the acquired images, reducing the amount of feature analysis and thus improving analysis efficiency. At the same time, it achieves accurate positioning and recognition of effective features based on three-dimensional sampling points, improves recognition accuracy, and further avoids the interference of annotation noise, focusing on stable detection of nearest neighbor obstacles. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the first embodiment of the obstacle detection method provided by the present invention;

[0056] Figure 2 A schematic diagram of a second embodiment of the obstacle detection method provided by the present invention;

[0057] Figure 3 A schematic diagram of a third embodiment of the obstacle detection method provided by the present invention;

[0058] Figure 4 A schematic diagram of an embodiment of the obstacle detection device provided in this invention;

[0059] Figure 5 A schematic diagram of another embodiment of the obstacle detection device provided in this invention;

[0060] Figure 6This is a schematic diagram of one embodiment of the obstacle detection device provided in this invention. Detailed Implementation

[0061] To address the shortcomings of existing technologies, this application employs a method of extracting image features by setting rays and three-dimensional sampling points. Specifically, it involves emitting extended rays from various directions in space and sampling feature points. Using camera intrinsic and extrinsic parameters, the ray sampling points in 3D space are projected onto the image to obtain the overall features of the 3D ray. These features are then further subjected to one-dimensional convolution in the tangential direction, and information is fused using a self-attention mechanism. Ultimately, the features from each different ray direction predict the parameters of an obstacle. Finally, Hungarian matching is performed between these obstacle predictions and the ground truth values, and the loss function under the optimal matching is calculated. This loss function serves as the final network loss function to supervise network training, thereby improving detection efficiency and accuracy.

[0062] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the obstacle detection method provided by this invention is mainly applied in the obstacle avoidance environment of autonomous driving vehicles. It detects nearest-neighbor obstacles among those detected in the autonomous vehicle to achieve interactive obstacle avoidance and ensure the safety of autonomous driving. The method specifically includes the following steps:

[0064] 101. Acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data;

[0065] It is understood that the executing entity of this invention can be an obstacle detection device, a vehicle-mounted terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a vehicle-mounted terminal as an example.

[0066] In this embodiment, the main vehicle is equipped with a vehicle steer-by-wire system, an image acquisition system, and an image processing control system. The vehicle steer-by-wire system consists of a steering wheel assembly, a steering actuator assembly, and auxiliary systems such as an automatic fault prevention system and a power supply. The steering wheel assembly includes a steering wheel, a steering wheel angle sensor, a torque sensor, and a steering wheel return torque motor. The steering actuator assembly includes a front wheel angle sensor, a steering actuator motor, a steering motor controller, and a front wheel steering component.

[0067] In this step, during the driving of the main vehicle, at least one image is captured from different directions with the main vehicle as the center by an image acquisition device installed in the main vehicle, and image data of the driving scene where the main vehicle is located is generated based on the images from each direction; using a preset convolutional neural network and feature pyramid network, features are extracted from the image data at different scales to obtain image features.

[0068] Specifically, with the main vehicle as the center, data acquisition signals are emitted from multiple directions such as up, down, left, and right through the radar equipment on the main vehicle. After the data acquisition signals collide with obstacles, data is acquired based on the collision results. Alternatively, different depths of field can be set for the image acquisition equipment to take multiple pictures in each direction. Based on the multiple pictures, image processing technology is used to stitch and merge the objects in the pictures to obtain an image. Then, three-dimensional image data is constructed based on the images from each direction.

[0069] Then, convolutional neural networks are used to perform convolution calculations at different scales on the image data to extract the image features of key points. A feature pyramid network is used to extract pixels that resemble obstacles from the image data. The extracted pixel features are then used to construct a pyramid feature structure tree based on the correlation between pixels. Finally, image features are constructed by matching and fusing the pyramid feature structure tree and the image features of key points. Here, the image features can be a feature set based on pixels.

[0070] In practical applications, in addition to the above methods, image data can also be acquired through input. That is, in the obstacle detection test environment of the main vehicle, image data in the test environment is collected in advance by external acquisition devices and labeled by human annotation to build training image data. Then, it is output to the main vehicle, where the image recognition neural network in the main vehicle is used to identify image features and extract key point image features.

[0071] 102. Using the main vehicle as the origin, select multiple three-dimensional sampling points by extending outwards in different directions;

[0072] In this embodiment, with the main vehicle as the center, the radar or ray generator on the main vehicle is controlled to emit rays in multiple directions such as up, down, left, and right to construct a spatial model centered on the main vehicle. Multiple curved surfaces at different distances from the main vehicle are set in the spatial model, and the focus of the curved surface and the ray is used as a three-dimensional sampling point. Then, the coordinate information of each three-dimensional sampling point is calculated with the main vehicle as the origin. The three-dimensional sampling point is used to collect corresponding image features at different time frames.

[0073] 103. Based on the image acquisition parameters of the main vehicle, determine the image coordinates of the three-dimensional sampling points in the image features, and extract the feature tensors of the corresponding three-dimensional sampling points based on the image coordinates;

[0074] In this embodiment, the image acquisition parameters of the main vehicle should be understood as the configuration parameters of the image acquisition device of the main vehicle to acquire images of the actual environment, such as the intrinsic and extrinsic parameters of the camera. Based on the intrinsic and extrinsic parameters, the captured environmental data is converted into coordinate image data of the corresponding canvas. Based on the configuration parameters, the projection relationship between the three-dimensional sampling points and the image is calculated, that is, the coordinate scaling ratio and the positional correspondence between each pixel.

[0075] Based on the projection relationship and position correspondence, the projection coordinates of the three-dimensional sampling points in the image are calculated based on the coordinates of the three-dimensional sampling points, which are the image coordinates. Finally, based on the image coordinates, the corresponding image content is extracted from the extracted image features, and a feature tensor corresponding to the three-dimensional sampling points is constructed based on the image content. This feature tensor includes two features: one is the position feature, and the other is the content feature.

[0076] In practical applications, if the image data consists of multiple frames, the step of extracting the feature tensor of each 3D point includes:

[0077] For each frame of the image, feature extraction is performed using the method described above to obtain the feature tensor for each frame. Then, the feature tensors of all frames at the same 3D sampling point are fused to obtain the final feature tensor for the 3D sampling point. Alternatively, regression calculation is used to perform regression calculation on the feature tensors of all frames at the same 3D sampling point, and the feature tensors corresponding to frames with similar regression values ​​are extracted and fused to obtain the final feature tensor for the 3D sampling point. Further, the fusion is specifically implemented by performing channel analysis on the image features corresponding to the feature tensor, generating feature vectors from the decomposed image features of each channel, and then deduplicating and merging the feature vectors to obtain the final feature tensor for the 3D sampling point.

[0078] 104. Based on the feature tensors, predict obstacles and determine the obstacles in the driving scene based on the prediction results.

[0079] In this embodiment, the three-dimensional sampling point can be understood as a pixel. Then, the similarity of the feature vectors is calculated, and multiple feature tensors with close similarity are fused into one. Based on the fused feature tensor, multi-dimensional parameters are parsed, and based on the dimensional parameters, matching is performed from the obstacle database to obtain the nearest neighbor obstacle.

[0080] Of course, it can also be predicted using obstacle prediction models. Specifically, the model identifies the parameter values ​​in the feature tensor, matches the parameter values ​​with the actual parameter values ​​of different types of obstacles, and determines the obstacles based on the matching results.

[0081] In this embodiment, the obstacle determined here is actually the nearest neighbor obstacle, that is, the characteristics of the actual obstacle calculated above. Based on these characteristics, the nearest neighbor obstacle is determined. Specifically, in the process of determining whether an obstacle belongs to an obstacle, the sensing distance between the driver vehicle and the obstacle is also calculated, and based on the sensing distance, it is determined whether the obstacle belongs to the nearest neighbor obstacle. Specifically, the distance between the driver vehicle's current position and the coordinate position of the obstacle is calculated, and based on the distance, it is determined whether it is in the nearest neighbor obstacle.

[0082] In this embodiment, the above steps can be implemented by training a model that includes a convolutional network for image feature extraction, a 3D sampling point selection network, a feature tensor extraction network, and an obstacle detection network. The methods for each step are implemented through these networks.

[0083] Furthermore, the model is trained by combining images with pre-marked obstacles with 3D ray sampling points. Specifically, after predicting the obstacles, we emit rays from the vehicle's position in all directions from the 3D obstacle annotation data. The first object encountered by these rays is the nearest neighbor obstacle.

[0084] Since an obstacle is detected in every direction, all detection results are considered as a set. The ground truth labels of the nearest neighbor obstacles are also a set. Therefore, the distance between the sets of obstacle boxes is used as the loss function. Specifically, for any pair of predicted and ground truth boxes, the loss function includes classification loss, center position regression loss, angle regression loss, size regression loss, and velocity regression loss. Next, the Hungarian algorithm is used to calculate the minimum loss among all possible pairings, which is then used as the final loss function to supervise the training of the neural network.

[0085] This embodiment acquires image data of the driving scene where the main vehicle is located, extracts image features from the image data, selects multiple three-dimensional sampling points extending outward in different directions with the main vehicle as the origin, determines the image coordinates of the three-dimensional sampling points in the image features based on the image acquisition parameters of the main vehicle, and extracts the feature tensors of the corresponding three-dimensional sampling points based on the image coordinates. Obstacle prediction is performed based on each feature tensor, and the nearest neighbor obstacle in the driving scene is determined based on the prediction results. Compared with the existing technology, it does not require comprehensive recognition and annotation of the acquired images, reducing the amount of feature analysis and thus improving the analysis efficiency. At the same time, it achieves accurate positioning and recognition of effective features based on three-dimensional sampling points, improving the recognition accuracy.

[0086] Please see Figure 2 The second embodiment of the obstacle detection method provided in this invention includes:

[0087] 201. Acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data;

[0088] In this step, the acquisition of image data mainly involves a set of multiple frames of images captured in real time by cameras positioned in different directions on the main vehicle. In practical applications, this image data can also be understood as radar point cloud data. The LiDAR scans the image information in a 360-degree direction centered on the location of the main vehicle and converts the image information into point cloud data.

[0089] Furthermore, for extracting image features from image data, the image feature extraction network in the pre-trained nearest neighbor obstacle detection model is used to extract features from each frame of the image data.

[0090] In practical applications, this image extraction network is a combination of a convolutional neural network and a feature pyramid network. Based on this network structure, it identifies object contours in each frame of the image. The object contours are compared with the labeled contours of obstacles. If the similarity is within a preset range, the image of the region containing the object contour is determined to be a valid image, and it is segmented or copied from the image data. Then, image analysis algorithms are used to perform channel decomposition, extracting features from each channel and generating corresponding vectors. Based on the features and vectors, combined with the channels, rendering and merging processes are performed to obtain the image features. The feature pyramid network is mainly used to associate the extracted image features, specifically by establishing a pyramid structure based on pixel adjacency, forming a pyramid image feature tree structure.

[0091] 202. Using the main vehicle as the origin, place rays at different directional positions and extend the rays outward to construct a three-dimensional ray model with the main vehicle as the coordinate origin;

[0092] In this embodiment, the lidar on the main vehicle is controlled to emit rays extending in different directions in the direction the main vehicle is traveling and / or in the opposite direction of the main vehicle's travel, by means of equidistant offset. For example, with a resolution of 1 degree, a total of 360 rays can be emitted (360° / 1° = 360). A three-dimensional ray model is constructed based on the rays in each direction and the main vehicle. This model is mainly used to set up the main vehicle to collect environmental data at different depths in the driving scene.

[0093] 203. Select K three-dimensional sampling points from near to far along each ray;

[0094] In this embodiment, after constructing the 3D ray model, each ray in the 3D ray model is used as a sampling direction, and multiple sampling points are set on each ray through equidistant offset. Each set of A sampling points forms a sphere, where the distance between these A sampling points and the main vehicle (i.e., the origin of the coordinate system) is equal. Then, K sampling points are selected from the multiple sampling points based on image validity. Image validity here can be understood as the depth of field of the effective image. Of course, the selection of multiple sampling points is based on a certain distance, which is the shortest distance at which the main vehicle can detect obstacles, as preset.

[0095] 204. Based on the intrinsic and extrinsic parameters of the image acquisition device, determine the correspondence between the three-dimensional sampling points and the pixels on the image features, and calculate the image coordinates of each three-dimensional sampling point in the image features based on the correspondence.

[0096] In this embodiment, the intrinsic and extrinsic parameters of the image acquisition device can be understood as the proportional relationship between the pixel coordinates in the canvas used to carry image information in the camera and the coordinates of each point in the actual environment, or as the proportional relationship and positional correspondence between the object and the image.

[0097] Furthermore, based on the three-dimensional sampling points and the origin, the actual coordinate information of each three-dimensional sampling point is calculated. Based on the actual coordinate information and intrinsic and extrinsic parameters, the camera image is calculated using the proportional relationship between the camera image and the actual object. This is the projection coordinate of each three-dimensional sampling point in the image features, which is the image coordinate.

[0098] In this embodiment, the image coordinates of each 3D sampling point located in the image features can also be calculated using reference points:

[0099] First, the center point of the image data and the main vehicle are set as the origin. The first distance from the center point to the edge position of the image data in each direction and the maximum perception distance of the main vehicle are calculated. Based on the first distance and the maximum perception distance, the ratio between the image and the actual image of the driving scene is calculated. Then, a point is selected from the edge position and a point selected from the actual image of the driving scene are used as reference points. Based on the coordinates and directions of the reference point, other three-dimensional sampling points, the image coordinates of each three-dimensional sampling point in the image are calculated.

[0100] 205. Extract the positional features of the same three-dimensional sampling point in the image features based on the image coordinates, and generate a feature tensor with the image coordinates;

[0101] In this embodiment, the feature tensor includes two features: a positional feature and a content feature. The positional feature is the image coordinates, and the content feature is the pixel feature in the image features corresponding to the image coordinates. Based on this, the feature tensor is implemented in the following way:

[0102] Based on the image coordinates, the features at the same three-dimensional sampling point in the image features are collected using a collection operator to obtain the corresponding content features;

[0103] The image coordinates of each 3D sampling point are superimposed onto the corresponding content features to generate a multidimensional feature tensor.

[0104] Specifically, based on image coordinates as the base point, the pixel content at the location of the image coordinates in each frame of the image is collected, including pixel information from different resource channels. Based on the pixel information and pixel content, operations such as superposition and deduplication are performed to obtain the content features of the three-dimensional sampling point. Finally, the position features (i.e., image coordinates) are superimposed on the content features to obtain the feature tensor.

[0105] 206. Using a preset feature fusion algorithm, the feature tensors corresponding to each three-dimensional sampling point are fused to obtain a fused feature tensor.

[0106] In this embodiment, the feature fusion algorithm is specifically a fusion model composed of a convolutional network and a self-attention mechanism. The feature tensors in each three-dimensional sampling point of the model are subjected to convolution operations and cross-fusion to obtain the fused feature tensor.

[0107] Specifically, firstly, all three-dimensional sampling points are convolved with their corresponding tensor features according to the ray-adjacent and point-adjacent relationships to obtain the first fused feature tensor; then, each of the first fused feature tensors is fused and calculated using a self-attention mechanism according to different frames and different ray directions to obtain the fused feature tensor.

[0108] In practical applications, based on a 3D ray model, 3D sampling points on two adjacent rays are selected, and these selected 3D sampling points are convolved using a convolutional network to obtain a first fusion feature tensor. This first fusion feature tensor can be understood as a neighborhood fusion feature tensor. Based on this first fusion feature tensor, feature tensors at the same image coordinates in different frames are further fused to obtain the final fusion feature.

[0109] 207. Extract the parameters from each fused feature tensor and identify obstacles in the driving scenario based on the parameters.

[0110] In this embodiment, the fused feature tensor is actually a multidimensional dataset carrying parameters such as obstacle category, whether it is an obstacle, specific location and size. By identifying the specific values ​​of each parameter in the fused feature tensor, it is detected whether each three-dimensional sampling point belongs to an obstacle point based on the specific values. If so, it is further determined whether the distance between it and the main vehicle is within the safety perception and recognition range of the main vehicle. If so, it is determined that the obstacle is the nearest neighbor obstacle.

[0111] In practical applications, for example, the fused feature tensor is a feature tensor M with dimensions N*360*K*(1+n_cat+2+3+1+2). Here, 1 represents the presence of an obstacle, n_cat represents the type of obstacle, 2 represents the position of the center point, 3 represents the length, width, and height, 1 represents the orientation angle, and 2 represents the velocity component. Obstacles are identified by recognizing the specific values ​​of 1, n_cat, 2, 3, 1, and 2 at their corresponding positions, and this, combined with the perceived distance, determines whether it is a nearest neighbor obstacle.

[0112] In practical applications, this method also includes setting up a neural network learning algorithm, which performs feature learning on steps 201-207 above, and trains and constructs the corresponding network layers respectively.

[0113] In practical applications, the image data used in training and building network layers consists of a set of images with nearest-neighbor obstacles pre-annotated manually. Features are extracted using different neural networks combined with a 3D ray model of the scene to generate feature tensors. These feature tensors are then fused, and obstacle prediction is performed based on the fusion results. Loss is calculated based on the prediction results and the annotated information. The structure and parameters of the corresponding neural networks are then adjusted based on the calculated loss to obtain a better model for subsequent obstacle detection.

[0114] This embodiment extracts the feature tensor of the image from the image features by setting rays and three-dimensional sampling points, and detects the nearest neighbor obstacle based on the feature tensor. Compared with the existing technology, it does not require comprehensive recognition and annotation of the acquired image, reducing the amount of feature analysis and thus improving the analysis efficiency. At the same time, it realizes the accurate positioning and recognition of effective features based on three-dimensional sampling points, improving the recognition accuracy, and further avoiding the interference of annotation noise, focusing on the stable detection of the nearest neighbor obstacle.

[0115] Please see Figure 3 The third embodiment of the obstacle detection method provided by this invention detects nearest-neighbor obstacles based on a test-training approach. Compared to the previous embodiments, it adds a loss calculation scheme to optimize the detection results, specifically including the following steps:

[0116] 301. Input an image centered on the main vehicle and extract image features from the image.

[0117] In this step, multi-scale image features F are extracted using convolutional neural networks (such as residual neural networks) and feature pyramid networks. For each training sample, M images can be input, with a feature size of N*M*H*W*C, where N is the batch size, M is the number of images in each sample, H is the height, W is the width, and C is the number of channels.

[0118] 302. Construct a 3D ray tracing centered on the main vehicle, and select sampling points based on the 3D ray tracing.

[0119] In this step, several rays can be emitted outwards from the vehicle as the origin. For example, with a resolution of 1 degree, a total of 360 rays can be emitted (360° / 1° = 360). For each ray, we select K sampling points from near to far. The size of this tensor is N*1800*K*3, where 3 represents the x, y, and z coordinates of each sampling point.

[0120] 303. Project the sampling points onto the image and calculate the image coordinates of the sampling points in the image.

[0121] In this embodiment, for each sampled 3D point, the 3D point is projected onto the image based on the camera's intrinsic and extrinsic parameters to obtain the row and column coordinates on the image. A sampling feature tensor P for the 3D point is constructed based on these row and column coordinates. The size of tensor P is N*360*K*3, where 3 represents u, v, and flag. Here, u represents the column coordinate, v represents the row coordinate, and flag indicates whether the point is within the image range.

[0122] 304. Collect radiation characteristics.

[0123] Specifically, using the obtained image coordinates, the gather operator (gather_nd) can collect features from the image features F onto each 3D point location. This ultimately yields an N*M*360*K*C dimensional content feature G. Simultaneously, the 3D point location features P are superimposed onto the content feature G, resulting in an N*M*360*K*C dimensional feature tensor Q.

[0124] 305. Radiation information fusion network for obstacle prediction.

[0125] In this step, fusion specifically involves fusing information between adjacent angular rays and radially adjacent points. Convolution operations are used to further obtain the fused feature tensor Q_1. Next, a self-attention mechanism is applied to fuse the features from different frames and different ray directions. Only the features corresponding to the rays in the current frame are retained, ultimately resulting in a feature tensor M with dimensions N*360*K*(1+n_cat+2+3+1+2). Here, 1 represents the presence of an obstacle, n_cat represents the type of obstacle, 2 represents the position of the center point, 3 represents the length, width, and height, 1 represents the orientation angle, and 2 represents the velocity component.

[0126] 306. Selection of nearest neighbor obstacles.

[0127] Specifically, from the 3D obstacle annotation data, rays are emitted in all directions centered on the vehicle's position. The first object encountered by these rays is the nearest neighbor obstacle.

[0128] 307. Loss function calculation.

[0129] In practical applications, since an obstacle is detected in every direction, all detection results are considered as a set. The ground truth labels of the nearest neighbor obstacles are also a set. Therefore, the distance between the sets of obstacle boxes is used as the loss function. Specifically, for any pair of predicted and ground truth boxes, the loss function includes classification loss, center position regression loss, angle regression loss, size regression loss, and velocity regression loss. Next, the Hungarian algorithm is used to calculate the minimum loss among all possible pairings, which is then used as the final loss function to supervise the training of the neural network.

[0130] This embodiment, building upon the previous embodiment, emits extended rays from various directions in space and samples feature points. Using camera intrinsic and extrinsic parameters, the ray sampling points in 3D space are projected onto the image to obtain the overall features of the 3D ray. These features are then further subjected to one-dimensional convolution in the tangential direction, and information is fused using a self-attention mechanism. Ultimately, the features from each different ray direction predict the parameters of an obstacle. Finally, Hungarian matching is performed between these obstacle predictions and the ground truth values, and the loss function under the optimal matching is calculated as the final network loss function to supervise network training.

[0131] The obstacle detection method provided by the embodiments of the present invention has been described above. The obstacle detection device of the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 4 An obstacle detection device according to an embodiment of the present invention includes:

[0132] Feature extraction module 401 is used to acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data;

[0133] The selection module 402 is used to select multiple three-dimensional sampling points extending outward in different directions with the main vehicle as the origin.

[0134] Tensor extraction module 403 is used to determine the image coordinates of the three-dimensional sampling point in the image features according to the image acquisition parameters of the main vehicle, and extract the feature tensor of the corresponding three-dimensional sampling point based on the image coordinates;

[0135] The obstacle detection module 404 is used to predict obstacles based on the feature tensors and determine the obstacles in the driving scenario based on the prediction results.

[0136] In this embodiment of the invention, the obstacle detection device operates the obstacle detection method described above. The obstacle detection device extracts the feature tensor of the image from the image features by setting rays and three-dimensional sampling points, and detects the nearest neighbor obstacle based on the feature tensor. Compared with the prior art, it does not require comprehensive identification and annotation of the acquired image, reducing the amount of feature analysis and thus improving the analysis efficiency. At the same time, it achieves accurate positioning and identification of effective features based on three-dimensional sampling points, improving the identification accuracy, and further avoiding the interference of annotation noise, focusing on the stable detection of the nearest neighbor obstacle.

[0137] Please see Figure 5 A second embodiment of the obstacle detection device provided in this invention specifically includes:

[0138] Feature extraction module 401 is used to acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data;

[0139] The selection module 402 is used to select multiple three-dimensional sampling points extending outward in different directions with the main vehicle as the origin.

[0140] Tensor extraction module 403 is used to determine the image coordinates of the three-dimensional sampling point in the image features according to the image acquisition parameters of the main vehicle, and extract the feature tensor of the corresponding three-dimensional sampling point based on the image coordinates;

[0141] The obstacle detection module 404 is used to predict obstacles based on the feature tensors and determine the obstacles in the driving scenario based on the prediction results.

[0142] In this embodiment, the feature extraction module 401 includes:

[0143] The acquisition unit 4011 is used to capture at least one image from different directions with the main vehicle as the center through an image acquisition device installed in the main vehicle, and generate image data of the driving scene where the main vehicle is located based on the images from each direction.

[0144] The feature extraction unit 4012 is used to extract features from the image data at different scales using a preset convolutional neural network and feature pyramid network to obtain image features.

[0145] In this embodiment, the selection module 402 includes:

[0146] Construction unit 4021 is used to place rays at different directional positions with the main vehicle as the origin, and extend the rays outward to construct a three-dimensional ray model with the main vehicle as the coordinate origin;

[0147] Selecting unit 4022 is used to select K three-dimensional sampling points from near to far on each ray.

[0148] In this embodiment, the tensor extraction module 403 includes:

[0149] The coordinate calculation unit 4031 is used to determine the correspondence between the three-dimensional sampling points and the pixels on the image features according to the intrinsic and extrinsic parameters of the image acquisition device, and to calculate the image coordinates of each three-dimensional sampling point in the image features based on the correspondence.

[0150] Tensor generation unit 4032 is used to extract positional features belonging to the same three-dimensional sampling point from the image features based on the image coordinates, and generate a feature tensor with the image coordinates.

[0151] In this embodiment, the tensor generation unit 4032 is specifically used for:

[0152] Based on the image coordinates, the features at the same three-dimensional sampling point in the image features are collected using a collection operator to obtain the corresponding content features;

[0153] The image coordinates of each 3D sampling point are superimposed onto the corresponding content features to generate a multidimensional feature tensor.

[0154] In this embodiment, the obstacle detection module 404 includes:

[0155] Tensor fusion unit 4041 is used to fuse the feature tensors corresponding to each three-dimensional sampling point using a preset feature fusion algorithm to obtain a fused feature tensor.

[0156] The obstacle detection unit 4042 is used to extract parameters from each of the fused feature tensors and identify obstacles in the driving scene based on the parameters.

[0157] In this embodiment, the tensor fusion unit 4041 is specifically used for:

[0158] Convolve all three-dimensional sampling points according to the ray adjacency and point adjacency relationships of the corresponding tensor features to obtain the first fused feature tensor.

[0159] The first fused feature tensors are fused and calculated using a self-attention mechanism according to different frames and different ray directions to obtain the fused feature tensor.

[0160] This embodiment, based on the previous embodiment, adds other functional modules. Through these modules, the device samples feature points by emitting extended rays from various directions in space. Using the camera's intrinsic and extrinsic parameters, the ray sampling points in 3D space are projected onto the image to obtain the overall features of the 3D ray. These features are further subjected to one-dimensional convolution in the tangential direction, and information fusion is performed using a self-attention mechanism. Ultimately, the features of each different ray direction predict the parameters of an obstacle. Based on these parameters, obstacle recognition is performed, achieving obstacle detection without requiring comprehensive identification and annotation of the acquired images. This reduces the amount of feature analysis and improves analysis efficiency. Simultaneously, the accurate localization and recognition of effective features based on 3D sampling points improves recognition accuracy and further avoids interference from annotation noise, focusing on stable detection of nearest-neighbor obstacles.

[0161] above Figure 4 and Figure 5 The obstacle detection device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The obstacle detection equipment in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0162] Figure 6This is a schematic diagram of the structure of an obstacle detection device 600 provided in an embodiment of the present invention. The obstacle detection device 600 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 633 or data 632. The memory 620 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the obstacle detection device 600. Furthermore, the processor 610 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the obstacle detection device 600 to implement the steps of the obstacle detection method described above.

[0163] The obstacle detection device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The obstacle detection device structure shown does not constitute a limitation on the obstacle detection device provided in this application. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0164] A third aspect of the present invention provides an obstacle detection device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the obstacle detection device to perform the steps of the obstacle detection method described above.

[0165] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the obstacle detection method.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an obstacle detection device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An obstacle detection method, characterized in that, The obstacle detection method includes: Acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data; Taking the main vehicle as the origin, multiple three-dimensional sampling points are selected by extending outward in different directions; Based on the image acquisition parameters of the main vehicle, the image coordinates of the three-dimensional sampling point in the image features are determined, and the feature tensor of the corresponding three-dimensional sampling point is extracted based on the image coordinates; Based on the relationships of ray adjacency and point adjacency, as well as different frames and different ray directions, obstacle prediction is performed based on the aforementioned feature tensors, and obstacles in the driving scene are determined based on the prediction results.

2. The obstacle detection method according to claim 1, characterized in that, The step of acquiring image data of the driving scene where the main vehicle is located, and extracting image features from the image data, includes: Using an image acquisition device installed in the main vehicle, at least one image is captured from different directions with the main vehicle as the center, and image data of the driving scene where the main vehicle is located is generated based on the images from each direction; Using a pre-defined convolutional neural network and feature pyramid network, features are extracted from the image data at different scales to obtain image features.

3. The obstacle detection method according to claim 2, characterized in that, The selection of multiple three-dimensional sampling points extending outward in different directions from the main vehicle as the origin includes: With the main vehicle as the origin, place rays at different directional positions and extend the rays outward to construct a three-dimensional ray model with the main vehicle as the coordinate origin; K three-dimensional sampling points are selected from near to far on each ray.

4. The obstacle detection method according to claim 3, characterized in that, The step of determining the image coordinates of the three-dimensional sampling point in the image features based on the image acquisition parameters of the main vehicle, and extracting the feature tensor of the corresponding three-dimensional sampling point based on the image coordinates, includes: Based on the intrinsic and extrinsic parameters of the image acquisition device, the correspondence between the three-dimensional sampling points and the pixels on the image features is determined, and the image coordinates of each three-dimensional sampling point in the image features are calculated based on the correspondence. Based on the image coordinates, extract the positional features of the same three-dimensional sampling point in the image features, and generate a feature tensor with the image coordinates.

5. The obstacle detection method according to claim 4, characterized in that, The step of extracting location features belonging to the same three-dimensional sampling point from the image features based on the image coordinates, and generating a feature tensor with the image coordinates, includes: Based on the image coordinates, the features at the same three-dimensional sampling point in the image features are collected using a collection operator to obtain the corresponding content features; The image coordinates of each 3D sampling point are superimposed onto the corresponding content features to generate a multidimensional feature tensor.

6. The obstacle detection method according to claim 3, characterized in that, The step of predicting obstacles based on the feature tensors according to the relationships between ray adjacency and point adjacency, as well as different frames and different ray directions, and determining obstacles in the driving scene based on the prediction results includes: Using a preset feature fusion algorithm, the feature tensors corresponding to each three-dimensional sampling point are fused according to the relationship between ray adjacency and point adjacency, as well as different frames and different ray directions, to obtain the fused feature tensor. Extract the parameters from each of the fused feature tensors, and identify obstacles in the driving scene based on the parameters.

7. The obstacle detection method according to claim 6, characterized in that, The method utilizes a preset feature fusion algorithm to fuse the feature tensors corresponding to each 3D sampling point according to the relationships between ray adjacency and point adjacency, as well as different frames and different ray directions, to obtain a fused feature tensor, including: Convolve all three-dimensional sampling points according to the ray adjacency and point adjacency relationships of the corresponding tensor features to obtain the first fused feature tensor. The first fused feature tensors are fused and calculated using a self-attention mechanism according to different frames and different ray directions to obtain the fused feature tensor.

8. An obstacle detection device, characterized in that, The obstacle detection device includes: The feature extraction module is used to acquire image data of the driving scene where the main vehicle is located, and extract image features from the image data; The selection module is used to select multiple three-dimensional sampling points by extending outward in different directions with the main vehicle as the origin. The tensor extraction module is used to determine the image coordinates of the three-dimensional sampling points in the image features based on the image acquisition parameters of the main vehicle, and to extract the feature tensors of the corresponding three-dimensional sampling points based on the image coordinates. The obstacle detection module is used to predict obstacles based on the feature tensors according to the relationships between ray adjacency and point adjacency, as well as different frames and different ray directions, and to determine the obstacles in the driving scene based on the prediction results.

9. An obstacle detection device, characterized in that, The obstacle detection device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; The at least one processor invokes the instructions in the memory to cause the obstacle detection device to perform the steps of the obstacle detection method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the obstacle detection method as described in any one of claims 1-7.

Images