Multi-task object detection method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a multi-task target detection method, device and equipment and a medium, the method comprising: obtaining 2D image features and 3D space coordinates of each of multiple images, each image comprising multiple sub-images under multiple perspectives; performing position alignment of 3D coordinates of the multiple images in a 3D space to obtain aligned 3D space coordinates of each of the multiple images; fusing the 2D image features and the aligned 3D space coordinates of each of the multiple images to obtain 2D image features containing 3D space position information of each of the multiple images; obtaining first embedding features of multiple fixed position points in a BEV space and second embedding features of multiple pre-trained position points in the 3D space; and obtaining semantic segmentation results and 3D target detection results according to the first embedding features, the second embedding features and the 2D image features containing 3D space position information of each of the multiple images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multi-task target detection method, apparatus, device, and medium. Background Technology

[0002] In the field of image processing, there is often a need for multi-task object detection. By processing images, it is necessary to obtain both semantic segmentation results (indicating which locations in the image are objects and what their categories are) and 3D object detection results (indicating which 3D objects, especially moving objects, are present in the image). Taking the processing of vehicle driving images as an example, during vehicle movement, dynamic perception and static perception are usually required. Dynamic perception refers to detecting moving objects around the vehicle, while static perception involves detecting the vehicle's driving environment (e.g., lane lines, driving area).

[0003] Currently, a common approach is to estimate the depth information of the image, then use the depth information to uniformly convert the 2D image features into BEV features in the bird's-eye view space, and then use the BEV features to perform both dynamic and static perception simultaneously.

[0004] However, due to the significant errors in depth estimation, the accuracy of the final BEV features is difficult to guarantee, thus limiting the accuracy of both dynamic and static perception. Similarly, it also limits the accuracy of semantic segmentation and 3D object detection results. Therefore, improving the accuracy of semantic segmentation and 3D object detection results has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of the above problems, a multi-task target detection method, apparatus, device and medium according to embodiments of the present invention are proposed to overcome the above problems or at least partially solve the above problems.

[0006] To address the aforementioned problems, a first aspect of the present invention discloses a multi-task target detection method, the method comprising:

[0007] Obtain the 2D image features and 3D spatial coordinates of multiple frames of images, with each frame including sub-images from multiple viewpoints;

[0008] The 3D coordinates of the multiple frames are aligned in 3D space to obtain the aligned 3D space coordinates of each of the multiple frames.

[0009] The 2D image features of each of the multiple frames are fused with their aligned 3D spatial coordinates to obtain the 2D image features of each of the multiple frames containing 3D spatial location information.

[0010] The first embedding features of multiple fixed location points in the BEV space are obtained, and the second embedding features of multiple pre-trained location points in the 3D space are obtained. The pre-training is: training on image samples carrying semantic segmentation labels and 3D target labels.

[0011] Based on the first embedding feature, the second embedding feature, and the 2D image features containing 3D spatial location information of each of the multiple frames, semantic segmentation results and 3D target detection results are obtained.

[0012] A second aspect of this application provides a multi-task target detection apparatus, the apparatus comprising:

[0013] The image acquisition module is used to obtain the 2D image features and 3D spatial coordinates of multiple frames of images. Each frame of image includes sub-images from multiple viewpoints.

[0014] The position alignment module is used to perform position alignment of the 3D coordinates of the multi-frame images in 3D space to obtain the aligned 3D space coordinates of each of the multi-frame images.

[0015] The fusion module is used to fuse the 2D image features of each of the multiple frames of images with their aligned 3D spatial coordinates to obtain the 2D image features of each of the multiple frames of images containing 3D spatial position information.

[0016] The embedding feature acquisition module is used to obtain the first embedding features of multiple fixed location points in the BEV space and the second embedding features of multiple pre-trained location points in the 3D space. The pre-training is: training that learns image samples carrying semantic segmentation labels and 3D target labels.

[0017] The detection module is used to obtain semantic segmentation results and 3D target detection results based on the first embedded feature, the second embedded feature, and the 2D image features containing 3D spatial location information of each of the multiple frames of images.

[0018] A third aspect of the present invention also discloses an electronic device, comprising:

[0019] One or more processors; and

[0020] One or more machine-readable media storing instructions thereon, when executed by the one or more processors, cause the device to perform the multi-task target detection method as described in the first aspect of the present invention.

[0021] A fourth aspect of the present invention also discloses a computer-readable storage medium storing a computer program that causes a processor to perform the multi-task target detection method as described in the first aspect of the present invention.

[0022] The embodiments of the present invention have the following advantages:

[0023] In this embodiment of the invention, there is no need to estimate the depth information of the image. Instead, multiple frames of images are aligned in 3D space, and then the 2D image features of each frame are fused with their aligned 3D spatial coordinates to obtain 2D image features containing 3D spatial position information for each frame. Then, semantic segmentation and 3D object detection are performed using these 2D image features containing 3D spatial position information, first embedding features of multiple fixed position points in BEV space, and second embedding features of multiple pre-trained position points in 3D space. Therefore, this embodiment of the invention proposes a technical solution for semantic segmentation and 3D object detection that does not rely on the depth information of the image. It is no longer limited by the accuracy of the depth information estimation and proposes a new technical solution for semantic segmentation and 3D object detection using 2D image features containing 3D spatial position information. Compared with technical solutions that rely on the depth information of the image, this improves the accuracy of semantic segmentation and 3D object detection. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of the steps of the multi-task target detection method according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of multi-task target detection using the current image and the previous image as examples in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram illustrating the encoding of 3D spatial location information in an embodiment of the present invention;

[0028] Figure 4 This is a structural block diagram of a multi-task target detection device according to an embodiment of the present invention. Detailed Implementation

[0029] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The applicant proposes the following core technical concept: Using multiple frames of images as input, the 2D image features and 3D spatial coordinates of each frame are obtained. The 3D coordinates of the multiple frames are then aligned in 3D space to avoid errors from BEV space transformation. The 2D image features of each frame are fused with their aligned 3D spatial coordinates to obtain 2D image features containing 3D spatial position information for each frame. These features are then combined with first embedded features of multiple fixed-position points in BEV space and second embedded features of multiple pre-trained position points in 3D space to perform feature interaction, resulting in semantic segmentation and 3D object detection results.

[0031] Reference Figure 1 As shown, Figure 1 A flowchart illustrating the steps of a multi-task target detection method according to an embodiment of the present invention is shown. Figure 1 As shown, a specific step of a multi-task target detection method according to an embodiment of the present invention may include the following steps:

[0032] Step S101: Obtain the 2D image features and 3D spatial coordinates of each of the multiple frames.

[0033] In this embodiment, each frame of a multi-frame image includes sub-images from multiple viewpoints. For example, the multiple viewpoints could be: a frontal view, a left frontal view, a right frontal view, a rearal view, a left rearal view, and a right rearal view, etc. Each frame then includes images acquired simultaneously from these multiple viewpoints. The component performing image acquisition can be an onboard camera. Depending on the arrangement of the onboard cameras on different vehicles, the number of viewpoints will vary, and correspondingly, the number of sub-images included in each frame will also vary.

[0034] In one possible embodiment, the multiple frames can be multiple video frame images extracted from the video stream, including the image at the current moment and the image at the previous moment. The image at the current moment includes multiple sub-images from different viewpoints, and the image at the previous moment also includes the same number of sub-images from different viewpoints. It is understood that as the vehicle moves, the content of each sub-image in the image at the previous moment may differ from the content of each sub-image in the image at the current moment, even though the viewpoints are the same.

[0035] In this embodiment of the invention, 2D image features refer to image features extracted from the two dimensions of image width and height. They are no different from image features in the usual sense. The reason for emphasizing 2D is to distinguish them from 3D spatial coordinates, 3D space, and BEV space.

[0036] In this embodiment of the invention, 3D space specifically refers to the real physical 3D space established with a fixed point on the vehicle (e.g., the center point of the vehicle) as the origin of the coordinate system. 3D space coordinates refer to the coordinate positions within the 3D space specifically referred to in this embodiment. BEV space is essentially 2D space, specifically the 2D space from the BEV perspective (i.e., a top-down view). It should be noted that the BEV perspective differs from the multiple perspectives of multiple sub-images included in each frame of the image acquisition; these multiple perspectives depend on the field of view of multiple image acquisition elements.

[0037] In step S101, obtaining the 2D image features of each of the multiple frames is accomplished by an image feature extraction network. This image feature extraction network can be a pre-trained neural network with image feature extraction capabilities (e.g., a backbone network), independent of the multi-task object detection network used for semantic segmentation and 3D object detection in this embodiment of the invention. Alternatively, the image feature extraction network can be integrated into the multi-task object detection network used for semantic segmentation and 3D object detection in this embodiment of the invention, serving as a sub-network of the multi-task object detection network. Its network parameters are updated as the multi-task object detection network is trained until the training of the multi-task object detection network is completed (the training process of the multi-task object detection network will be described below). This yields an image feature extraction sub-network suitable for multi-task object detection in this embodiment of the invention. Then, the image feature extraction sub-network is used to extract features from the multiple frames of images to obtain the 2D image features of each of the multiple frames.

[0038] In practical applications, for multi-frame images, the 2D image features of each extracted frame can be cached for subsequent use in multi-task object detection, thereby improving the efficiency and real-time performance of multi-task object detection. The efficiency improvement and high real-time performance brought about by caching 2D image features are particularly prominent when multiple frames are involved, including the current time frame image and the previous time frame image.

[0039] When multiple images include the current time frame image and the previous time frame image, obtaining the 2D image features of each of the multiple frames can include the following steps:

[0040] Feature extraction is performed on the image at the current moment to obtain the 2D image features of the image at the current moment, and the 2D image features of the image at the previous moment are read from the pre-cached image at the previous moment; the 2D image features of the image at the current moment are cached to achieve multi-task target detection at the next moment together with the image at the next moment.

[0041] In practical applications, under the aforementioned strategy of "caching 2D image features," when performing multi-task object detection at each time step, it is only necessary to extract features from the image at that time step to obtain the 2D image features for that time step, and then read the pre-cached 2D image features from the previous time step. Similarly, after extracting the 2D image features for that time step, they can be cached for use in the next time step of multi-task object detection.

[0042] In step S101, the 3D spatial coordinates of each of the multiple frames of images are obtained. The specific process is as follows:

[0043] Based on the camera parameters used to acquire multiple frames of images, the transformation matrix T is obtained from the 3D space (the 3D space with a fixed point on the vehicle, for example, the center point of the vehicle, as the origin of the coordinate system) in this embodiment of the application to the image space with depth.

[0044] The image space with depth is a space that considers the depth dimension in addition to the width and height dimensions of the image. Its dimensions are D, HF, and WF. HF and WF are the height and width of the image, which are the same as the "2D" in the 2D image features mentioned above. D is the size of the depth dimension.

[0045] For example, the coordinates of a point in the image space with depth are (uj, vj, dj), and the coordinates of the point in the 3D space transformed from that point (xj, yj, zj) can be calculated using the following formula:

[0046]

[0047] Step S102: Align the 3D coordinates of the multi-frame images in 3D space to obtain the aligned 3D space coordinates of each of the multi-frame images.

[0048] In this embodiment, as the vehicle moves, the origin of the 3D coordinates of each of the multiple frames shifts. The specific shift can be represented by a transformation matrix P obtained based on the vehicle's driving parameters (e.g., vehicle speed). It is understood that the transformation matrix P between any two frames needs to be calculated based on the vehicle's driving parameters corresponding to those two frames.

[0049] Taking the vehicle's center point as the origin of the 3D coordinate system as an example, the vehicle's center point in the current image has shifted compared to the vehicle's center point in the previous image. Therefore, it is necessary to align the 3D coordinates of multiple frames in 3D space to obtain the aligned 3D coordinates of each frame, meaning that the aligned 3D coordinates of multiple frames use the same origin. In the specific alignment process, the origin of the 3D coordinates of one frame can be used as a reference (referred to as the reference origin). A transformation matrix P is then used to align the origins of the 3D coordinates of the other frames to the reference origin.

[0050] In the case of multiple frames including the current time frame and the previous time frame, the 3D coordinates of the multiple frames are aligned in 3D space to obtain the aligned 3D space coordinates of each of the multiple frames, including:

[0051] Keeping the 3D coordinates of the current moment image unchanged, the 3D spatial coordinates of the previous moment image are aligned with the 3D coordinates of the current moment image in 3D space to obtain the aligned 3D spatial coordinates of the previous moment.

[0052] In this embodiment, the transformation matrix P can be obtained based on the vehicle's driving parameters (e.g., vehicle speed) at the current and previous moments. Then, while keeping the 3D coordinates of the current image unchanged, the 3D spatial coordinates of the previous image are multiplied by the transformation matrix P to obtain the aligned 3D spatial coordinates of the previous moment.

[0053] Step S103: Fuse the 2D image features of each of the multiple frames with their aligned 3D spatial coordinates to obtain the 2D image features of each of the multiple frames containing 3D spatial position information.

[0054] This embodiment proposes to fuse the aligned 3D spatial coordinates of an image into the 2D image features of that image, resulting in a new parameter for semantic segmentation and 3D object detection: 2D image features containing 3D spatial location information.

[0055] When multiple frames of images include the current time frame image and the previous time frame image, fusing the 2D image features of each of the multiple frames with their aligned 3D spatial coordinates can include the following steps:

[0056] The 2D image features of the current moment image are fused with the 3D spatial coordinates of the current moment image, and the 2D image features of the previous moment image are fused with the aligned 3D spatial coordinates of the previous moment image.

[0057] During spatial alignment, the 3D spatial coordinates of the current image remain unchanged; only the 3D spatial coordinates of the previous image are adjusted so that their origins are aligned with the origins of the current image's 3D spatial coordinates. Therefore, during fusion, for the current image, the 2D image features are fused with its 3D spatial coordinates; while for the previous image, the 2D image features are fused with its aligned 3D spatial coordinates.

[0058] In one possible implementation, 2D image features containing 3D spatial location information can be obtained by encoding 3D spatial location information, including the following steps:

[0059] The aligned 3D spatial coordinates of the frame image are processed by the convolution module and the first activation function module to obtain 3D features;

[0060] The 2D image features of this frame are processed by the convolution module and the second activation function module to obtain 2D features;

[0061] The 3D features are fused with the 2D features to obtain 3D spatial location features;

[0062] The 3D spatial location features are fused with the 2D image features of the frame to obtain the 2D image features of the frame containing 3D spatial location information.

[0063] The first activation function module can be a ReLU activation function module, and the second activation function module can be a sigmoid activation function module.

[0064] In this embodiment, as Figure 3 As shown, encoding 3D spatial location information for any frame of a multi-frame image (e.g., the current frame or the previous frame) includes: on one hand, processing the aligned 3D spatial coordinates of the frame image using two 1x1 convolutional modules and a ReLU activation function module to obtain 3D features; on the other hand, processing the 2D image features of the frame image using two 1x1 convolutional modules and a sigmoid activation function module to obtain 2D features, where the 2D and 3D features have the same dimension; then, fusing the 3D features of the frame image with the 2D features. Figure 3 Taking multiplication as an example, the 3D spatial location features are obtained; finally, the 3D spatial location features of the frame image are fused with the 2D image features of the frame image. Figure 3 Taking direct addition as an example, we obtain the 2D image features of the frame containing 3D spatial location information.

[0065] In the case of multiple frames including the current time frame and the previous time frame, in the step of obtaining 3D features, for the current time frame, two 1x1 convolutional modules and a ReLU activation function module are used to process the 3D spatial coordinates of the current time frame to obtain the 3D features of the current time frame; while for the previous time frame, two 1x1 convolutional modules and a ReLU activation function module are used to process the aligned 3D spatial coordinates of the previous time frame to obtain the 3D features of the previous time frame.

[0066] Step S104: Obtain the first embedding features of multiple fixed location points in the BEV space, and obtain the second embedding features of multiple pre-trained location points in the 3D space. The pre-training is: training on image samples carrying semantic segmentation labels and 3D target labels.

[0067] In this embodiment, the parameters used for semantic segmentation and 3D object detection, in addition to the aforementioned 2D image features containing 3D spatial location information, also include: a first embedding feature and a second embedding feature. The first embedding feature can be represented in vector form and is called the first query vector. The second embedding feature can also be represented in vector form and is called the second query vector for multiple learnable location points.

[0068] The first embedding features of multiple fixed-location points in the BEV space are obtained, including:

[0069] The BEV space is uniformly divided into multiple BEV space grids, and the center point of each BEV space grid is determined as a multiple fixed location point. The BEV space coordinates of the multiple fixed location points are input into a pre-trained encoder to obtain the first embedded feature, which is the first query vector, providing a location reference for multi-task target detection.

[0070] In this embodiment, the BEV space (which is essentially a 2D space, specifically the 2D space viewed from the BEV perspective (i.e., a top-down view)) is uniformly divided into multiple BEV space grids. The center points of these grids are designated as fixed locations (e.g., the BEV space can be divided into a 16×16 grid, with each center point having a fixed position). The BEV space coordinates of these fixed locations are then input into a pre-trained encoder to obtain the first embedded feature. These fixed locations provide a standard reference when performing multi-task object detection tasks.

[0071] Obtaining the second embedding features of multiple pre-trained location points in the 3D space includes:

[0072] The second embedding feature is obtained by inputting the 3D spatial coordinates of multiple pre-trained 3D spatial location points into a pre-trained encoder.

[0073] The second embedded feature is the second query vector of multiple learnable location points, which is used for querying detection results in multi-task object detection. The multiple pre-trained 3D spatial location points stop updating after the pre-training ends. They are obtained by updating multiple random 3D spatial location points initialized in the 3D space multiple times.

[0074] In this embodiment, firstly, multiple random 3D spatial locations are randomly selected in 3D space (with a fixed point on the vehicle, such as the center point of the vehicle, as the origin of the coordinate system). These multiple random 3D spatial locations are multiple learnable points in 3D space. Then, during the training of the multi-task object detection network used for dynamic and static detection in this embodiment, the 3D spatial coordinates of the multiple random 3D spatial locations are updated as the multi-task object detection network is trained until the multi-task object detection network is trained (the training process of the multi-task object detection network will be described below). This yields multiple pre-trained 3D spatial locations. The 3D spatial coordinates of these pre-trained 3D spatial locations are then input into a pre-trained encoder to obtain the second embedding feature. It is understood that as the multi-task object detection network is trained, the multiple random 3D spatial locations are fixed, becoming multiple pre-trained 3D spatial locations, and consequently, the second embedding feature is also fixed. The first and second embedding features are shared by multiple frames of images. That is, they are input into a pre-trained feature interaction network together with the 2D image features containing 3D spatial location information of multiple frames (see below for details) to obtain semantic segmentation results and 3D object detection results.

[0075] In one possible implementation, after the multi-task object detection network has been trained, if the network parameters of the multi-task object detection network are updated using new samples, then multiple pre-trained 3D spatial location points will also be updated to obtain new 3D spatial location points. Subsequently, multi-task object detection will be performed using the updated multi-task object detection network and new second embedding features generated using the new 3D spatial location points.

[0076] The pre-trained encoder used to generate the first and second embedded features is integrated into the multi-task object detection network for dynamic and static detection in this embodiment of the invention. As a sub-network of the multi-task object detection network, its network parameters are updated as the multi-task object detection network is trained until the training of the multi-task object detection network is completed (the training process of the multi-task object detection network will be described below). Then, the encoder applicable to the embodiment of the invention can be obtained, and the first and second embedded features are generated through the encoder.

[0077] Step S105: Based on the first embedding feature, the second embedding feature, and the 2D image features containing 3D spatial location information of each of the multiple frames, obtain the semantic segmentation result and the 3D target detection result.

[0078] When multiple frames are images of vehicles in motion, the semantic segmentation result represents the position of static driving elements, and the 3D target detection result represents the position of moving objects.

[0079] This embodiment proposes using three parameters for semantic segmentation and 3D object detection: 2D image features containing 3D spatial location information for each of the multiple frames, a first embedding feature, and a second embedding feature. When the multiple frames are vehicle driving images, the semantic segmentation result is a static perception result, representing the position of static driving elements in the vehicle's driving environment, such as the position of lane lines and the driving area. The 3D object detection result is a dynamic perception result, representing the position of moving objects, such as the position of other vehicles or pedestrians around the vehicle.

[0080] Figure 2 Taking a multi-frame image including the current time image and the previous time image as an example, this paper illustrates the process of obtaining semantic segmentation results and 3D object detection results based on the 2D image features containing 3D spatial location information, the first embedding feature, and the second embedding feature of the current time image and the previous time image.

[0081] In one possible implementation, the 3D target detection result is: the 3D target detection result under at least one of the multiple perspectives; and / or the 3D target detection result in the 3D space.

[0082] In practical applications, 3D object detection results are presented in various ways. One presentation method is as follows: the dynamic perception result is the 3D object detection result from any one of multiple viewpoints. For example, multiple viewpoints could be: the vehicle's front view, left front view, right front view, rear view, left rear view, and right rear view, totaling six viewpoints. The 3D object detection result is a 2D planar image from the front viewpoint, along with the planar bounding boxes of other vehicles or pedestrians around the vehicle. Another presentation method is as follows: the 3D object detection result is the 3D object detection result from each of the multiple viewpoints. For example, the 3D object detection result is a 2D planar image from six viewpoints, with the planar bounding boxes of other vehicles or pedestrians around the vehicle in each viewpoint. Yet another presentation method is as follows: the 3D object detection result is a 2D planar image synthesized from multiple viewpoints, with the stereo bounding boxes of other vehicles or pedestrians around the vehicle, such as... Figure 2 As shown.

[0083] In one possible implementation, step S105 includes the following steps:

[0084] The first interaction result between the first embedded feature and the second embedded feature is obtained through a pre-trained feature interaction network;

[0085] Through the pre-trained feature interaction network, a second interaction result is obtained between the first embedded feature and the 2D image features containing 3D spatial location information of each of the multiple frames of images; and a third interaction result is obtained between the second embedded feature and the 2D image features containing 3D spatial location information of each of the multiple frames of images.

[0086] The semantic segmentation result is obtained based on the first interaction result and the second interaction result; the 3D target detection result is obtained based on the second interaction result and the third interaction result.

[0087] A pre-trained feature interaction network is a network with feature interaction capabilities, typically using a transformer network. The network inputs a first embedded feature, a second embedded feature, and 2D image features containing 3D spatial location information from multiple frames into the feature interaction network for various interactions. Specifically, these interactions include: interactions between the first and second embedded features, interactions between the first embedded feature and 2D image features containing 3D spatial location information, and interactions between the second embedded feature and 2D image features containing 3D spatial location information.

[0088] Specifically, semantic segmentation results can be obtained based on the interaction results (i.e., the first interaction result and the second interaction result) obtained by the participation of the first embedded features; 3D object detection results can be obtained based on the interaction results (i.e., the first interaction result and the third interaction result) obtained by the participation of the second embedded features.

[0089] The pre-trained feature interaction network is integrated into the multi-task object detection network for 3D object detection and semantic segmentation in the embodiments of the present invention. As a sub-network of the multi-task object detection network, its network parameters are updated as the multi-task object detection network is trained until the multi-task object detection network is trained (the training process of the multi-task object detection network will be described below). Then, the feature interaction network applicable to the embodiments of the invention can be obtained, and the 3D object detection results and semantic segmentation results are generated through the feature interaction network.

[0090] In combination with the above implementation methods, the multi-task object detection method in this application embodiment is implemented by a pre-trained multi-task object detection network including an encoder and a feature interaction network. The training samples of the multi-task object detection network are image samples carrying semantic segmentation labels and 3D object labels. Taking a multi-task object detection network for multi-task object detection of vehicle driving images as an example, the process of generating training samples can be as follows: acquire multiple frames of vehicle images (each frame includes sub-images from multiple viewpoints) as image samples, then mark the positions of static driving elements such as lane lines and driving areas on each image sample as semantic segmentation labels, and mark the positions of other moving objects such as vehicles and pedestrians around the vehicle on each image sample as 3D object labels.

[0091] The training process of the multi-task object detection network in this embodiment of the invention includes:

[0092] Initialize multiple random 3D spatial locations in the 3D space;

[0093] Using the image samples carrying semantic segmentation labels and 3D object detection labels, a multi-task object detection network including an encoder and a feature interaction network is trained. During the training process, the network parameters of the multi-task object detection network are updated, and the random location points in 3D space are also updated.

[0094] At the end of training, a trained multi-task object detection network including an encoder and a feature interaction network, and multiple pre-trained 3D spatial location points are obtained.

[0095] As mentioned earlier, the pre-trained 3D spatial location points are updated during the training of the multi-task object detection network and cease updating once the network training is complete. How to update the network parameters during the training process using image samples carrying semantic segmentation labels and 3D object detection labels can be found in related technical documentation on neural network training, and will not be elaborated upon here. The principle for updating random 3D spatial location points is similar to the principle for updating the network parameters of the multi-task object detection network, and will not be elaborated upon here either.

[0096] Reference Figure 4 As shown, a structural block diagram of a multi-task target detection device according to an embodiment of the present invention is illustrated. Figure 4 As shown, the device may specifically include the following modules:

[0097] The image acquisition module 401 is used to acquire the 2D image features and 3D spatial coordinates of multiple frames of images, and each frame of image includes sub-images from multiple viewpoints;

[0098] The position alignment module 402 is used to perform position alignment of the 3D coordinates of the multi-frame images in 3D space to obtain the aligned 3D space coordinates of each of the multi-frame images.

[0099] The fusion module 403 is used to fuse the 2D image features of each of the multiple frames of images with their aligned 3D spatial coordinates to obtain the 2D image features of each of the multiple frames of images containing 3D spatial position information.

[0100] The embedding feature acquisition module 404 is used to obtain the first embedding features of multiple fixed location points in the BEV space and the second embedding features of multiple pre-trained location points in the 3D space. The pre-training is: training on image samples carrying semantic segmentation labels and 3D object detection labels.

[0101] The detection module 405 is used to obtain semantic segmentation results and 3D target detection results based on the first embedded feature, the second embedded feature, and the 2D image features containing 3D spatial location information of each of the multiple frames of images.

[0102] Optionally, the multi-frame images include: the current time-lapse image and the previous time-lapse image; the image acquisition module includes:

[0103] The feature extraction submodule is used to extract features from the image at the current time to obtain the 2D image features of the image at the current time, and to read the 2D image features of the image at the previous time from the pre-cached cache.

[0104] The device further includes:

[0105] The caching module is used to cache the 2D image features of the current time step image, so that they can be used together with the image of the next time step image to achieve multi-task object detection in the next time step.

[0106] Optionally, the multi-frame images include: the current time-lapse image and the previous time-lapse image; the position alignment module includes:

[0107] The alignment submodule is used to keep the 3D coordinates of the current moment image unchanged, and to align the 3D spatial coordinates of the previous moment image to the 3D coordinates of the current moment image in 3D space to obtain the aligned 3D spatial coordinates of the previous moment.

[0108] The fusion module includes:

[0109] The fusion submodule is used to fuse the 2D image features of the current time image with the 3D spatial coordinates of the current time image, and to fuse the 2D image features of the previous time image with the aligned 3D spatial coordinates of the previous time image.

[0110] Optionally, the fusion module includes:

[0111] The first processing submodule is used to process the aligned 3D spatial coordinates of each frame of the multi-frame image through the convolution module and the first activation function module to obtain 3D features.

[0112] The second processing submodule is used to process the 2D image features of each frame in the multi-frame image by using the convolution module and the second activation function module to obtain 2D features.

[0113] The first feature fusion submodule is used to fuse the 3D features with the 2D features to obtain 3D spatial location features;

[0114] The second feature fusion submodule is used to fuse the 3D spatial location features with the 2D image features of each frame in the multi-frame image to obtain the 2D image features of the frame containing 3D spatial location information.

[0115] Optionally, the embedding feature acquisition module includes:

[0116] The partitioning submodule is used to uniformly divide the BEV space into multiple BEV space grids, and to determine the center point of the multiple BEV space grids as the multiple fixed position points.

[0117] The first input submodule is used to input the BEV spatial coordinates of the multiple fixed position points into a pre-trained encoder to obtain the first embedded feature;

[0118] The second input submodule is used to input the 3D spatial coordinates of multiple pre-trained 3D spatial location points into a pre-trained encoder to obtain the second embedded feature.

[0119] Optionally, the detection module includes:

[0120] The first acquisition submodule is used to obtain the first interaction result between the first embedded feature and the second embedded feature through a pre-trained feature interaction network;

[0121] The second acquisition submodule is used to obtain, through the pre-trained feature interaction network, a second interaction result between the first embedded feature and the 2D image features containing 3D spatial location information of each of the multiple frames of images, and a third interaction result between the second embedded feature and the 2D image features containing 3D spatial location information of each of the multiple frames of images.

[0122] A semantic segmentation submodule is used to obtain the semantic segmentation result based on the first interaction result and the second interaction result;

[0123] The 3D target detection submodule is used to obtain the 3D target detection result based on the second interaction result and the third interaction result.

[0124] Optionally, the multi-task object detection method is implemented through a pre-trained multi-task object detection network including an encoder and a feature interaction network. The training process of the multi-task object detection network includes:

[0125] Initialize multiple random 3D spatial locations in the 3D space;

[0126] Using the image samples carrying semantic segmentation labels and 3D target labels, a multi-task target detection network including an encoder and a feature interaction network is trained. During the training process, the network parameters of the multi-task target detection network are updated, and the random location points in 3D space are also updated.

[0127] At the end of training, a trained multi-task object detection network including an encoder and a feature interaction network, and multiple pre-trained 3D spatial location points are obtained.

[0128] Optionally, the 3D target detection result is: the 3D target detection result under at least one of the multiple perspectives; and / or the 3D target detection in the 3D space.

[0129] This invention also provides an electronic device that can be used to execute a multi-task target detection method. The electronic device may include a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to execute the multi-task target detection method.

[0130] This invention also provides a computer-readable storage medium storing a computer program that causes a processor to execute the multi-task target detection method as described in this invention.

[0131] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0132] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0137] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0138] The above provides a detailed description of the multi-task target detection method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-task object detection method, characterized in that, The method includes: Obtain the 2D image features and 3D spatial coordinates of multiple frames of images, with each frame including sub-images from multiple viewpoints; The 3D coordinates of the multiple frames are aligned in 3D space to obtain the aligned 3D space coordinates of each of the multiple frames. The 2D image features of each of the multiple frames are fused with their aligned 3D spatial coordinates to obtain the 2D image features of each of the multiple frames containing 3D spatial location information. The first embedding features of multiple fixed location points in the BEV space are obtained, and the second embedding features of multiple pre-trained location points in the 3D space are obtained. The pre-training is: training on image samples carrying semantic segmentation labels and 3D target labels. Based on the first embedding feature, the second embedding feature, and the 2D image features containing 3D spatial location information of each of the multiple frames, semantic segmentation results and 3D target detection results are obtained.

2. The method of claim 1, wherein, The multi-frame images include: the current time-lapse image and the previous time-lapse image; obtaining the 2D image features of each of the multi-frame images includes: Feature extraction is performed on the current image to obtain the 2D image features of the current image, and the 2D image features of the previous image are read from the pre-cached cache. The method further includes: Cache the 2D image features of the current time step image to achieve multi-task object detection in the next time step together with the image of the next time step image.

3. The method according to claim 1 or 2, characterized in that, The multi-frame images include: the current time-lapse image and the previous time-lapse image; the 3D coordinates of the multi-frame images are aligned in 3D space to obtain the aligned 3D space coordinates of each of the multi-frame images, including: Keeping the 3D coordinates of the current image unchanged, the 3D spatial coordinates of the previous image are aligned with the 3D coordinates of the current image in 3D space to obtain the aligned 3D spatial coordinates of the previous image. The fusion of the 2D image features of each of the multiple frames with their aligned 3D spatial coordinates includes: The 2D image features of the current moment image are fused with the 3D spatial coordinates of the current moment image, and the 2D image features of the previous moment image are fused with the aligned 3D spatial coordinates of the previous moment image.

4. The method according to any one of claims 1-2, characterized in that, For each of the multiple image frames, the 2D image features of that frame are fused with its aligned 3D spatial coordinates to obtain the 2D image features of that frame containing 3D spatial location information, including: The aligned 3D spatial coordinates of the frame image are processed by the convolution module and the first activation function module to obtain 3D features; The 2D image features of this frame are processed by the convolution module and the second activation function module to obtain 2D features; The 3D features are fused with the 2D features to obtain 3D spatial location features; The 3D spatial location features are fused with the 2D image features of the frame to obtain the 2D image features of the frame containing 3D spatial location information.

5. The method according to any one of claims 1-2, characterized in that, Obtain the first embedding features of multiple fixed-location points in the BEV space, including: The BEV space is uniformly divided into multiple BEV space grids, and the center point of the multiple BEV space grids is determined as the multiple fixed position points. The BEV spatial coordinates of the multiple fixed location points are input into a pre-trained encoder to obtain the first embedding feature; Obtaining the second embedding features of multiple pre-trained location points in the 3D space includes: The second embedding feature is obtained by inputting the 3D spatial coordinates of multiple pre-trained 3D spatial location points into a pre-trained encoder.

6. The method according to any one of claims 1-2, characterized in that, Based on the first embedding feature, the second embedding feature, and the 2D image features containing 3D spatial location information of each of the multiple frames, semantic segmentation results and 3D object detection results are obtained, including: The first interaction result between the first embedded feature and the second embedded feature is obtained through a pre-trained feature interaction network; Through the pre-trained feature interaction network, a second interaction result is obtained between the first embedded feature and the 2D image features containing 3D spatial location information of each of the multiple frames of images; and a third interaction result is obtained between the second embedded feature and the 2D image features containing 3D spatial location information of each of the multiple frames of images. The semantic segmentation result is obtained based on the first interaction result and the second interaction result; The 3D target detection result is obtained based on the second interaction result and the third interaction result.

7. The method according to any one of claims 1-2, characterized in that, The multi-task object detection method is implemented through a pre-trained multi-task object detection network including an encoder and a feature interaction network. The training process of the multi-task object detection network includes: Initialize multiple random 3D spatial locations in the 3D space; Using the image samples carrying semantic segmentation labels and 3D target labels, a multi-task target detection network including an encoder and a feature interaction network is trained. During the training process, the network parameters of the multi-task target detection network are updated, and the random location points in 3D space are also updated. At the end of training, a trained multi-task object detection network including an encoder and a feature interaction network, and multiple pre-trained 3D spatial location points are obtained.

8. The method according to any one of claims 1-2, characterized in that, The 3D target detection result is: the 3D target detection result under at least one of the multiple perspectives; and / or the 3D target detection result in the 3D space.

9. A multi-task target detection device, characterized in that, include: The image acquisition module is used to obtain the 2D image features and 3D spatial coordinates of multiple frames of images. Each frame of image includes sub-images from multiple viewpoints. The position alignment module is used to perform position alignment of the 3D coordinates of the multi-frame images in 3D space to obtain the aligned 3D space coordinates of each of the multi-frame images. The fusion module is used to fuse the 2D image features of each of the multiple frames of images with their aligned 3D spatial coordinates to obtain the 2D image features of each of the multiple frames of images containing 3D spatial position information. The embedding feature acquisition module is used to obtain the first embedding features of multiple fixed location points in the BEV space and the second embedding features of multiple pre-trained location points in the 3D space. The pre-training is: training that learns image samples carrying semantic segmentation labels and 3D target labels. The detection module is used to obtain semantic segmentation results and 3D target detection results based on the first embedded feature, the second embedded feature, and the 2D image features containing 3D spatial location information of each of the multiple frames of images.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the multi-task target detection method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The stored computer program causes the processor to execute the multi-task target detection method as described in any one of claims 1-8.

Images