Three-dimensional target detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a three-dimensional target detection method, comprising: generating a prediction box corresponding to a three-dimensional target box of a target vehicle according to the three-dimensional target box formed in a self-coordinate system of the target vehicle and a global feature map of multi-view images of the target vehicle; adjusting parameters of an original detection model for generating the prediction box based on a deviation value between the prediction box and the three-dimensional target box to obtain a pose detection model for generating an obstacle detection box, wherein the obstacle detection box is used to represent an obstacle pose value; and detecting obstacles in a field of view range of the target vehicle by using the pose detection model to determine the obstacle pose value in the field of view range. The present disclosure also provides a three-dimensional target detection device, an electronic device and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and autonomous driving technology, and in particular to a three-dimensional target detection method, apparatus, electronic device and storage medium. Background Technology

[0002] Autonomous driving is a product of the deep integration of the automotive industry with next-generation information technologies such as artificial intelligence, transportation, and urban management. It effectively reduces traffic congestion and accident rates, and helps build a safe and efficient future mobility infrastructure. Object detection, a crucial component of artificial intelligence, provides key technical support for obstacle recognition, obstacle location, and orientation determination in autonomous driving, enabling vehicles to accurately avoid obstacles and plan reasonable routes. DETR (Detection Transformer), an end-to-end object detection network based on Transformer, enables object detection and segmentation, driving the development and application of Transformer in object detection. With the advancement of object detection, DETR-3D was proposed, building upon the 2D image-based DETR framework. This applies DETR's object detection concepts to 3D scenes to meet the obstacle detection needs of autonomous driving in 3D environments.

[0003] However, both DETR and DETR-3D require a training set with ground truth values ​​for network training. In autonomous driving scenarios, however, the collected training data requires extensive manual annotation of ground truth values, incurring significant human and financial costs. Insufficient training data results in poor object detection performance for DETR-3D. In other words, DETR-3D, capable of accurately outputting object detection results, is heavily reliant on a large amount of training data with ground truth values. How to reduce DETR-3D's dependence on ground truth-annotated data while simultaneously training a detection model capable of producing more accurate results is a pressing problem for those skilled in the art. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this disclosure provides a three-dimensional target detection method, apparatus, electronic device, and storage medium.

[0005] One aspect of this disclosure provides a three-dimensional target detection method, comprising: generating a predicted bounding box corresponding to the three-dimensional target bounding box based on a three-dimensional target bounding box formed in the vehicle coordinate system of a target vehicle and a global feature map of a multi-view image of the target vehicle; adjusting the parameters of an original detection model for generating the predicted bounding box based on a deviation value between the predicted bounding box and the three-dimensional target bounding box to obtain a pose detection model for generating obstacle detection boxes, wherein the obstacle detection boxes are used to characterize obstacle pose values; and using the pose detection model to perform obstacle detection within the field of view of the target vehicle to determine obstacle pose values ​​within the field of view.

[0006] In some embodiments, generating a predicted bounding box corresponding to the three-dimensional target bounding box based on a three-dimensional target bounding box formed in the vehicle coordinate system of the target vehicle and a global feature map of multi-view images of the target vehicle includes: mapping the three-dimensional target bounding box onto each of the multi-view images to obtain a set of two-dimensional projection points of the three-dimensional target bounding box on each of the multi-view images; determining a projection area image corresponding to the set of two-dimensional projection points on the multi-view images, and extracting multiple pose features of the projection area image on the global feature map; and generating a predicted bounding box corresponding to the pose features using the original detection model based on the pose features, wherein the predicted bounding box includes at least position prediction information and attitude angle prediction information.

[0007] In some embodiments, mapping the three-dimensional target bounding box onto each of the multi-view images to obtain a set of two-dimensional projection points of the three-dimensional target bounding box on each of the multi-view images includes: using a coordinate transformation formula to convert each three-dimensional coordinate of the three-dimensional target bounding box in the vehicle coordinate system into pixel coordinates of each projection point in the multi-view images; and integrating the pixel coordinates of each projection point to obtain a set of two-dimensional projection points of the three-dimensional target bounding box on each of the multi-view images.

[0008] In some implementations, the coordinate transformation formula is:

[0009]

[0010] Where (u, v) represent the pixel coordinates of the projection point in the pixel coordinate system, dx represents the length occupied by a unit pixel in the horizontal axis x-direction of the vehicle coordinate system in the pixel coordinate system, dy represents the length occupied by a unit pixel in the vertical axis y-direction of the vehicle coordinate system in the pixel coordinate system, u0 represents the horizontal difference between the center pixel coordinates and the origin pixel coordinates of the multi-view image, v0 represents the vertical difference between the center pixel coordinates and the origin pixel coordinates of the multi-view image, f is the focal length of the camera capturing the multi-view image, R is the extrinsic rotation vector of the camera, T is the extrinsic translation vector of the camera, (X s Y s Z s ) represents the three-dimensional coordinates of any point in the three-dimensional target box in the vehicle coordinate system.

[0011] In some embodiments, determining a projection area image corresponding to the set of two-dimensional projection points on the multi-view image and extracting multiple pose features of the projection area image on the global feature map includes: cropping the projection area image on the corresponding multi-view image according to the set of two-dimensional projection points; and mapping the projection area image to the global feature map to extract multiple pose features of the mapped region of the projection area image in the global feature map, and using the multiple pose features of the mapped region as multiple pose features of the projection area image.

[0012] In some embodiments, after extracting multiple pose features of the projection area image corresponding to the global feature map, the method further includes: performing feature processing on the multiple pose features of each projection area image to generate multiple effective features with a target dimension.

[0013] In some implementations, adjusting the parameters of the original detection model that generates the predicted bounding box based on the deviation value between the predicted bounding box and the 3D target bounding box to obtain a pose detection model for generating obstacle detection boxes includes: comparing the predicted bounding box and the 3D target bounding box using a pose loss function to obtain the deviation value between them; and adjusting the parameters of the original detection model according to the deviation value to obtain a pose detection model for generating obstacle detection boxes.

[0014] In some implementations, before generating a predicted bounding box corresponding to the three-dimensional bounding box based on the three-dimensional bounding box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view image of the target vehicle, the method includes: generating the three-dimensional bounding box containing the pose ground truth based on the vehicle coordinate system of the target vehicle.

[0015] Another aspect of this disclosure provides a three-dimensional target detection apparatus, comprising: a prediction module, configured to generate a predicted bounding box corresponding to the three-dimensional target bounding box based on a three-dimensional target bounding box formed in the vehicle coordinate system of a target vehicle and a global feature map of a multi-view image of the target vehicle; a model generation module, configured to adjust the parameters of an original detection model for generating the predicted bounding box based on a deviation value between the predicted bounding box and the three-dimensional target bounding box, to generate a pose detection model for an obstacle detection bounding box, wherein the obstacle detection bounding box is used to characterize the pose value of an obstacle; and a detection module, configured to use the pose detection model to perform obstacle detection within the field of view of the target vehicle to determine the pose value of an obstacle within the field of view. Yet another aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the three-dimensional target detection method described in any of the above embodiments.

[0016] Another aspect of this disclosure provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the three-dimensional target detection method described in any of the above embodiments. Attached Figure Description

[0017] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0018] Figure 1 This is a flowchart of a three-dimensional target detection method according to an exemplary embodiment of the present disclosure;

[0019] Figure 2 A flowchart for generating a pose detection model according to an exemplary embodiment of this disclosure;

[0020] Figure 3 This is a schematic diagram of the projection point transformation process in an exemplary embodiment of this disclosure;

[0021] Figure 4 This is a schematic diagram illustrating pose feature processing in an exemplary embodiment of this disclosure; and

[0022] Figure 5 This is a schematic diagram of a three-dimensional target detection device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0023] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0024] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0027] Figure 1 This is a flowchart illustrating a 3D target detection method according to an exemplary embodiment of this disclosure. The following will be combined with… Figure 1 The three-dimensional target detection method S100 of this disclosure is described.

[0028] Step S102: Based on the 3D target bounding box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view image of the target vehicle, a prediction bounding box corresponding to the 3D target bounding box is generated.

[0029] The vehicle coordinate system is a three-dimensional rectangular coordinate system constructed with the midpoint of the rear of the target vehicle as the origin. The vertical coordinate is the direction of the front of the target vehicle, the horizontal coordinate is the direction perpendicular to the front of the vehicle and pointing to the right side of the driver's cab, and the vertical coordinate is the direction perpendicular to the vehicle body (i.e., the ground) upward.

[0030] A 3D bounding box is a randomly drawn 3D region in the vehicle coordinate system. Although the position of the 3D bounding box is randomly generated, the position and attitude angle of each bounding box in the vehicle coordinate system are fixed and unique after generation. In other words, when the 3D bounding box is generated, it is matched with a ground truth pose value that represents its 3D position and attitude angle information. The direct matching of the ground truth pose value during 3D bounding box generation avoids the need for training samples with manually labeled ground truth values, reducing the generation cost of training the target detection model.

[0031] Multi-view images are raw images captured from various perspectives by image acquisition devices positioned in multiple directions around the target vehicle. These multi-view images can be images of the vehicle's front, rear, left side of the driver's cab, right side of the driver's cab, etc., and can be set according to the vehicle's dimensions and the acquisition capabilities of the image acquisition devices. The specific angles and number are not limited. The image acquisition devices can be cameras. In this embodiment, the number of image acquisition devices is set to six, so each device can capture raw images from six perspectives, resulting in six multi-view images.

[0032] The global feature map contains all pose features and pose information of the multi-view image. Specifically, the two-dimensional feature map of the multi-view image is converted into a one-dimensional vector form, and the position encoding information is also initialized into several one-dimensional vectors. After aligning their dimensions, the two are added together to form the global feature map. The position encoding information is used to represent the pose information of each image region in the multi-view image.

[0033] The predicted bounding box is a 3D bounding box generated by the original detection model based on the pose features of the 3D target bounding box in the global feature map. When the deviation between the predicted bounding box and the 3D target bounding box is less than an error threshold, the predicted bounding box is used as the obstacle detection bounding box. When the deviation is greater than or equal to the error threshold, the difference between the predicted bounding box and the 3D target bounding box is considered large. In this case, the original detection model is adjusted, and new predicted bounding boxes are continuously output until the obstacle detection bounding box is obtained. The predicted bounding box includes a pose prediction value composed of position prediction information and attitude angle prediction information.

[0034] Step S104: Based on the deviation between the predicted bounding box and the 3D target bounding box, the parameters of the original detection model for generating the predicted bounding box are adjusted to obtain a pose detection model for generating obstacle detection boxes, wherein the obstacle detection boxes are used to characterize the obstacle pose values.

[0035] The original detection model is a prediction network to be trained. The pose loss function is used to calculate the loss value between the predicted box and the 3D target box. It is then determined whether the deviation value (loss value) between the two exceeds the error threshold. If the deviation value exceeds the error threshold, the parameters of the original detection model are adjusted according to the deviation value; otherwise, the original detection model is considered to be usable as a pose detection model for generating obstacle detection boxes.

[0036] An obstacle detection box is a predicted box whose deviation from the 3D target box is less than an error threshold. The difference between the obstacle pose value corresponding to the obstacle detection box and the true pose value is small. Therefore, the obstacle pose value corresponding to the obstacle detection box can be used as the true pose value of the obstacle, thereby realizing the identification of obstacles and serving as the basis for obstacle avoidance path planning.

[0037] Step S106: Use the pose detection model to detect obstacles within the field of view of the target vehicle to determine the pose values ​​of obstacles within the field of view.

[0038] A pose detection model is a model used to detect the position and attitude angle information of obstacles within the field of view (especially the direction of movement) of a configured mobile device (i.e., a target vehicle). Furthermore, since obstacles typically have volume, their position information includes at least the length of the obstacle in each direction at its position coordinates.

[0039] Figure 2 A flowchart for generating a pose detection model according to an exemplary embodiment of this disclosure is provided below. Figure 2 The complete generation process of the pose detection model is described in detail.

[0040] First, feature extraction is performed on the multi-view images acquired by the image acquisition device using a backbone network to obtain two-dimensional feature maps for each multi-view image. The backbone network is a pre-trained ResNet (residual neural network)50, and the extracted two-dimensional feature maps are stored in the NuScence database. After pre-training, the weights in the ResNet50 are frozen to preserve the pose features required for classification in downstream tasks. In this disclosure, the image acquisition device can be a camera or a sensor, and six devices are configured to acquire six multi-view images. The specific number of image acquisition devices can be set according to requirements and is not limited here.

[0041] Furthermore, the multi-view 2D feature map is deformed to transform the extracted two-dimensional image features into a one-dimensional vector, aligning it with the dimension of the positional encoding information. Then, the two are added together so that the fused global feature map contains all pose information from the multi-view images. Further, the global feature map is used as input to the encoder of the original detection model to obtain multiple pose features (keys) and the pose values ​​matched by each pose feature. A deep learning model, Transformer, can be used as the original detection model. In some implementations, before step S102, the method includes: generating a three-dimensional target bounding box containing the ground truth pose values ​​based on the target vehicle's vehicle coordinate system.

[0042] Further, step S102 is executed. Step S102 can be specifically executed as follows: mapping the 3D target bounding box onto each multi-view image to obtain a set of 2D projection points of the 3D target bounding box on each multi-view image; determining the projection area image corresponding to the set of 2D projection points on the multi-view image, and extracting multiple pose features of the projection area image from the global feature map corresponding to the multi-view image; and generating a prediction bounding box corresponding to the pose features using the original detection model based on the pose features, wherein the prediction bounding box includes at least position prediction information and pose angle prediction information.

[0043] Specifically, multiple 3D target bounding boxes are randomly generated in the vehicle coordinate system:

[0044] B = {b1,…,b} j ,…,b m}∈b 7 Where B represents the 3D bounding box, m is the number of 3D bounding boxes, j is the index of the 3D bounding box, and b j This represents the j-th 3D bounding box. Each b... j All include position, size, and attitude angle (also known as heading angle), b 7 b j It is a seven-dimensional vector space, where the seven-dimensional vector includes three-dimensional coordinates (X,Y,Z) and three-dimensional coordinate direction dimensions (X,Y,Z). S Y S Z S ) and attitude angle yaw. b j It can be represented as:

[0045] b j =(X,Y,Z,X) S ,Y S Z S ,yaw)

[0046] Where (X,Y,Z) represents the three-dimensional coordinates of the j-th 3D target bounding box in the vehicle coordinate system, X SThe dimension of the j-th 3D target bounding box along the X-axis is represented by the Y-axis. S Z represents the dimension of the j-th 3D target bounding box along the Y-axis. S Let represent the dimension of the j-th 3D target bounding box along the Z-axis, and yaw represent the attitude angle of the j-th 3D target bounding box. Multiple 3D target bounding boxes are randomly generated in the vehicle coordinate system and projected onto a multi-view image. The projection process involves converting the 3D coordinates of each pixel within the 3D target bounding box into the coordinates of a 2D projection point in the pixel coordinate system using coordinate transformation formulas.

[0047] Specifically, the three-dimensional target box is mapped onto each multi-view image to obtain a set of two-dimensional projection points of the three-dimensional target box on each multi-view image. This includes: using coordinate transformation formulas to convert each three-dimensional coordinate of the three-dimensional target box in the vehicle coordinate system into the pixel coordinates of each projection point in the multi-view image; and integrating the pixel coordinates of each projection point to obtain a set of two-dimensional projection points of the three-dimensional target box on each multi-view image.

[0048] The coordinate transformation formula is as follows:

[0049]

[0050] Where (u, v) represent the pixel coordinates of the projection point in the pixel coordinate system, dx represents the length of a unit pixel in the x-axis of the vehicle coordinate system, dy represents the length of a unit pixel in the y-axis of the vehicle coordinate system, u0 represents the x-value difference between the center pixel coordinates and the origin pixel coordinates of the multi-view image, v0 represents the y-value difference between the center pixel coordinates and the origin pixel coordinates of the multi-view image, f is the focal length of the camera capturing the multi-view image, R is the extrinsic rotation vector of the camera, T is the extrinsic translation vector of the camera, (X S ,Y S Z S () represents the three-dimensional coordinates of any point in the three-dimensional target bounding box in the vehicle coordinate system.

[0051] In other words, using the camera's extrinsic rotation vector R (3*3 matrix) and extrinsic translation vector T (1*3 matrix), the 3D coordinates of any point in the 3D target bounding box in the vehicle coordinate system are [X...]. S ,Y S Z S ,1] -1 Convert to camera coordinates, then use the camera's intrinsic parameter matrix. The camera coordinates are converted into image coordinates, and then the image coordinates are converted into pixel coordinates through a pixel transformation matrix. Finally, the three-dimensional coordinates of any pixel in the vehicle coordinate system can be converted into two-dimensional pixel coordinates in the pixel coordinate system.

[0052] Figure 3 This is a schematic diagram of the projection point transformation process in an exemplary embodiment of this disclosure, which can be referred to. Figure 3 .exist Figure 3 (A) shows a schematic diagram of the vehicle coordinate system, with the target vehicle's front end as the vertical axis Y, the direction perpendicular to the vertical axis Y and along the right side of the driver's cab as the horizontal axis X, the direction perpendicular to the vehicle body (i.e., the ground) pointing towards the sky as the vertical axis Z, and the midpoint of the rear of the vehicle as its origin o. Figure 3 (B) illustrates the position of a random 3D target bounding box in the vehicle coordinate system. Refer to the illustration; the 3D target bounding box can be a rectangle with a spatial structure. Figure 3 (C) shows the projection transformation results of mapping the three-dimensional target box to various multi-view images based on each two-dimensional projection point; the projection points of the three-dimensional target box in the multi-view images can be mapped to multiple multi-view images according to their position and attitude angle. The figure shows the case where the four vertex mapping points of the three-dimensional target box are in two multi-view projection images.

[0053] Specifically, the projection area image corresponding to the set of two-dimensional projection points is determined on the multi-view image, and multiple pose features of the projection area image are extracted on the global feature map corresponding to the multi-view image. This includes: cropping the projection area image on the corresponding multi-view image according to the set of two-dimensional projection points; and mapping the projection area image to the global feature map of the multi-view image to extract multiple pose features of the mapped region of the projection area image in the global feature map, and using the multiple pose features of the mapped region as multiple pose features of the projection area image.

[0054] Specifically, after extracting multiple pose features of the projection area image from the global feature map corresponding to the multi-view image, the method further includes: performing feature processing on the multiple pose features of each projection area image to generate multiple effective features with the target dimension.

[0055] The multi-view images were collected by six cameras with a resolution of 1600×900. The camera at the rear of the vehicle has a field of view of 110°, the other cameras have a field of view of 70°, and the front and side cameras have a centerline angle of 55°. This means the cameras cover 360° of the target vehicle, resulting in overlapping areas in the multi-view images. When a projection point is scattered across two multi-view images, the pose features corresponding to the two images are added together to obtain the complete features of the 3D target bounding box. This method of feature overlay from multiple images solves the problem of feature loss caused by the projection point falling outside the projection area. Furthermore, since the projection point is not visible in all multi-view images, invalid pose features from the multi-view images need to be filtered out. A binary value σ (σ = 0 or σ = 1) is defined, and the specific value of σ depends on whether the projection point is projected outside the multi-view image. If the projection point is not projected onto the multi-view image, σ is 0; otherwise, σ is 1. i Let represent the pose features extracted from the i-th multi-view image. Then, the average pose features corresponding to each multi-view image can be expressed as:

[0056]

[0057] Where n represents the number of valid multi-view images (n∈[0,6]).

[0058] Furthermore, global average pooling is performed on the average pose features to remove redundant channel dimensions, so that the dimensions of the acquired pose features match the dimensions of the pose information. The alignment of the two dimensions facilitates querying.

[0059] Figure 4 This is a schematic diagram of pose feature processing in an exemplary embodiment of this disclosure. Figure 4 This illustrates the case where the projection point appears on two multi-view images. The pose features of one multi-view image are extracted as w1, and the pose features of the other multi-view image are extracted as w2. The final pose features obtained for this 3D target bounding box are then... Finally, global average pooling (GAP) is performed on w.

[0060] Based on the foregoing, we can see the feature mapping formed by the multi-view camera parameter matrix through the 3D coordinates of each 3D target box (i.e., the random crop query 3D pixel blocks 1 to m in the figure). Then, by performing global average adaptive pooling on the pose features of each 3D target box, we can obtain the complete pose features corresponding to each 3D target box.

[0061] Further, the process of querying 3D target boxes is performed, which involves initializing m 3D target boxes (i.e., 3D query features 1 to 3D query features M). These 3D target boxes are used as the query value of the original detection model and fused with multiple pose features (Keys) output by the encoder. Specifically, pose features with the same or similar pose features as the 3D target boxes are selected from the global feature map, and these pose features are used as input data to the decoder (i.e., the Transformer decoder) of the original detection model. The decoder of the original detection model then outputs predicted boxes (e.g., detection box 1 for category 1, detection box 2 for category 2, and so on, detection box N for category N).

[0062] In some implementations, step S104 specifically involves: comparing the predicted bounding box and the 3D target bounding box using a pose loss function to obtain the deviation value between them; and adjusting the parameters of the original detection model based on the deviation value to obtain a pose detection model for generating obstacle detection boxes.

[0063] The pose loss function can be expressed as:

[0064]

[0065] in, Indicates the deviation value. This represents the cross-loss value between query values ​​that match a 3D target bounding box and query values ​​that do not match a 3D target bounding box. This represents the loss value between the 3D target bounding box and the predicted bounding box. This indicates the predicted matching status of the 3D target box that matches each query value (matching c). i =1 or not matching c i Binary classification with =0), b represents the center coordinates (x, y, z) of the predicted bounding box. i The true pose of the 3D bounding box corresponding to each query value, where i represents the query value index, and λ... {i} is the weight of the query value, and N is the number of query values, where N > m.

[0066] The 3D target detection method proposed in this disclosure achieves unsupervised pre-task by randomly generating 3D target boxes, avoiding the cost of manually annotating the pose ground truth. In addition, this disclosure ensures the integrity and effectiveness of the pose features of the acquired obstacle detection boxes by averaging and pooling the pose features of various multi-view images.

[0067] Figure 5 This is a schematic diagram of a three-dimensional target detection device according to an exemplary embodiment of the present disclosure.

[0068] like Figure 5As shown, another aspect of this disclosure provides a three-dimensional target detection device 1000, comprising: a prediction module 1002, configured to generate a prediction box corresponding to the three-dimensional target box based on a three-dimensional target box formed in the vehicle coordinate system of the target vehicle and a global feature map of a multi-view image of the target vehicle; a model generation module 1004, configured to adjust the parameters of an original detection model for generating the prediction box based on the deviation value between the prediction box and the three-dimensional target box, so as to generate a pose detection model for an obstacle detection box, wherein the obstacle detection box is used to characterize the pose value of an obstacle; and a detection module 1006, configured to use the pose detection model to detect obstacles within the field of view of the target vehicle, so as to determine the pose value of obstacles within the field of view.

[0069] The various modules of the 3D target detection device 1000 are designed to implement the various steps of the 3D target detection method. Therefore, the principles and execution processes involved can be referred to in the previous text and are not limited here.

[0070] The apparatus may include corresponding modules that perform one or more steps in the flowchart above. Therefore, each or more steps in the flowchart above can be performed by a corresponding module, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0071] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0072] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0073] The three-dimensional target detection device proposed in this disclosure achieves unsupervised pre-task by randomly generating three-dimensional target boxes, avoiding the cost of manually annotating the pose ground truth. In addition, this disclosure ensures the integrity of the acquired obstacle detection boxes and the effectiveness of the pose features by averaging and pooling the pose features of various multi-view images.

[0074] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0075] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0076] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0077] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0078] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0079] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0080] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A three-dimensional target detection method, characterized in that, include: Based on the 3D target bounding box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view image of the target vehicle, a prediction bounding box corresponding to the 3D target bounding box is generated. Based on the deviation between the predicted bounding box and the 3D target bounding box, the parameters of the original detection model that generated the predicted bounding box are adjusted to obtain a pose detection model for generating obstacle detection boxes, wherein the obstacle detection boxes are used to characterize the obstacle pose values. as well as The pose detection model is used to detect obstacles within the field of view of the target vehicle to determine the pose values ​​of obstacles within the field of view. The step of generating a predicted bounding box corresponding to the 3D target bounding box based on the 3D target bounding box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view images of the target vehicle includes: mapping the 3D target bounding box onto each of the multi-view images to obtain a set of two-dimensional projection points of the 3D target bounding box on each of the multi-view images; determining a projection area image corresponding to the set of two-dimensional projection points on the multi-view images, and extracting multiple pose features of the projection area image on the global feature map; and generating a predicted bounding box corresponding to the pose features using the original detection model based on the pose features, wherein the predicted bounding box includes at least position prediction information and attitude angle prediction information. The step of adjusting the parameters of the original detection model that generates the predicted bounding box based on the deviation value between the predicted bounding box and the 3D target bounding box to obtain a pose detection model for generating obstacle detection boxes includes: comparing the predicted bounding box and the 3D target bounding box using a pose loss function to obtain the deviation value between them; and adjusting the parameters of the original detection model according to the deviation value to obtain a pose detection model for generating obstacle detection boxes.

2. The three-dimensional target detection method according to claim 1, characterized in that, The step of mapping the 3D target bounding box onto each of the multi-view images to obtain a set of 2D projection points of the 3D target bounding box on each of the multi-view images includes: Using coordinate transformation formulas, the three-dimensional coordinates of the target bounding box in the vehicle coordinate system are converted into the pixel coordinates of each projection point in the multi-view image; and The pixel coordinates of each projection point are integrated to obtain a set of two-dimensional projection points of the three-dimensional target box on each of the multi-view images.

3. The three-dimensional target detection method according to claim 2, characterized in that, The coordinate transformation formula is: ， Where (u, v) represents the pixel coordinates of the projection point in the pixel coordinate system, dx represents the length occupied by a unit pixel in the horizontal x-axis of the vehicle coordinate system in the pixel coordinate system, and dy represents the length occupied by a unit pixel in the vertical y-axis of the vehicle coordinate system in the pixel coordinate system. This represents the abscissa difference between the center pixel coordinates and the origin pixel coordinates of the multi-view image. The ordinate difference between the center pixel coordinates and the origin pixel coordinates of the multi-view image is given by f, where f is the focal length of the camera that captured the multi-view image, R is the extrinsic rotation vector of the camera, and T is the extrinsic translation vector of the camera. , , ) represents the three-dimensional coordinates of any point in the three-dimensional target box in the vehicle coordinate system.

4. The three-dimensional target detection method according to claim 1, characterized in that, The step of determining the projection area image corresponding to the set of two-dimensional projection points on the multi-view image, and extracting multiple pose features of the projection area image on the global feature map, includes: Based on the set of two-dimensional projection points, crop the projection area image on the corresponding multi-view image; and The projection area image is mapped to the global feature map to extract multiple pose features of the region mapped by the projection area image in the global feature map, and the multiple pose features of the mapped region are used as multiple pose features of the projection area image.

5. The three-dimensional target detection method according to claim 1 or 4, characterized in that, After extracting multiple pose features of the projection region image corresponding to the global feature map, the method further includes: Feature processing is performed on multiple pose features of each of the projection area images to generate multiple effective features with target dimensions.

6. The three-dimensional target detection method according to claim 1, characterized in that, Before generating a predicted bounding box corresponding to the 3D target bounding box based on the 3D target bounding box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view image of the target vehicle, the process includes: Based on the target vehicle's own coordinate system, a 3D target bounding box containing the true pose value is generated.

7. A three-dimensional target detection device, characterized in that, include: The prediction module is used to generate a prediction box corresponding to the three-dimensional target box based on the three-dimensional target box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view image of the target vehicle. The model generation module is used to adjust the parameters of the original detection model that generates the prediction box based on the deviation value between the prediction box and the 3D target box, so as to obtain a pose detection model for generating obstacle detection boxes, wherein the obstacle detection boxes are used to characterize the obstacle pose value. as well as The detection module is used to detect obstacles in the field of vision of the target vehicle using the pose detection model, so as to determine the pose values ​​of obstacles in the field of vision. The step of generating a predicted bounding box corresponding to the 3D target bounding box based on the 3D target bounding box formed in the vehicle coordinate system of the target vehicle and the global feature map of the multi-view images of the target vehicle includes: mapping the 3D target bounding box onto each of the multi-view images to obtain a set of two-dimensional projection points of the 3D target bounding box on each of the multi-view images; determining a projection area image corresponding to the set of two-dimensional projection points on the multi-view images, and extracting multiple pose features of the projection area image on the global feature map; and generating a predicted bounding box corresponding to the pose features using the original detection model based on the pose features, wherein the predicted bounding box includes at least position prediction information and attitude angle prediction information. The step of adjusting the parameters of the original detection model that generates the predicted bounding box based on the deviation value between the predicted bounding box and the 3D target bounding box to obtain a pose detection model for generating obstacle detection boxes includes: comparing the predicted bounding box and the 3D target bounding box using a pose loss function to obtain the deviation value between them; and adjusting the parameters of the original detection model according to the deviation value to obtain a pose detection model for generating obstacle detection boxes.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the three-dimensional target detection method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the three-dimensional target detection method according to any one of claims 1 to 6.

Images