Training method of three-dimensional object detection model and three-dimensional object detection method

Abstract

The present disclosure provides a training method for a three-dimensional object detection model and a three-dimensional object detection method, which relate to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, and can be applied to scenes such as autonomous driving. The training method for a three-dimensional object detection model includes: obtaining a first sample image, a depth map label of the first sample image, and a first attribute label of a target object in the first sample image; inputting the first sample image into a feature extraction module to obtain a first image feature; inputting the first image feature into an attribute detection module to obtain a first predicted attribute of the target object; based on the first image feature, determining a predicted depth map of the first sample image; based on the first predicted attribute, the first attribute label, the predicted depth map and the depth map label, determining a first loss value of the three-dimensional object detection model; based on the first loss value, adjusting the parameters of the three-dimensional object detection model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, and can be applied to scenarios such as autonomous driving. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Autonomous driving technology involves multiple aspects such as environmental perception, behavioral decision-making, trajectory planning, and motion control. Relying on the coordinated cooperation of sensors, visual computing systems, and positioning systems, vehicles with autonomous driving functions can operate automatically without the need for driver operation or with only a small amount of driver operation.

[0004] Artificial intelligence technology can be used to achieve three-dimensional (3D) object detection. 3D object detection refers to the detection of the three-dimensional information of the object (such as length, width, height, center point coordinates, category, orientation angle, etc.), which is widely used in autonomous driving tasks such as obstacle perception, trajectory planning, and motion control.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] The present disclosure provides a training method and device for a three-dimensional object detection model, a three-dimensional object detection method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0007] According to one aspect of the present disclosure, a training method for a three-dimensional object detection model is provided, the three-dimensional object detection model including a feature extraction module and an attribute detection module, the method including: acquiring a first sample image, a depth map label of the first sample image, and a first attribute label of a target object in the first sample image; inputting the first sample image into the feature extraction module to obtain a first image feature output by the feature extraction module; inputting the first image feature into the attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module; determining a predicted depth map of the first sample image based on the first image feature; determining a first loss value of the three-dimensional object detection model based on the first predicted attribute, the first attribute label, the predicted depth map, and the depth map label; and adjusting parameters of the three-dimensional object detection model based on the first loss value.

[0008] According to one aspect of the present disclosure, a three-dimensional object detection method is provided, comprising: acquiring an image to be detected; and inputting the image into a trained three-dimensional object detection model to obtain three-dimensional properties of a target object in the image output by the three-dimensional object detection model, wherein the three-dimensional object detection model is trained according to the above-mentioned three-dimensional object detection model training method.

[0009] According to one aspect of the present disclosure, a training device for a three-dimensional object detection model is provided, the three-dimensional object detection model includes a feature extraction module and an attribute detection module, the device includes: a first acquisition module, configured to acquire a first sample image, a depth map label of the first sample image and a first attribute label of a target object in the first sample image; a first input module, configured to input the first sample image into the feature extraction module to obtain a first image feature output by the feature extraction module; a second input module, configured to input the first image feature into the attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module; a first determination module, configured to determine a predicted depth map of the first sample image based on the first image feature; a second determination module, configured to determine a first loss value of the three-dimensional object detection model based on the first predicted attribute, the first attribute label, the predicted depth map and the depth map label; and an adjustment module, configured to adjust the parameters of the three-dimensional object detection model based on the first loss value.

[0010] According to one aspect of the present disclosure, a three-dimensional object detection device is provided, comprising: an acquisition module, configured to acquire an image to be detected; and an input module, configured to input the image into a trained three-dimensional object detection model to obtain three-dimensional properties of a target object in the image output by the three-dimensional object detection model, wherein the three-dimensional object detection model is trained according to the training device of the above-mentioned three-dimensional object detection model.

[0011] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.

[0012] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any of the above methods.

[0013] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program can implement any of the above methods when executed by a processor.

[0014] According to one or more embodiments of the present disclosure, the accuracy of three-dimensional object detection can be improved.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 A schematic diagram showing an exemplary system in which various methods described herein may be implemented according to an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of a method for training a three-dimensional object detection model according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram showing the structure and training process of a three-dimensional object detection model according to an embodiment of the present disclosure is shown;

[0020] Figure 4 A flow chart of a three-dimensional object detection method according to an embodiment of the present disclosure is shown;

[0021] Figure 5 A structural block diagram of a training device for a three-dimensional object detection model according to an embodiment of the present disclosure is shown;

[0022] Figure 6 A structural block diagram of a three-dimensional object detection device according to an embodiment of the present disclosure is shown; and

[0023] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations. "Multiple" refers to two or more.

[0027] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Three-dimensional object detection is widely used in the field of autonomous driving. In the related art, an end-to-end monocular three-dimensional detection model can be used to implement three-dimensional object detection. Specifically, in the training stage of the model, sample images annotated with three-dimensional attribute labels of the target object can be used to train the monocular three-dimensional detection model. In the application stage of the model, the monocular image collected by the monocular image acquisition device (i.e., an image acquisition device with only one camera) is input into the trained monocular three-dimensional detection model, and the model will output the three-dimensional detection result of the monocular image, that is, the three-dimensional attributes of the target object (such as vehicles, pedestrians, etc.) in the monocular image, such as length, width, height, coordinates of the center point, category, orientation angle, etc.

[0029] The end-to-end monocular 3D detection model has good detection efficiency and real-time performance. However, since the sample images in the training phase only include two-dimensional image information and lack three-dimensional depth information, it is difficult for the model to learn accurate depth information from the sample images, resulting in insufficient perception of three-dimensional information and low accuracy of the output 3D detection results.

[0030] In response to the above problems, an embodiment of the present disclosure provides a training method for a three-dimensional object detection model. According to an embodiment of the present disclosure, during the training process of the three-dimensional object detection model, the first loss value of the model includes two parts: attribute loss and depth loss. Among them, attribute loss can ensure the accuracy of attribute prediction; depth loss can add a depth supervision signal to the model, so that the first image feature extracted by the model from the first sample image can accurately express the depth information, thereby making the first predicted attribute predicted based on the first image feature more accurate, thereby improving the model's perception of three-dimensional spatial information and achieving high-precision three-dimensional object detection.

[0031] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0032] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0033] In an embodiment of the present disclosure, the motor vehicle 110 may include an electronic device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0034] The server 120 may run one or more services or software applications that enable the execution of the methods of the disclosed embodiments. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0035] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0036] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0037] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0038] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, Wi-Fi), and / or any combination of these and other networks.

[0039] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The databases 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 150 may be of different types. In some embodiments, the databases 150 may be relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0040] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0041] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0042] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth using the IEEE 802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0043] The motor vehicle 110 may further include an inertial navigation module. The inertial navigation module and the satellite positioning module may be combined into a combined positioning system to achieve initial positioning of the motor vehicle 110.

[0044] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle, and accordingly, the motor vehicle 110 is an automatic driving vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of an electronic device on the motor vehicle side (client) according to the present disclosure.

[0045] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0046] According to some embodiments, the motor vehicle 110 or the server 120 may be configured to execute the training method of the three-dimensional object detection model of the embodiment of the present disclosure. The motor vehicle 110 or the server 120 obtains a trained three-dimensional object detection model by executing the training method.

[0047] The trained 3D object detection model can be deployed on a server, such as server 120, or on another server other than server 120 ( Figure 1 During its operation, the motor vehicle 110 can send the environment image collected by the visual camera to the server. The server obtains the 3D object detection result by executing the 3D object detection method of the embodiment of the present disclosure using the trained 3D object detection model, and returns the detection result to the motor vehicle 110.

[0048] The trained 3D object detection model can also be deployed on the vehicle side, for example, on the motor vehicle 110. During its operation, the motor vehicle 110 can use the trained 3D object detection model to execute the 3D object detection method of the embodiment of the present disclosure, and perform 3D object detection on the environment image collected by the visual camera to obtain a 3D object detection result.

[0049] Figure 2FIG. 2 is a flowchart of a method 200 for training a three-dimensional object detection model according to an embodiment of the present disclosure. As described above, the execution subject of the method 200 may be a client device (eg, Figure 1 ), or a server (e.g., a client device 101-106 shown in FIG. Figure 1 , server 120 shown in FIG.

[0050] The three-dimensional object detection model of the embodiment of the present disclosure includes a feature extraction module and an attribute detection model. The output end of the feature extraction module is connected to the input end of the attribute detection module. Figure 2 As shown, method 200 includes steps S210 - S260 .

[0051] In step S210, a first sample image, a depth map label of the first sample image, and a first attribute label of a target object in the first sample image are obtained.

[0052] In step S220, the first sample image is input into a feature extraction module to obtain a first image feature output by the feature extraction module.

[0053] In step S230, the first image feature is input into an attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module.

[0054] In step S240 , a predicted depth map of the first sample image is determined based on the first image feature.

[0055] In step S250, a first loss value of the three-dimensional object detection model is determined based on the first predicted attribute, the first attribute label, the predicted depth map, and the depth map label.

[0056] In step S260, parameters of the three-dimensional object detection model are adjusted based on the first loss value.

[0057] According to an embodiment of the present disclosure, during the training process of the three-dimensional object detection model, the first loss value of the model includes two parts: attribute loss and depth loss. Among them, the attribute loss is determined based on the first predicted attribute and the first attribute label, which can ensure the accuracy of the attribute prediction. The depth loss is determined based on the predicted depth map and the depth map label, which can add a depth supervision signal to the model, so that the first image feature extracted by the model from the first sample image can accurately express the depth information in the three-dimensional space, thereby making the first predicted attribute predicted based on the first image feature more accurate, thereby improving the model's perception of three-dimensional space information and achieving high-precision three-dimensional object detection.

[0058] In step S210, a first sample image, a depth map label of the first sample image, and a first attribute label of a target object in the first sample image are obtained.

[0059] The first sample image may be an image acquired by any image acquisition device. For example, the image acquisition device may be a visual camera deployed on a motor vehicle, a visual camera deployed on the side of a road, etc. The first sample image only includes two-dimensional pixel coordinates and color information, and does not include three-dimensional depth information.

[0060] The first attribute label of the target object in the first sample image includes, for example, a first two-dimensional attribute label and a first three-dimensional attribute label. The first two-dimensional attribute label includes, for example, the category of the target object, a two-dimensional bounding box (Bounding Box, BBox for short), etc. The first three-dimensional attribute label includes, for example, the length, width, height, coordinates of the 3D center point, and orientation angle of the target object.

[0061] The depth map label of the first sample image includes the depth value of each pixel in the first sample image, that is, the distance from the spatial point corresponding to each pixel to the image acquisition device.

[0062] According to some embodiments, the depth map label of the first sample image may be obtained by manual annotation.

[0063] According to some embodiments, the first sample image and its depth map label may be obtained by taking a depth camera (also called a 3D camera). Specifically, the depth camera may take an image including four channels of RGB-D, wherein the RGB channel is the first sample image and the D (Depth) channel is the depth map label corresponding to the first sample image.

[0064] According to some embodiments, the depth map label of the first sample image may be obtained by processing point cloud data collected by a laser radar. In this case, the method 200 may further include steps S270 and S280.

[0065] In step S270, point cloud data corresponding to the first sample image is acquired, wherein the point cloud data includes depth information of the first sample image.

[0066] In step S280 , a depth map label of the first sample image is determined based on the point cloud data.

[0067] According to the above embodiment, the depth map label is calculated from the point cloud data corresponding to the first sample image without manual labeling, thereby improving the efficiency of obtaining sample data and thus improving the training efficiency.

[0068] The point cloud data in step S270 can be acquired by a laser radar. The laser radar can be deployed, for example, on a motor vehicle or on one side of a road. Point cloud data generally includes the three-dimensional coordinates of multiple spatial points and the laser reflection intensity at each spatial point. The perception range of the laser radar can be configured to be greater than or equal to the perception range of the image acquisition device that acquires the first sample image, so that the spatial points corresponding to each pixel in the first sample image are included in the multiple spatial points of the point cloud data, that is, the point cloud data includes the depth information of the first sample image. By processing the point cloud data, a depth map label of the first sample image can be obtained.

[0069] According to some embodiments, the first sample image is acquired by an image acquisition device at a first moment, and the point cloud data is acquired by a laser radar at a second moment. The image acquisition device and the laser radar are configured to perceive the same environment, and the difference between the first moment and the second moment is within a preset range.

[0070] It can be understood that in the above embodiment, the first moment is the timestamp of the first sample image, and the second moment is the timestamp of the point cloud data. The preset range can be, for example, 5ms, 10ms, etc. If the difference between the first moment and the second moment is within the preset range, it can be considered that the first sample image and the point cloud data are the perception results of the same environment at the same moment, and therefore the depth map label of the first sample image can be determined based on the point cloud data.

[0071] According to the above embodiment, the first sample image and point cloud data that are paired in time are obtained by using an image acquisition device and a laser radar whose perception fields are paired, thereby efficiently obtaining a large amount of sample data, thereby improving training efficiency.

[0072] According to some embodiments, the depth map tag includes a depth value tag for each of the plurality of pixels in the first sample image. Determining the depth map tag is equivalent to determining a depth value tag for each pixel in the first sample image. Accordingly, the above step S280 may include steps S281 and S282.

[0073] In step S281, based on the parameters of the image acquisition device that acquires the first sample image, the correspondence between multiple spatial points in the point cloud data and multiple pixels in the first sample image is determined.

[0074] In step S282, for any pixel in the first sample image, a depth value label of the pixel is determined based on the coordinates of the space point corresponding to the pixel.

[0075] According to the above embodiment, the point cloud data is projected into the image plane of the first sample image using the parameters of the image acquisition device, thereby automatically and efficiently obtaining the depth value label of each pixel, thereby obtaining the depth map label.

[0076] The parameters of the image acquisition device include intrinsic parameters and extrinsic parameters. The parameters of the image acquisition device can be obtained, for example, by calibrating the image acquisition device.

[0077] The intrinsic parameters of the image acquisition device include the focal length f of the image acquisition device, the scaling factor α of the pixel plane relative to the imaging plane on the horizontal axis, and the translation amount c of the origin. x , the scaling factor β on the ordinate axis and the translation c of the origin y The intrinsic parameters of the image acquisition device can be expressed as an intrinsic parameter matrix. The intrinsic parameter matrix is ​​used to describe the coordinate transformation relationship from the camera coordinate system to the pixel coordinate system.

[0078] The extrinsic parameters of an image acquisition device are the parameters of the camera in the world coordinate system, such as the camera's position, rotation angle, etc. The extrinsic parameters of an image acquisition device can be expressed as an extrinsic parameter matrix. The extrinsic parameter matrix is ​​used to describe the coordinate transformation relationship from the world coordinate system to the camera coordinate system.

[0079] By calibrating the laser radar, we can get the position and rotation angle of the laser radar in the world coordinate system, and then get the transformation matrix from the coordinates in the world coordinate system to the coordinates in the laser radar coordinate system.

[0080] According to some embodiments, in step S281, the coordinates of the spatial points in the point cloud data are three-dimensional coordinates in the laser radar coordinate system. Based on the transformation matrix of the laser radar coordinate system, the external parameter matrix and the internal parameter matrix of the image acquisition device, the three-dimensional coordinates of the spatial points in the laser radar coordinate system can be converted into three-dimensional coordinates in the world coordinate system, three-dimensional coordinates in the camera coordinate system and two-dimensional coordinates in the pixel coordinate system of the first sample image in sequence, thereby obtaining the correspondence between the spatial points in the point cloud data and the pixels in the first sample image.

[0081] In step S282, for any pixel in the first sample image, the spatial point in the point cloud corresponding to the pixel can be determined based on the corresponding relationship obtained in step S281. The z coordinate value of the spatial point in the three-dimensional coordinates of the camera coordinate system is the depth value label of the pixel.

[0082] There may be multiple first sample images for training the 3D object detection model, and different first sample images may be acquired by different image acquisition devices. For example, multiple (e.g., 4, 6, etc.) visual cameras may be deployed at different positions of a motor vehicle, and the environment image acquired by each visual camera may be used as the first sample image.

[0083] In the case where the first sample image is acquired by multiple image acquisition devices, the parameters of the multiple image acquisition devices are usually different. Due to the different parameters of the multiple image acquisition devices (such as focal length, etc.), the depth values ​​of the pixels in the first sample images acquired by different image acquisition devices have different value ranges, which in turn causes the parameters of the three-dimensional object detection model to be greatly affected by the parameters of the image acquisition devices, resulting in a slow convergence speed and poor generalization.

[0084] According to some embodiments, step S280 further includes step S283. In step S283, the depth value labels obtained in step S282 are normalized based on the parameters of the image acquisition device. That is, the depth value labels are normalized to the range of [0, 1]. This can eliminate the influence of different image acquisition device parameters on image depth estimation, and improve the convergence speed and generalization of the model.

[0085] According to some embodiments, the depth value labels may be normalized according to the following formula:

[0086] dist=D×P×C

[0087]

[0088] Wherein, dist is the normalized depth value label, D is the depth value label obtained by coordinate projection (steps S281 and S282), and C is a very small constant, for example f x is the product of the focal length f of the image acquisition device and the scaling factor α of the pixel plane relative to the imaging plane on the horizontal axis, f y It is the product of the focal length f of the image acquisition device and the scaling factor β of the pixel plane relative to the imaging plane on the vertical axis.

[0089] It should be noted that if the first sample image undergoes a pre-processing step of scaling before being input into the 3D object detection model, and the scaling factor is x, then f in the above formula is x and f y The same scaling factor x needs to be multiplied.

[0090] In step S220, the first sample image is input into a feature extraction module to obtain a first image feature output by the feature extraction module.

[0091] The feature extraction module may be implemented as a neural network, such as a convolutional neural network, etc. According to some embodiments, the feature extraction module may be, for example, a ResNet (residual network), a FPN (Feature Pyramid Network), etc.

[0092] In step S230, the first image feature is input into an attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module.

[0093] The attribute detection module may include, for example, a convolution layer, a fully connected layer, a classification layer, etc. The attribute detection module processes the first image feature and outputs a first predicted attribute of the target object in the first sample image.

[0094] According to some embodiments, the attribute detection module may include a two-dimensional attribute detection module and a three-dimensional attribute detection module. The two-dimensional attribute detection module may be implemented as two convolution layers and multiple classification layers, each classification layer is used to predict a two-dimensional attribute. The two-dimensional attributes include, for example, the category of the target object, BBox, etc. The three-dimensional attribute detection module may be implemented as one convolution layer and multiple classification layers, each classification layer is used to predict a three-dimensional attribute. The three-dimensional attributes include, for example, the length, width, height, coordinates of the 3D center point, and the orientation angle of the target object.

[0095] The input end of the three-dimensional attribute detection module is connected to the output end of the two-dimensional attribute detection module. According to some embodiments, in step S230, the first image feature can be input into the two-dimensional attribute detection module to obtain the first predicted two-dimensional attribute output by the two-dimensional attribute detection module, such as category, BBox, etc. Subsequently, the first predicted two-dimensional attribute is input into the three-dimensional attribute detection module to obtain the first predicted three-dimensional attribute output by the three-dimensional attribute detection module, such as the length, width, height, coordinates of the 3D center point, orientation angle, etc. of the target object. In some embodiments, the first predicted two-dimensional attribute and the image area where the BBox in the first sample image is located can also be input into the three-dimensional attribute detection module together to obtain the first predicted three-dimensional attribute output by the three-dimensional attribute detection module.

[0096] According to the above embodiment, the attribute detection module can predict the two-dimensional attributes and three-dimensional attributes of the target object, so that during the training process of the model, supervision signals can be added to the two-dimensional attributes and the three-dimensional attributes at the same time, further improving the detection accuracy of the model.

[0097] In step S240 , a predicted depth map of the first sample image is determined based on the first image feature.

[0098] According to some embodiments, in step S240, a mapping function from the first image feature to the predicted depth map may be preset, and the predicted depth map of the first sample image may be determined using the mapping function.

[0099] According to some embodiments, in step S240, a predicted depth map can be determined using a depth prediction module including adjustable parameters. The depth prediction module can be implemented as a neural network, for example, a convolutional layer. Specifically, the first image feature is input into the depth prediction module to obtain a depth prediction map output by the depth prediction module. Accordingly, in step S260, the parameters of the three-dimensional object detection model and the depth prediction module can be adjusted simultaneously based on the first loss value. According to the above embodiment, using a depth prediction module including adjustable parameters to obtain a predicted depth map can improve the flexibility and generalization of depth map prediction.

[0100] According to some embodiments, in response to the completion of the training of the 3D object detection model, the depth prediction module is deleted. It is understood that the depth prediction module is only used to add a depth supervision signal to the model during the training phase of the model and does not play a role in the application phase of the model. Therefore, after the model training is completed, the depth prediction module is deleted to save storage space.

[0101] In step S250, a first loss value of the three-dimensional object detection model is determined based on the first predicted attribute, the first attribute label, the predicted depth map, and the depth map label.

[0102] According to some embodiments, when the first predicted attribute includes a first predicted two-dimensional attribute and a first predicted three-dimensional attribute, and the first attribute label includes a first two-dimensional attribute label and a first three-dimensional attribute label, in step S250, a first loss value of the three-dimensional object detection model can be determined based on the first predicted two-dimensional attribute, the first two-dimensional attribute label, the first predicted three-dimensional attribute, the first three-dimensional attribute label, the predicted depth map, and the depth map label.

[0103] According to some embodiments, the two-dimensional attribute loss may be calculated based on the first predicted two-dimensional attribute and the first two-dimensional attribute label. The two-dimensional attribute loss is the sum of the losses of each two-dimensional attribute. For category attributes, a cross entropy loss function may be used to calculate the loss. For the length and width attributes of the BBox, a smooth L1 loss function may be used to calculate the loss.

[0104] According to some embodiments, a three-dimensional attribute loss may be calculated based on the first predicted three-dimensional attribute and the first three-dimensional attribute label. The three-dimensional attribute loss is the sum of the losses of each three-dimensional attribute. For three-dimensional attributes such as length, width, height, center point coordinates, and orientation angle, a smooth L1 loss function may be used to calculate the loss.

[0105] According to some embodiments, the depth loss can be calculated based on the predicted depth map and the depth map label. The depth values ​​in the predicted depth map and the depth map label can be rounded (i.e., uniformly processed), and the rounded integer value is used as the category corresponding to the depth value. For example, each depth value can be converted into an integer in the range of [2,200] (unit: meter), that is, there are 199 categories of depth values. The cross entropy loss function is then used to calculate the depth loss between the predicted depth map and the depth map label.

[0106] The first loss value may be, for example, a weighted sum of the above-mentioned two-dimensional attribute loss, three-dimensional attribute loss, and depth loss.

[0107] In step S260, parameters of the three-dimensional object detection model are adjusted based on the first loss value.

[0108] In step S260, the parameters of the 3D object detection model may be adjusted, for example, by back propagation of the error. In the case where the predicted depth map is determined by a depth prediction module including adjustable parameters, in step S260, the parameters of the 3D object detection model and the depth prediction module may be adjusted simultaneously based on the first loss value.

[0109] It is understood that the above steps S210-S260 can be executed repeatedly for multiple times until a preset termination condition is reached, the 3D object detection model training is completed, and a trained 3D object detection model is obtained. The preset termination condition can be, for example, the first loss value converges, the first loss value is less than a preset threshold, the number of cycles reaches a preset threshold, etc.

[0110] In response to the completion of the 3D object detection model training, the depth prediction module is deleted to save storage space.

[0111] According to some embodiments, the method 200 further includes: before the above step S210, pre-training the 3D object detection model using the second sample image and the second attribute label of the target object in the second sample image. Subsequently, fine-tuning the pre-trained 3D object detection model using the first sample image. That is, the above steps S210-S260 are the fine-tuning process of the 3D object detection model.

[0112] According to the above embodiment, the convergence speed and generalization of the model can be improved by using the pre-training and fine-tuning training mode.

[0113] According to some embodiments, the number of second sample images used in pre-training is greater than the number of first sample images used in fine-tuning. The ratio of the two may be, for example, 6:1, 10:1, etc.

[0114] According to some embodiments, pre-training of a 3D object detection model includes the following steps S201 - S204 .

[0115] In step S201, a second sample image is input into the feature extraction module to obtain a second image feature output by the feature extraction module.

[0116] In step S202, the second image feature is input into the attribute detection module to obtain a second predicted attribute output by the attribute detection module.

[0117] In step S203, a second loss value of the three-dimensional object detection model is determined based on the second predicted attribute and the second attribute label.

[0118] In step S204, parameters of the three-dimensional object detection model are adjusted based on the second loss value.

[0119] Step S201 and step S202 may be implemented with reference to the above steps S220 and S230, and will not be described in detail here.

[0120] In step S203, the second predicted attribute may include a second predicted two-dimensional attribute and a second predicted three-dimensional attribute. The second attribute label may include a second two-dimensional attribute label and a second three-dimensional attribute label. Accordingly, the second loss value may be determined based on the second predicted two-dimensional attribute, the second two-dimensional attribute label, the second predicted three-dimensional attribute and the second three-dimensional attribute label. For category attributes, a cross entropy loss function may be used to calculate the loss. For the length and width attributes of the BBox, a smooth L1 loss function may be used to calculate the loss. For three-dimensional attributes such as length, width, height, center point coordinates, and orientation angle, a smooth L1 loss function may be used to calculate the loss.

[0121] In step S204, the parameters of the three-dimensional object detection model may be adjusted by back propagation of errors.

[0122] It is understood that the above steps S201-S204 can be executed repeatedly until the preset termination condition is reached, the pre-training of the three-dimensional object detection model is completed, and the pre-trained three-dimensional object detection model is obtained. The preset termination condition can be, for example, the second loss value converges, the second loss value is less than a preset threshold, the number of cycles reaches a preset threshold, etc.

[0123] The pre-trained 3D object detection model is further fine-tuned through steps S210-S260. The fine-tuned 3D object detection model is a trained 3D object detection model that can be put into use.

[0124] Figure 3 A schematic diagram showing the structure and training process of a three-dimensional object detection model 300 according to an embodiment of the present disclosure is shown.

[0125] like Figure 3As shown, the three-dimensional object detection model 300 includes a feature extraction module 310 and an attribute detection module 320 .

[0126] The feature extraction module 310 may use a feature graph pyramid structure to extract image features of different scales. Figure 3 As shown, the feature extraction module 310 includes a backbone network 312 and a feature fusion network 314. The backbone network 312 may include, for example, three feature extraction layers 3121, 3122, and 3123, which are respectively used to extract image features of different scales. That is, the image features extracted by the feature extraction layers 3121-3123 have different sizes. The sizes of the image features extracted by the feature extraction layers 3121-3123 may decrease in sequence, that is, the size of the image features extracted by the feature extraction layer 3121 is the largest and contains the most image details; the size of the image features extracted by the feature extraction layer 3121 is the smallest and contains the most high-level semantic information.

[0127] The feature fusion network 314 includes a plurality of fusion nodes 3141, which are used to fuse the image features output by the feature extraction layers 3121-3123 to obtain fused image features of different scales. The size of the fused image features can be, for example, W / 8*H / 8, W / 16*H / 16, or W / 32*H / 32, where W and H are the width and height of the input image, respectively.

[0128] The attribute detection module 320 includes a two-dimensional attribute detection module 322 and a three-dimensional attribute detection module 324. The two-dimensional attribute detection module 322 includes three two-dimensional detection units 3221, 3222, and 3223, which respectively take the image features of different scales output by the feature extraction module 310 as input and output predicted two-dimensional attributes, including the category and BBox of the target object. The two-dimensional detection units 3221-3223 can be implemented as convolutional layers.

[0129] The three-dimensional attribute detection module 324 includes a region of interest (ROI) alignment module 3241 and a three-dimensional detection unit 3242. The region of interest alignment module 3241 is used to align the BBox output by the two-dimensional detection units 3221-3223. The three-dimensional detection unit 3242 takes the aligned BBox as input and outputs predicted three-dimensional attributes, including the length, width, height, center point coordinates, and orientation angle of the target object.

[0130] In the training process of the model 300, first, a large number of sample images 340 (corresponding to the second sample image above) are used to pre-train the model 300. The sample image 340 is input into the model 300 to obtain the predicted two-dimensional attributes and the predicted three-dimensional attributes output by the model 300. Based on the difference between the predicted two-dimensional attributes and the two-dimensional attribute labels of the sample image 340, and the difference between the predicted three-dimensional attributes and the three-dimensional attribute labels of the sample image 340, the loss value of the model 300 in the pre-training process is determined. Based on the loss value, the parameters of the model 300 are adjusted. When the loss value converges, the pre-training process ends and the pre-trained model 300 is obtained.

[0131] Subsequently, a small amount of image-point cloud pairing data 350 is used to fine-tune the pre-trained model 300. Each pairing data 350 includes a sample image 351 and point cloud data 352 in the same environment. Based on the point cloud data 352, the depth map label of the sample image 351 can be calculated. The sample image 351 is input into the model 300 to obtain the image features output by the feature extraction module 310, and further obtain the predicted two-dimensional attributes and predicted three-dimensional attributes output by the model 300. The image features are input into the depth prediction module 330 to obtain the predicted depth map output by the depth prediction module 330. Based on the difference between the predicted two-dimensional attributes and the two-dimensional attribute labels of the sample image 340, and the difference between the predicted three-dimensional attributes and the three-dimensional attribute labels of the sample image 340, the attribute loss Loss1 is calculated. Based on the difference between the predicted depth map and the depth map label, the depth loss Loss2 is calculated. The weighted sum of the attribute loss Loss1 and the depth loss Loss2 is used as the loss value of the model 300 during the fine-tuning process. Based on the loss value, the parameters of the model 300 are adjusted. When the loss value converges, the fine-tuning process ends and a fine-tuned model 300 is obtained.

[0132] The fine-tuned model 300 can be put into practical application. For example, the fine-tuned model 300 can be deployed in an autonomous driving vehicle or a server to perform three-dimensional object detection on the environment image perceived by the autonomous driving vehicle in real time.

[0133] According to an embodiment of the present disclosure, a three-dimensional object detection method is also provided. Figure 4 FIG. 4 is a flowchart of a three-dimensional object detection method 400 according to an embodiment of the present disclosure. The execution subject of the method 400 may be a server (eg Figure 1 ), or a client device (e.g., a server 120 shown in FIG. Figure 1 1-106). Figure 4 As shown, method 400 includes steps S410 and S420.

[0134] In step S410, an image to be detected is acquired. The image to be detected may be, for example, an environment image captured by a visual camera on a motor vehicle.

[0135] In step S420, the image is input into a trained 3D object detection model to obtain the 3D attributes of the target object in the image output by the 3D object detection model, wherein the 3D object detection model is trained according to the 3D object detection model training method of the embodiment of the present disclosure.

[0136] According to the embodiments of the present disclosure, the trained three-dimensional object detection model can achieve accurate three-dimensional object detection, thereby improving the accuracy of the three-dimensional object detection results.

[0137] According to an embodiment of the present disclosure, a training device for a three-dimensional object detection model is also provided. Figure 5 FIG. 5 shows a structural block diagram of a training device 500 for a three-dimensional object detection model according to an embodiment of the present disclosure. Figure 5 As shown, the apparatus 500 includes a first acquisition module 510 , a first input module 520 , a second input module 530 , a first determination module 540 , a second determination module 550 and an adjustment module 560 .

[0138] The first acquisition module 510 is configured to acquire a first sample image, a depth map label of the first sample image, and a first attribute label of a target object in the first sample image.

[0139] The first input module 520 is configured to input the first sample image into the feature extraction module to obtain the first image feature output by the feature extraction module.

[0140] The second input module 530 is configured to input the first image feature into the attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module.

[0141] The first determination module 540 is configured to determine a predicted depth map of the first sample image based on the first image feature.

[0142] The second determination module 550 is configured to determine a first loss value of the three-dimensional object detection model based on the first predicted attribute, the first attribute label, the predicted depth map and the depth map label.

[0143] The adjustment module 560 is configured to adjust parameters of the 3D object detection model based on the first loss value.

[0144] According to an embodiment of the present disclosure, during the training process of the three-dimensional object detection model, the first loss value of the model includes two parts: attribute loss and depth loss. Among them, the attribute loss is determined based on the first predicted attribute and the first attribute label, which can ensure the accuracy of the attribute prediction. The depth loss is determined based on the predicted depth map and the depth map label, which can add a depth supervision signal to the model, so that the first image feature extracted by the model from the first sample image can accurately express the depth information in the three-dimensional space, thereby making the first predicted attribute predicted based on the first image feature more accurate, thereby improving the model's perception of three-dimensional space information and achieving high-precision three-dimensional object detection.

[0145] According to some embodiments, the device 500 also includes: a second acquisition module, configured to acquire point cloud data corresponding to the first sample image, wherein the point cloud data includes depth information of the first sample image; and a third determination module, configured to determine the depth map label of the first sample image based on the point cloud data.

[0146] According to some embodiments, the first sample image is acquired by an image acquisition device at a first moment, and the point cloud data is acquired by a laser radar at a second moment. The image acquisition device and the laser radar are configured to perceive the same environment, and the difference between the first moment and the second moment is within a preset range.

[0147] According to some embodiments, the depth map label includes depth value labels of respective pixels in the first sample image, and wherein the third determination module includes: a first determination unit, configured to determine the correspondence between multiple spatial points in the point cloud data and multiple pixels in the first sample image based on parameters of an image acquisition device that acquires the first sample image; and a second determination unit, configured to determine, for any pixel in the first sample image, the depth value label of the pixel based on the coordinates of the spatial point corresponding to the pixel.

[0148] According to some embodiments, the third determination module further includes: a normalization unit configured to normalize the depth value label based on parameters of the image acquisition device.

[0149] According to some embodiments, the first determination module includes: a first input unit, configured to input the first image feature into a depth prediction module to obtain the predicted depth map output by the depth prediction module, wherein the adjustment module is further configured to: adjust the parameters of the three-dimensional object detection model and the depth prediction module based on the first loss value.

[0150] According to some embodiments, the apparatus 500 further comprises: a deletion module configured to delete the depth prediction module in response to completion of the training of the three-dimensional object detection model.

[0151] According to some embodiments, the device 500 also includes: a pre-training module, configured to pre-train the three-dimensional object detection model using a second sample image and a second attribute label of the target object in the second sample image, wherein the first sample image is used to fine-tune the pre-trained three-dimensional object detection model.

[0152] According to some embodiments, the pre-training module includes: a second input unit, configured to input the second sample image into the feature extraction module to obtain a second image feature output by the feature extraction module; a third input unit, configured to input the second image feature into the attribute detection module to obtain a second predicted attribute output by the attribute detection module; a third determination unit, configured to determine a second loss value of the three-dimensional object detection model based on the second predicted attribute and the second attribute label; and an adjustment unit, configured to adjust the parameters of the three-dimensional object detection model based on the second loss value.

[0153] According to some embodiments, the first attribute label includes a first two-dimensional attribute label and a first three-dimensional attribute label, the attribute detection module includes a two-dimensional attribute detection module and a three-dimensional attribute detection module, the second input module includes: a fourth input unit, configured to input the first image feature into the two-dimensional attribute detection module to obtain a first predicted two-dimensional attribute output by the two-dimensional attribute detection module; a fifth input unit, configured to input the first predicted two-dimensional attribute into the three-dimensional attribute detection module to obtain a first predicted three-dimensional attribute output by the three-dimensional attribute detection module, and the second determination module is further configured to: determine a first loss value of the three-dimensional object detection model based on the first predicted two-dimensional attribute, the first two-dimensional attribute label, the first predicted three-dimensional attribute, the first three-dimensional attribute label, the predicted depth map and the depth map label.

[0154] It should be understood that Figure 5 The modules and units of the apparatus 500 shown in FIG. 5 can be compared with those of the reference Figure 2 The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 500 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0155] According to an embodiment of the present disclosure, a three-dimensional object detection device is also provided. Figure 6 FIG. 6 shows a structural block diagram of a three-dimensional object detection device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes an acquisition module 610 and an input module 620 .

[0156] The acquisition module 610 is configured to acquire an image to be detected.

[0157] The input module 620 is configured to input the image into a trained three-dimensional object detection model to obtain the three-dimensional properties of the target object in the image output by the three-dimensional object detection model, wherein the three-dimensional object detection model is trained according to the training device of the three-dimensional object detection model of the embodiment of the present disclosure.

[0158] According to the embodiments of the present disclosure, the trained three-dimensional object detection model can achieve accurate three-dimensional object detection, thereby improving the accuracy of the three-dimensional object detection results.

[0159] It should be understood that Figure 6 The various modules and units of the apparatus 600 shown in FIG. 6 can be compared with those of the reference Figure 4 The steps in the method 400 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 400 are also applicable to the device 600 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0160] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module.

[0161] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 5 , Figure 6The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of modules 510-560, 610-620 can be implemented together in a system on chip (System on Chip, SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.

[0162] According to one aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned three-dimensional object detection model training method 200 and / or three-dimensional object detection method 400.

[0163] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned three-dimensional object detection model training method 200 and / or three-dimensional object detection method 400.

[0164] According to one aspect of the present disclosure, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the above-mentioned three-dimensional object detection model training method 200 and / or three-dimensional object detection method 400.

[0165] refer to Figure 7, a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0166] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0167] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the device 700, the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0168] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the training method 200 of the three-dimensional object detection model or the three-dimensional object detection method 400. For example, in some embodiments, the method 200 and the method 400 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method 200 and the method 400 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method 200 and / or the method 400 in any other appropriate manner (eg, by means of firmware).

[0169] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0171] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0173] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0174] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0175] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0176] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples. Various elements in the embodiments or examples may be omitted or may be replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A training method for a three-dimensional object detection model, wherein the three-dimensional object detection model comprises a feature extraction module and an attribute detection module, and the method comprises: Pre-training the three-dimensional object detection model using the second sample image and the second attribute label of the target object in the second sample image, wherein the pre-training includes: Inputting the second sample image into the feature extraction module to obtain a second image feature output by the feature extraction module; Inputting the second image feature into the attribute detection module to obtain a second predicted attribute output by the attribute detection module; Determining a second loss value of the 3D object detection model based on the second predicted attribute and the second attribute label; and Based on the second loss value, adjusting parameters of the three-dimensional object detection model; Fine-tuning the pre-trained three-dimensional object detection model using the first sample images, wherein the number of the first sample images used in the fine-tuning is less than the number of the second sample images used in the pre-training, the fine-tuning comprising: Acquire a first sample image, a depth map label of the first sample image, and a first attribute label of a target object in the first sample image; Inputting the first sample image into the feature extraction module to obtain a first image feature output by the feature extraction module; Inputting the first image feature into the attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module; Inputting the first image feature into a depth prediction module to obtain a predicted depth map of the first sample image output by the depth prediction module; determining a first loss value for the 3D object detection model based on the first predicted attribute, the first attribute label, the predicted depth map, and the depth map label; Based on the first loss value, adjusting parameters of the 3D object detection model and the depth prediction module; and In response to completion of the training of the three-dimensional object detection model, deleting the depth prediction module, Among them, the depth prediction module is used to add a depth supervision signal to the three-dimensional object detection model during the fine-tuning stage of the three-dimensional object detection model, and does not play a role in the pre-training stage and application stage of the three-dimensional object detection model.

2. The method according to claim 1, wherein: The depth map label is calculated from the point cloud data corresponding to the first sample image, the first sample image is acquired by an image acquisition device at a first moment, and the point cloud data is acquired by a laser radar at a second moment, the image acquisition device and the laser radar are configured to perceive the same environment, and the difference between the first moment and the second moment is within a preset range.

3. The method according to claim 1 or 2, wherein: The first attribute label includes a first two-dimensional attribute label and a first three-dimensional attribute label, the attribute detection module includes a two-dimensional attribute detection module and a three-dimensional attribute detection module, and the first image feature is input into the attribute detection module to obtain the first predicted attribute of the target object output by the attribute detection module, including: Inputting the first image feature into the two-dimensional attribute detection module to obtain a first predicted two-dimensional attribute output by the two-dimensional attribute detection module; Inputting the first predicted two-dimensional attribute into the three-dimensional attribute detection module to obtain a first predicted three-dimensional attribute output by the three-dimensional attribute detection module, The determining, based on the first predicted attribute, the first attribute label, the predicted depth map, and the depth map label, a first loss value of the 3D object detection model comprises: A first loss value of the three-dimensional object detection model is determined based on the first predicted two-dimensional attribute, the first two-dimensional attribute label, the first predicted three-dimensional attribute, the first three-dimensional attribute label, the predicted depth map, and the depth map label.

4. A three-dimensional object detection method, comprising: Acquire an image to be detected; as well as Inputting the image into a trained 3D object detection model to obtain 3D properties of a target object in the image output by the 3D object detection model, The three-dimensional object detection model is trained according to the method described in any one of claims 1-3.

5. A training device for a three-dimensional object detection model, the three-dimensional object detection model comprising a feature extraction module and an attribute detection module, the device comprising: A pre-training module is configured to pre-train the three-dimensional object detection model using a second sample image and a second attribute label of a target object in the second sample image, the pre-training module comprising: A second input unit is configured to input the second sample image into the feature extraction module to obtain a second image feature output by the feature extraction module; A third input unit is configured to input the second image feature into the attribute detection module to obtain a second predicted attribute output by the attribute detection module; A third determining unit is configured to determine a second loss value of the three-dimensional object detection model based on the second predicted attribute and the second attribute label; and an adjusting unit, configured to adjust parameters of the 3D object detection model based on the second loss value; A fine-tuning module is configured to fine-tune the pre-trained three-dimensional object detection model using the first sample image, wherein the number of the first sample images used in the fine-tuning is less than the number of the second sample images used in the pre-training, and the fine-tuning module includes: a first acquisition module, configured to acquire the first sample image, a depth map label of the first sample image, and a first attribute label of the target object in the first sample image; A first input module is configured to input the first sample image into the feature extraction module to obtain a first image feature output by the feature extraction module; A second input module is configured to input the first image feature into the attribute detection module to obtain a first predicted attribute of the target object output by the attribute detection module; A first determination module is configured to input the first image feature into a depth prediction module to obtain a predicted depth map of the first sample image output by the depth prediction module; a second determination module configured to determine a first loss value of the three-dimensional object detection model based on the first predicted attribute, the first attribute label, the predicted depth map, and the depth map label; an adjustment module configured to adjust parameters of the 3D object detection model and the depth prediction module based on the first loss value; and a deleting module, configured to delete the depth prediction module in response to completion of the training of the three-dimensional object detection model, Among them, the depth prediction module is used to add a depth supervision signal to the three-dimensional object detection model during the fine-tuning stage of the three-dimensional object detection model, and does not play a role in the pre-training stage and application stage of the three-dimensional object detection model.

6. The device according to claim 5, wherein: The depth map label is calculated from the point cloud data corresponding to the first sample image, the first sample image is acquired by an image acquisition device at a first moment, and the point cloud data is acquired by a laser radar at a second moment, the image acquisition device and the laser radar are configured to perceive the same environment, and the difference between the first moment and the second moment is within a preset range.

7. The device according to claim 5 or 6, wherein: The first attribute tag includes a first two-dimensional attribute tag and a first three-dimensional attribute tag, the attribute detection module includes a two-dimensional attribute detection module and a three-dimensional attribute detection module, and the second input module includes: a fourth input unit, configured to input the first image feature into the two-dimensional attribute detection module to obtain a first predicted two-dimensional attribute output by the two-dimensional attribute detection module; a fifth input unit, configured to input the first predicted two-dimensional attribute into the three-dimensional attribute detection module to obtain the first predicted three-dimensional attribute output by the three-dimensional attribute detection module, The second determining module is further configured to: A first loss value of the three-dimensional object detection model is determined based on the first predicted two-dimensional attribute, the first two-dimensional attribute label, the first predicted three-dimensional attribute, the first three-dimensional attribute label, the predicted depth map, and the depth map label.

8. A three-dimensional object detection device, comprising: An acquisition module is configured to acquire an image to be detected; as well as an input module configured to input the image into a trained 3D object detection model to obtain 3D properties of a target object in the image output by the 3D object detection model, Wherein, the three-dimensional object detection model is trained by the device according to any one of claims 5-7.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-4.

11. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

Images