Target detection model training method, target detection method, device and equipment

Abstract

The present disclosure provides a training method for a target detection model, a target detection method, a device and an apparatus, which relate to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, and can be applied to scenes such as autonomous driving. The training method includes: using a coding sub-model to encode a sample image and corresponding three-dimensional position information to obtain a comprehensive coding feature, and the sample image is obtained by shooting the surrounding environment based on multiple perspectives; using a decoding sub-model to decode multiple target query features based on the comprehensive coding feature to obtain multiple decoding features; using a three-dimensional target detection sub-model and a two-dimensional target detection sub-model to perform three-dimensional target detection and two-dimensional target detection on multiple decoding features; training based on the detection results corresponding to the real label and multiple decoding features.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, which can be applied to scenarios such as autonomous driving, and in particular to a target detection model training method, a target detection method, a target detection model training device, a target detection device, an electronic device, a computer-readable storage medium and a computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a target detection model training method, a target detection method, a target detection model training device, a target detection device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, a method for training a target detection model is provided, including: a method for training a target detection model, including: using a coding sub-model in a deep learning model to encode multiple sample images and three-dimensional position information of the multiple sample images to obtain multiple comprehensive coding features corresponding to the multiple sample images, wherein the multiple sample images are obtained by shooting the surrounding environment based on multiple perspectives; using a decoding sub-model in the deep learning model to decode multiple target query features to be trained based on the multiple comprehensive coding features to obtain multiple decoding features corresponding one to one to the multiple target query features; using a three-dimensional target detection sub-model in the deep learning model to perform three-dimensional target detection on the multiple decoding features to obtain three-dimensional target detection results; using a two-dimensional target detection sub-model in the deep learning model to perform two-dimensional target detection on the multiple decoding features to obtain two-dimensional target detection results; and based on the three-dimensional target detection results, the two-dimensional target detection results and the real labels corresponding to the multiple sample images, training the deep learning model and the multiple target query features to obtain the target detection model and the multiple trained target query features.

[0006] According to one aspect of the present disclosure, a target detection method is provided, including: using a coding sub-model in a target detection model to encode multiple images to be detected and three-dimensional position information of the multiple images to be detected, and obtaining multiple comprehensive coding features corresponding to the multiple images to be detected, wherein the multiple images to be detected are obtained by shooting the surrounding environment based on multiple perspectives; using a decoding sub-model in the target detection model, decoding multiple trained target query features based on the multiple comprehensive coding features, and obtaining multiple decoding features corresponding to the multiple target query features one by one; using the three-dimensional target detection sub-model in the target detection model to perform three-dimensional target detection on the multiple decoding features to obtain three-dimensional target detection results; using the two-dimensional target detection sub-model in the target detection model to perform two-dimensional target detection on the multiple decoding features to obtain two-dimensional target detection results.

[0007] According to one aspect of the present disclosure, a training device for a target detection model is provided, comprising: a first encoding unit, configured to encode a plurality of sample images and three-dimensional position information of the plurality of sample images using an encoding sub-model in a deep learning model, and obtain a plurality of comprehensive encoding features corresponding to the plurality of sample images, wherein the plurality of sample images are obtained by photographing a surrounding environment based on multiple perspectives; a first decoding unit, configured to decode a plurality of target query features to be trained based on the plurality of comprehensive encoding features using the decoding sub-model in the deep learning model, and obtain a plurality of decoding features corresponding one to one to the plurality of target query features; a first three-dimensional target prediction unit, configured to perform three-dimensional target detection on the plurality of decoding features using the three-dimensional target detection sub-model in the deep learning model, to obtain a three-dimensional target detection result; a first two-dimensional target prediction unit, configured to perform two-dimensional target detection on the plurality of decoding features using the two-dimensional target detection sub-model in the deep learning model, to obtain a two-dimensional target detection result; and a training unit, configured to train the deep learning model and the plurality of target query features based on the three-dimensional target detection result corresponding to each of the plurality of decoding features, the two-dimensional target detection result, and the target detection true label, to obtain a target detection model and a plurality of trained target query features.

[0008] According to one aspect of the present disclosure, a target detection device is provided, including: a second encoding unit, configured to encode multiple images to be detected and three-dimensional position information of the multiple images to be detected by using an encoding sub-model in a target detection model, and obtain multiple comprehensive encoding features corresponding to the multiple images to be detected, wherein the multiple images to be detected are obtained by shooting the surrounding environment based on multiple perspectives; a second decoding unit, configured to decode multiple trained target query features based on the multiple comprehensive encoding features by using the decoding sub-model in the target detection model, and obtain multiple decoding features corresponding to the multiple target query features one by one; a second three-dimensional target prediction unit, configured to perform three-dimensional target detection on the multiple decoding features by using the three-dimensional target detection sub-model in the target detection model, so as to obtain a three-dimensional target detection result; and a second two-dimensional target prediction unit, configured to perform two-dimensional target detection on the multiple decoding features by using the two-dimensional target detection sub-model in the target detection model, so as to obtain a two-dimensional target detection result.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0012] According to one or more embodiments of the present disclosure, by reusing the decoding features corresponding to the same target query feature to synchronously predict the three-dimensional target detection results and the two-dimensional target detection results, accurate two-dimensional target detection results can be directly obtained, and by simultaneously utilizing the two to train the deep learning model, the three-dimensional target detection capability and the two-dimensional target detection capability of the model can promote each other, thereby improving the prediction ability of the trained model.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 A flowchart of a method for training a target detection model according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 3 A flowchart of encoding a plurality of sample images and three-dimensional position information corresponding to each of the plurality of sample images according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 4 A flowchart of a method for training a target detection model according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 5 A flowchart of training a deep learning model and multiple target query features according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 6A flow chart of a target detection method according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 7 A structural block diagram of a training device for a target detection model according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 8 A structural block diagram of a target detection device according to an exemplary embodiment of the present disclosure is shown; and

[0023] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0027] In the related art, after performing three-dimensional target detection to obtain three-dimensional information of the target, two-dimensional information of the target is usually obtained by projection, but the accuracy of the result obtained by this method is poor.

[0028] To solve the above problems, the present invention discloses to synchronously predict three-dimensional target detection results and two-dimensional target detection results by reusing the decoding features corresponding to the same target query feature, so that accurate two-dimensional target detection results can be directly obtained. By simultaneously using these two to train the deep learning model, the three-dimensional target detection capability and two-dimensional target detection capability of the model can promote each other, thereby improving the prediction ability of the trained model.

[0029] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0030] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0031] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the methods of the present disclosure.

[0032] In some embodiments, server 120 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) network.

[0033] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0034] The user can use client devices 101, 102, 103, 104, 105 and / or 106 to perform human-computer interaction. The client device can provide an interface that enables the user of the client device to interact with the client device. The client device can also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0035] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0036] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0037] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0038] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0039] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0040] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0041] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the database used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0042] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0043] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0044] According to one aspect of the present disclosure, a method for training a target detection model is provided. Figure 2 As shown, the training method of the target detection model includes: step S201, using the encoding sub-model in the deep learning model to encode multiple sample images and the three-dimensional position information of the multiple sample images, and obtain multiple comprehensive encoding features corresponding to the multiple sample images, and the multiple sample images are obtained by shooting the surrounding environment based on multiple perspectives; step S202, using the decoding sub-model in the deep learning model, based on the multiple comprehensive encoding features, decode the multiple target query features to be trained, and obtain multiple decoding features corresponding to the multiple target query features one by one; step S203, using the three-dimensional target detection sub-model in the deep learning model to perform three-dimensional target detection on the multiple decoding features; step S204, using the two-dimensional target detection sub-model in the deep learning model to perform two-dimensional target detection on the multiple decoding features to obtain two-dimensional target detection results; and step S205, based on the three-dimensional target detection results, the two-dimensional target detection results and the real labels corresponding to the multiple sample images, train the deep learning model and the multiple target query features to obtain the target detection model and the multiple trained target query features.

[0045] Therefore, by reusing the decoding features corresponding to the same target query feature to synchronously predict the three-dimensional target detection results and the two-dimensional target detection results, accurate two-dimensional target detection results can be directly obtained. By simultaneously using these two to train the deep learning model, the model's three-dimensional target detection capabilities and two-dimensional target detection capabilities can promote each other, thereby improving the prediction ability of the trained model.

[0046] In some embodiments, the multiple sample images can be obtained by taking pictures of the surrounding environment based on multiple perspectives by a camera or other information collection device in an autonomous driving vehicle or an information collection vehicle. In an exemplary embodiment, the multiple perspectives may include six perspectives: front, left front, right front, rear, left rear, and right rear. In some embodiments, the deep learning model and the trained target detection model can be used to detect vehicles in the surrounding environment.

[0047] In some embodiments, the three-dimensional position information may be, for example, the three-dimensional position information of the content captured by the corresponding sample image. Such information may be obtained by converting the camera space into a three-dimensional world space. Different sample images correspond to different three-dimensional world spaces, so an absolute three-dimensional world space may be established, and a mapping of the three-dimensional world space of each sample image to the absolute three-dimensional world space may be established based on the relationship between the acquisition device of each sample image and the absolute three-dimensional world space.

[0048] According to some embodiments, the three-dimensional position information of multiple sample images (for example, the three-dimensional position information corresponding to each sample image) may include the three-dimensional world coordinates of multiple points on the rays corresponding to the pixels in the multiple sample images, and the three-dimensional position information may be determined based on the camera internal and external parameters corresponding to the viewing angle of the sample image where the point is located. Thus, an absolute three-dimensional world space coordinate system may be established, and the points on the camera cone space corresponding to each sample image may be mapped into the three-dimensional world space and represented by corresponding three-dimensional world coordinates.

[0049] In some embodiments, the camera cone space can be transformed into a three-dimensional world space corresponding to the camera based on the camera's intrinsic parameters, and then the three-dimensional world space corresponding to the camera can be transformed into an absolute three-dimensional world space based on the camera's extrinsic parameters. In some embodiments, the absolute three-dimensional world space can also be normalized.

[0050] According to some embodiments, the encoding sub-model may include an image feature extraction backbone network and a three-dimensional position embedding sub-model. Figure 3 As shown, step S201 in the training method of the target detection model described above, using the encoding sub-model in the deep learning model to encode multiple sample images and the three-dimensional position information of multiple sample images to obtain multiple comprehensive coding features corresponding to the multiple sample images may include: step S301, using the image feature extraction backbone network to extract features from multiple sample images to obtain multiple sample image features corresponding to the multiple sample images; step S302, using the three-dimensional position embedding sub-model to embed features of the three-dimensional position information of multiple sample images to obtain multiple three-dimensional position embedded features corresponding to the multiple sample images; and step S303, feature fusion of multiple sample image features and multiple three-dimensional position embedded features to obtain multiple comprehensive coding features.

[0051] Therefore, by extracting the sample image features of the sample image and embedding the three-dimensional position information into the corresponding three-dimensional position embedding features, and then fusing the sample image features with the three-dimensional position embedding features, a comprehensive coding feature containing visual information and three-dimensional position information can be obtained, so that accurate three-dimensional target detection results and two-dimensional target detection results can be obtained based on the feature.

[0052] In some embodiments, in step S301, an existing image feature extraction backbone network (eg, ResNet50) may be used to extract image features, or a self-designed network may be used.

[0053] In some embodiments, in step S302, a self-designed network may be used for feature embedding. Feature embedding means converting explicit information (e.g., three-dimensional world coordinates) into implicit information (eigenvalues ​​or eigenvectors), thereby improving the accuracy and efficiency of subsequent model processing.

[0054] In some embodiments, the multiple sample image features and the corresponding multiple three-dimensional position embedding features may have the same size. In step S303, the multiple sample image features and the multiple three-dimensional position embedding features may be fused by direct addition, weighted summation, concatenation, processing using a neural network, or a combination of multiple of the above methods to obtain multiple comprehensive coding features.

[0055] In some embodiments, other methods may also be used to encode the multiple sample images and the three-dimensional position information corresponding to the multiple sample images, which is not limited here.

[0056] The multiple object query features to be trained may be learnable features obtained by random initialization, and each query Query represents a feature of a potential object. In an exemplary embodiment, the number of the object query features may be 900.

[0057] In step S202, multiple target query features and multiple comprehensive encoding features may be input into the decoding sub-model to obtain a decoding result for each target query feature, i.e., a decoding feature. The decoding sub-model may adopt a Transformer structure, take multiple target query features as query features (Query), take multiple comprehensive encoding features as key features (Key) and value features (Value), and use an attention mechanism to process the target query features and the comprehensive encoding features to obtain multiple decoding features corresponding to the multiple target query features one by one.

[0058] In some embodiments, the deep learning model may include multiple networks for predicting results. In step S203, each decoded feature may be input into a three-dimensional target detection submodel to obtain a three-dimensional target detection result for the decoded feature. As described above, each target query feature (or the decoded feature obtained after decoding it) corresponds to a potential target, and it can be determined whether the corresponding predicted target is detected based on the target query feature (or the corresponding decoded feature) according to the three-dimensional target detection result. In some embodiments, the three-dimensional target detection result may include a three-dimensional classification result and a three-dimensional regression result. The three-dimensional classification result indicates whether the target query feature (or the corresponding decoded feature) detects the corresponding predicted target, and the type of the detected predicted target, so the three-dimensional target detection result can also indicate that the target query feature (or the corresponding decoded feature) does not detect the corresponding predicted target. In some embodiments, "undetected target" can be used as a special target type. The three-dimensional regression result indicates three-dimensional information such as the three-dimensional position and three-dimensional shape of the detected predicted target.

[0059] According to some embodiments, the three-dimensional object detection result may indicate a predicted three-dimensional detection frame based on three-dimensional world coordinates of at least one predicted target detected in multiple sample images based on multiple decoded features. The predicted three-dimensional detection frame may be, for example, a three-dimensional detection frame surrounding the predicted target in the absolute three-dimensional world space described above, and may specifically include the three-dimensional coordinates of the center point of the three-dimensional detection frame (in the absolute three-dimensional world space), length, width, height, orientation angle, and other information.

[0060] In some embodiments, in step S204, each decoded feature can be input into a two-dimensional target detection sub-model to obtain a two-dimensional target detection result for the decoded feature. The two-dimensional target detection result may include a two-dimensional classification result and a two-dimensional regression result. The two-dimensional classification result indicates in which sample images of a plurality of sample images taken from a plurality of perspectives the predicted target detected by the decoded feature is located, i.e., the sample images of the first part. Training the model using the result can be used to improve the model's ability to judge the spatial position of the target. The two-dimensional regression result indicates two-dimensional information such as the two-dimensional position and two-dimensional shape of the detected predicted target in each sample image of the first part of the sample images.

[0061] According to some embodiments, the two-dimensional target detection result may indicate a sample image of the first part including at least one predicted target among a plurality of sample images, and indicate a predicted two-dimensional detection frame of the at least one predicted target in the corresponding sample image of the first part. The predicted two-dimensional detection frame may be a two-dimensional detection frame surrounding the predicted target in the sample image of the first part, and may specifically include the two-dimensional coordinates of the center point of the two-dimensional detection frame (in the corresponding sample image), width, height and other information.

[0062] Therefore, by using the classification task of two-dimensional target detection (i.e., determining in which perspective the predicted target appears in the sample image), the model's ability to predict the spatial position of the target can be further improved.

[0063] According to some embodiments, Figure 4 Another embodiment of a method for training a target detection model is shown. It is understandable that Figure 4 The operations of steps S401 to S404 and S406 are respectively Figure 2 The operations of step S201 to step S205 in are similar and will not be described in detail here. Figure 4 As shown, the training method may also include: step S405, using the attribute prediction sub-model in the deep learning model to perform attribute prediction on multiple decoding features to obtain attribute prediction results, and the attribute prediction results indicate the predicted attributes of the prediction targets detected in multiple sample images based on the multiple decoding features.

[0064] Therefore, by predicting the attributes of the predicted target, the attribute information of each target in the sample image can be directly and conveniently obtained based on the decoded features. In addition, this attribute prediction framework has good scalability and can give multiple different attributes of the target at the same time.

[0065] In some embodiments, the attribute sub-model may perform attribute prediction on the decoded features of the predicted target to obtain the predicted attribute of the predicted target.

[0066] According to some embodiments, the deep learning model and the target detection model may be used to detect vehicles in multiple sample images. The predicted attributes may include the prediction results of the vehicle model, color, taillight state, occlusion state, and truncation state of the corresponding predicted target. It is understandable that the attribute prediction sub-model may also be used to predict more abundant attributes, which is not limited here.

[0067] According to some embodiments, the real labels corresponding to multiple sample images include the real category of each real target in at least one real target in the surrounding environment, a real three-dimensional detection box, a sample image of the second part of the multiple sample images including the real target, and a real two-dimensional detection box of the real target in each sample image of the sample images of the second part.

[0068] Therefore, by obtaining the above-mentioned real labels, it is possible to combine them with the three-dimensional target detection results and the two-dimensional target detection results to obtain the three-dimensional information loss value and the two-dimensional information loss value respectively, and then use these two parts of the loss values ​​to train the deep learning model, so that the three-dimensional target detection ability and the two-dimensional target detection ability of the model can promote each other, thereby improving the prediction ability of the trained model.

[0069] According to some embodiments, Figure 5 As shown, step S205 in the training method of the target detection model described above, training the deep learning model and multiple target query features based on the three-dimensional target detection results, the two-dimensional target detection results and the real labels corresponding to the multiple sample images to obtain the target detection model and the multiple trained target query features may include: step S501, matching at least one predicted target with at least one real target; step S502, determining the three-dimensional regression loss based on the real three-dimensional detection frame of the at least one real target and the predicted three-dimensional detection frame of the predicted target matching the at least one real target; step S503, determining the two-dimensional classification loss based on the sample image of the second part corresponding to the at least one real target and the sample image of the first part corresponding to the predicted target matching the at least one real target; step S504, determining the two-dimensional regression loss based on the real two-dimensional detection frame of the at least one real target and the predicted two-dimensional detection frame of the predicted target matching the at least one real target; and step S505, training the deep learning model and multiple target query features based on the three-dimensional classification loss, the three-dimensional regression loss, the two-dimensional classification loss and the two-dimensional regression loss.

[0070] Therefore, by calculating the three-dimensional classification loss, three-dimensional regression loss, two-dimensional classification loss and two-dimensional regression loss respectively, and training the deep learning model and target query features based on these loss values, the training effect is improved, so that the trained model has better prediction ability.

[0071] In some embodiments, a three-dimensional classification loss may also be determined based on a true category of at least one true target and a predicted category of a predicted target matching the at least one true target, and parameters of a deep learning model and target query features may be adjusted based on the three-dimensional classification loss.

[0072] In some embodiments, in step S501, Hungarian matching can be performed on the real target and the predicted target based on the position of the target, the intersection-over-union ratio of the corresponding detection frame, the type, etc. It is understandable that the real target and the predicted target can also be matched in other ways, which are not limited here.

[0073] In some embodiments, the three-dimensional classification loss and the two-dimensional classification loss may adopt the cross entropy loss, and the three-dimensional regression loss and the two-dimensional regression loss may adopt the L2 loss. It is understandable that the above losses may also be obtained by other means, which are not limited here.

[0074] In some embodiments, the target detection real label may also include a real label corresponding to an attribute, i.e., a real attribute. Step S205 in the target detection model training method described above, training a deep learning model and multiple target query features may also include: determining an attribute loss based on the real attribute of at least one real target and the predicted attribute of the predicted target matched by at least one real target; training a deep learning model and multiple target query features based on a three-dimensional classification loss, a three-dimensional regression loss, a two-dimensional classification loss, a two-dimensional regression loss, and an attribute loss.

[0075] According to another aspect of the present disclosure, a target detection method is provided. Figure 6 As shown, the method includes: step S601, using the encoding sub-model in the target detection model to encode multiple images to be detected and the three-dimensional position information of the multiple images to be detected, and obtain multiple comprehensive encoding features corresponding to the multiple images to be detected, and the multiple images to be detected are obtained by shooting the surrounding environment based on multiple perspectives; step S602, using the decoding sub-model in the target detection model, based on the multiple comprehensive encoding features, decode the trained multiple target query features, and obtain multiple decoding features corresponding to the multiple target query features one by one; step S603, using the three-dimensional target detection sub-model in the target detection model to perform three-dimensional target detection on the multiple decoding features to obtain three-dimensional target detection results; step S604, using the two-dimensional target detection sub-model in the target detection model to perform two-dimensional target detection on the multiple decoding features to obtain two-dimensional target detection results. It can be understood that Figure 6 The operations and effects of steps S601 to S604 in the above description can be referred to. Figure 2 In addition, the target detection method and the plurality of target query features used in the above target detection method may be obtained by, for example, using the training method of the target detection model provided by the present disclosure.

[0076] Therefore, by reusing the decoded features corresponding to the same target query feature to synchronously predict the three-dimensional target detection result and the two-dimensional target detection result, accurate two-dimensional target detection results can be directly obtained.

[0077] According to some embodiments, the three-dimensional position information corresponding to each of the multiple images to be detected may include the three-dimensional world coordinates of multiple points on the ray corresponding to each pixel in the image to be detected, and the three-dimensional position information may be determined based on the internal and external parameters of the camera corresponding to the viewing angle of the image to be detected.

[0078] Thus, an absolute three-dimensional world space coordinate system can be established, and each point on the camera cone space corresponding to the image to be detected can be mapped into the three-dimensional world space and represented by the corresponding three-dimensional world coordinates.

[0079] According to some embodiments, the encoding sub-model may include an image feature extraction backbone network and a three-dimensional position embedding sub-model. Step S601 in the target detection method described above, using the encoding sub-model in the target detection model to encode multiple images to be detected and the three-dimensional position information of multiple images to be detected, and obtaining multiple comprehensive encoding features corresponding to the multiple images to be detected may include: using the image feature extraction backbone network to extract features from the multiple images to be detected to obtain multiple image features to be detected corresponding to the multiple images to be detected; using the three-dimensional position embedding sub-model to embed features of the three-dimensional position information corresponding to each of the multiple images to be detected to obtain multiple three-dimensional position embedding features corresponding to the multiple images to be detected; and performing feature fusion on the multiple image features to be detected and the multiple three-dimensional position embedding features to obtain multiple comprehensive encoding features.

[0080] Therefore, by extracting the image features to be detected of the image to be detected, embedding the three-dimensional position information into the corresponding three-dimensional position embedding features, and then fusing the image features to be detected with the three-dimensional position embedding features, a comprehensive coding feature containing visual information and three-dimensional position information can be obtained, so that accurate three-dimensional target detection results and two-dimensional target detection results can be obtained based on the feature in the subsequent process.

[0081] According to some embodiments, the three-dimensional target detection result may indicate a three-dimensional detection frame based on three-dimensional world coordinates of at least one target detected in multiple images to be detected based on multiple decoding features, and the two-dimensional target detection result may indicate a first part of the images to be detected including at least one target in the multiple images to be detected, and indicate a two-dimensional detection frame of at least one target in the corresponding first part of the images to be detected. The predicted three-dimensional detection frame may be, for example, a three-dimensional detection frame that surrounds the predicted target in the absolute three-dimensional world space described above, and may specifically include the three-dimensional coordinates of the center point of the three-dimensional detection frame (in the absolute three-dimensional world space), length, width, height, orientation angle, and other information. The predicted two-dimensional detection frame may be a two-dimensional detection frame that surrounds the predicted target in the first part of the images to be detected, and may specifically include the two-dimensional coordinates of the center point of the two-dimensional detection frame (in the corresponding images to be detected), width, height, and other information.

[0082] According to some embodiments, the target detection method may further include: using an attribute sub-model in the target detection model to perform attribute prediction on multiple decoding features to obtain attribute prediction results, wherein the attribute prediction results indicate the attributes of the target detected in multiple images to be detected based on the multiple decoding features.

[0083] Therefore, by predicting the attributes of the target, the attribute information of each target in the image to be detected can be directly and conveniently obtained based on the decoded features. In addition, this attribute prediction framework has good scalability and can give multiple different attributes of the target at the same time.

[0084] According to some embodiments, the target detection model can be used to detect vehicles in multiple images to be detected, and the attributes can include the model, color, taillight status, occlusion status, and truncation status of the corresponding target. It is understandable that the attribute prediction sub-model can also be used to predict more abundant attributes, which is not limited here.

[0085] According to another aspect of the present disclosure, a training device for a target detection model is provided. Figure 7 As shown, the training device 700 includes: a first encoding unit 710, configured to encode multiple sample images and three-dimensional position information of multiple sample images using an encoding sub-model in a deep learning model to obtain multiple comprehensive encoding features corresponding to the multiple sample images, where the multiple sample images are obtained by photographing the surrounding environment based on multiple perspectives; a first decoding unit 720, configured to decode multiple target query features to be trained based on multiple comprehensive encoding features using a decoding sub-model in the deep learning model to obtain multiple decoding features corresponding to the multiple target query features one by one; a first three-dimensional target prediction unit 730, configured to A three-dimensional target detection sub-model in the deep learning model is used to perform three-dimensional target detection on multiple decoded features to obtain three-dimensional target detection results; a first two-dimensional target prediction unit 740 is configured to perform two-dimensional target detection on multiple decoded features using the two-dimensional target detection sub-model in the deep learning model to obtain two-dimensional target detection results; and a training unit 750 is configured to train the deep learning model and multiple target query features based on the three-dimensional target detection result, the two-dimensional target detection result and the true target detection label corresponding to each of the multiple decoded features to obtain a target detection model and multiple trained target query features.

[0086] It is understood that the operations and effects of the units 710 to 750 in the device 700 can be referred to above. Figure 2 The description of steps S201 to S205 in the process are not repeated here.

[0087] According to some embodiments, the three-dimensional position information corresponding to each sample image in a plurality of sample images may include the three-dimensional world coordinates of a plurality of points on a ray corresponding to each pixel in the sample image, and the three-dimensional position information may be determined based on internal and external parameters of a camera corresponding to the viewing angle of the sample image.

[0088] According to some embodiments, the encoding sub-model may include an image feature extraction backbone network and a three-dimensional position embedding sub-model. The first encoding unit in the training device of the target detection model described above may include: a first feature extraction sub-unit, configured to perform feature extraction on multiple sample images using the image feature extraction backbone network to obtain multiple sample image features corresponding to the multiple sample images; a first feature embedding sub-unit, configured to perform feature embedding on the three-dimensional position information of the multiple sample images using the three-dimensional position embedding sub-model to obtain multiple three-dimensional position embedding features corresponding to the multiple sample images; and a first feature fusion sub-unit, configured to perform feature fusion on multiple sample image features and multiple three-dimensional position embedding features to obtain multiple comprehensive encoding features.

[0089] According to some embodiments, a three-dimensional object detection result may indicate a predicted three-dimensional detection box based on three-dimensional world coordinates of at least one predicted target detected in multiple sample images based on multiple decoding features, and a two-dimensional object detection result may indicate a sample image of a first part of the multiple sample images including at least one predicted target, and may indicate a predicted two-dimensional detection box of at least one predicted target in the corresponding first part of the sample image.

[0090] According to some embodiments, the training device described above may also include (not shown in the figure): a first attribute prediction unit, configured to use an attribute prediction sub-model in a deep learning model to perform attribute prediction on multiple decoding features to obtain attribute prediction results, wherein the attribute prediction results indicate predicted attributes of the prediction targets detected in multiple sample images based on the multiple decoding features.

[0091] According to some embodiments, a deep learning model and a target detection model may be used to detect vehicles in multiple sample images, and the predicted attributes may include the prediction results of the vehicle model, color, taillight status, occlusion status, and truncation status of the corresponding predicted target.

[0092] According to some embodiments, the real label may include the real category of each real target in at least one real target in the surrounding environment, a real three-dimensional detection box, a sample image of the second part of a plurality of sample images including the real target, and a real two-dimensional detection box of the real target in each sample image of the sample images of the second part.

[0093] According to some embodiments, the training unit in the training device of the target detection model described above may include: a matching subunit, configured to match at least one predicted target with at least one real target; a three-dimensional regression loss determination subunit, configured to determine the three-dimensional regression loss based on the real three-dimensional detection box of at least one real target and the predicted three-dimensional detection box of the predicted target that matches the at least one real target; a two-dimensional classification loss determination subunit, configured to determine the two-dimensional classification loss based on the sample image of the second part corresponding to the at least one real target and the sample image of the first part corresponding to the predicted target that matches the at least one real target; a two-dimensional regression loss determination subunit, configured to determine the two-dimensional regression loss based on the real two-dimensional detection box of at least one real target and the predicted two-dimensional detection box of the predicted target that matches the at least one real target; and a training subunit, configured to train the deep learning model and multiple target query features based on the three-dimensional regression loss, the two-dimensional classification loss and the two-dimensional regression loss.

[0094] According to another aspect of the present disclosure, a target detection device is provided. Figure 8 As shown, the device 800 includes: a second encoding unit 810, which is configured to encode multiple images to be detected and three-dimensional position information of multiple images to be detected using an encoding sub-model in a target detection model to obtain multiple comprehensive encoding features corresponding to the multiple images to be detected, where the multiple images to be detected are obtained by photographing the surrounding environment based on multiple perspectives; a second decoding unit 820, which is configured to decode multiple trained target query features based on multiple comprehensive encoding features using a decoding sub-model in the target detection model to obtain multiple decoding features corresponding to the multiple target query features one by one; a second three-dimensional target prediction unit 830, which is configured to perform three-dimensional target detection on multiple decoding features using the three-dimensional target detection sub-model in the target detection model to obtain a three-dimensional target detection result; and a second two-dimensional target prediction unit 840, which is configured to perform two-dimensional target detection on multiple decoding features using the two-dimensional target detection sub-model in the target detection model to obtain a two-dimensional target detection result.

[0095] It is understood that the operations and effects of the units 810 to 840 in the device 800 can be referred to above. Figure 6 The description of steps S601 to S604 in the process are not repeated here.

[0096] According to some embodiments, the three-dimensional position information corresponding to each of the multiple images to be detected may include the three-dimensional world coordinates of multiple points on the ray corresponding to each pixel in the image to be detected, and the three-dimensional position information may be determined based on the internal and external parameters of the camera corresponding to the viewing angle of the image to be detected.

[0097] According to some embodiments, the encoding sub-model in the target detection device described above may include an image feature extraction backbone network and a three-dimensional position embedding sub-model. The second encoding unit may include: a second feature extraction sub-unit, configured to use the image feature extraction backbone network to extract features from multiple images to be detected to obtain multiple image features to be detected corresponding to the multiple images to be detected; a second feature embedding sub-unit, configured to use the three-dimensional position embedding sub-model to embed features of the three-dimensional position information of the multiple images to be detected to obtain multiple three-dimensional position embedding features corresponding to the multiple images to be detected; and a second feature fusion sub-unit, configured to fuse the multiple image features to be detected and the multiple three-dimensional position embedding features to obtain multiple comprehensive coding features.

[0098] According to some embodiments, a three-dimensional object detection result may indicate a three-dimensional detection box based on three-dimensional world coordinates of at least one object detected in multiple images to be detected based on multiple decoding features, and a two-dimensional object detection result may indicate an image to be detected including a first part of at least one object in multiple images to be detected, and may indicate a two-dimensional detection box of at least one object in the corresponding first part of the image to be detected.

[0099] According to some embodiments, the target detection device described above may further include (not shown in the figure): a second attribute prediction unit, configured to use the attribute sub-model in the target detection model to perform attribute prediction on multiple decoding features to obtain attribute prediction results, and the attribute prediction results indicate the attributes of the targets detected in multiple images to be detected based on the multiple decoding features.

[0100] According to some embodiments, the target detection model may be used to detect vehicles in a plurality of images to be detected, and the attributes may include the vehicle model, color, taillight status, occlusion status, and truncation status of the corresponding target.

[0101] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0102] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0103] refer to Fig. 9, a block diagram of an electronic device 900 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0104] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0105] Multiple components in the device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the device 900. The input unit 906 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 908 can include but is not limited to a disk, an optical disk. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TM devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0106] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method and / or target detection method of the target detection model. For example, in some embodiments, the training method and / or target detection method of the target detection model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method and / or target detection method of the target detection model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the training method of the target detection model and / or the target detection method in any other suitable manner (for example, by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0109] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0114] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for training a target detection model, comprising: Encoding a plurality of sample images and three-dimensional position information of the plurality of sample images by using a coding sub-model in a deep learning model to obtain a plurality of comprehensive coding features corresponding to the plurality of sample images, wherein the plurality of sample images are obtained by photographing a surrounding environment based on multiple perspectives, and the three-dimensional position information of the plurality of sample images includes three-dimensional world coordinates of a plurality of points on rays corresponding to pixels in the plurality of sample images, and the three-dimensional position information is obtained by converting a camera space into a three-dimensional world space based on internal and external parameters of a camera corresponding to a perspective of the sample image where the point is located; Using the decoding sub-model in the deep learning model, decoding the multiple target query features to be trained based on the multiple comprehensive coding features, and obtaining multiple decoding features corresponding to the multiple target query features one by one; Using a three-dimensional object detection sub-model in the deep learning model to perform three-dimensional object detection on the multiple decoded features to obtain a three-dimensional object detection result; Performing two-dimensional object detection on the multiple decoding features using a two-dimensional object detection sub-model in the deep learning model to obtain a two-dimensional object detection result, wherein the two-dimensional object detection result includes a two-dimensional classification result, and the two-dimensional classification result indicates a sample image in which at least one predicted target detected by the multiple decoding features in the multiple sample images is located; as well as Based on the three-dimensional target detection results, the two-dimensional target detection results and the true labels corresponding to the multiple sample images, the deep learning model and the multiple target query features are trained to obtain a target detection model and multiple trained target query features.

2. The method according to claim 1, wherein: The three-dimensional object detection result indicates a predicted three-dimensional detection box of the at least one predicted target based on the three-dimensional world coordinates, and the at least one predicted target is located in the sample image of the first part of the multiple sample images. The two-dimensional object detection result includes a two-dimensional regression result, and the two-dimensional regression result indicates a predicted two-dimensional detection box of the at least one predicted target in the corresponding first part of the sample image.

3. The method according to claim 2, wherein: The real label includes a real three-dimensional detection box of at least one real target in the surrounding environment, a sample image of a second part including the at least one real target among the multiple sample images, and a real two-dimensional detection box of the at least one real target in the sample image of the second part.

4. The method according to claim 3, wherein: Based on the three-dimensional target detection result, the two-dimensional target detection result, and the true labels corresponding to the multiple sample images, training the deep learning model and the multiple target query features to obtain the target detection model and the multiple trained target query features includes: matching the at least one predicted target with the at least one real target; Determining a three-dimensional regression loss based on a real three-dimensional detection frame of the at least one real target and a predicted three-dimensional detection frame of a predicted target matching the at least one real target; Determining a two-dimensional classification loss based on a second portion of sample images corresponding to the at least one real target and a first portion of sample images corresponding to a predicted target matching the at least one real target; Determining a two-dimensional regression loss based on a true two-dimensional detection box of the at least one true target and a predicted two-dimensional detection box of a predicted target matching the at least one true target; and The deep learning model and the plurality of target query features are trained based on the three-dimensional regression loss, the two-dimensional classification loss, and the two-dimensional regression loss.

5. The method according to claim 1, wherein: The encoding sub-model includes an image feature extraction backbone network and a three-dimensional position embedding sub-model. The encoding sub-model in the deep learning model is used to encode multiple sample images and the three-dimensional position information of the multiple sample images, and the multiple comprehensive encoding features corresponding to the multiple sample images are obtained, including: Using the image feature extraction backbone network to perform feature extraction on the multiple sample images to obtain multiple sample image features corresponding to the multiple sample images; Using the three-dimensional position embedding sub-model to perform feature embedding on the three-dimensional position information of the multiple sample images to obtain a plurality of three-dimensional position embedding features corresponding to the multiple sample images; and The multiple sample image features and the multiple three-dimensional position embedding features are feature fused to obtain the multiple comprehensive coding features.

6. The method according to claim 1, further comprising: The attribute prediction sub-model in the deep learning model is used to perform attribute prediction on the multiple decoding features to obtain attribute prediction results, wherein the attribute prediction results indicate predicted attributes of the prediction targets detected in the multiple sample images based on the multiple decoding features.

7. The method according to claim 6, wherein: The deep learning model and the target detection model are used to detect vehicles in the multiple sample images, and the predicted attributes include prediction results of the vehicle model, color, taillight status, occlusion status, and truncation status of the corresponding predicted target.

8. A target detection method, comprising: Encoding a plurality of images to be detected and three-dimensional position information of the plurality of images to be detected by using a coding sub-model in a target detection model to obtain a plurality of comprehensive coding features corresponding to the plurality of images to be detected, wherein the plurality of images to be detected are obtained by photographing a surrounding environment based on multiple perspectives, and the three-dimensional position information of the plurality of images to be detected includes three-dimensional world coordinates of a plurality of points on rays corresponding to pixels in the plurality of images to be detected, and the three-dimensional position information is obtained by converting a camera space into a three-dimensional world space based on internal and external parameters of a camera corresponding to the perspective of the image to be detected where the point is located; Using a decoding sub-model in the target detection model, decoding a plurality of trained target query features based on the plurality of comprehensive coding features to obtain a plurality of decoding features corresponding one-to-one to the plurality of target query features; Using a three-dimensional target detection sub-model in the target detection model to perform three-dimensional target detection on the plurality of decoded features to obtain a three-dimensional target detection result; as well as The two-dimensional target detection sub-model in the target detection model is used to perform two-dimensional target detection on the multiple decoding features to obtain a two-dimensional target detection result, wherein the two-dimensional target detection result includes a two-dimensional classification result, and the two-dimensional classification result indicates the image to be detected where at least one target detected by the multiple decoding features in the multiple images to be detected is located.

9. The method according to claim 8, wherein: The three-dimensional target detection result indicates a three-dimensional detection box of the at least one target based on three-dimensional world coordinates, and the at least one target is located in the first part of the images to be detected among the multiple images to be detected. The two-dimensional target detection result includes a two-dimensional regression result, and the two-dimensional regression result indicates a two-dimensional detection box of the at least one target in the corresponding first part of the images to be detected.

10. The method according to claim 8, wherein: The encoding sub-model includes an image feature extraction backbone network and a three-dimensional position embedding sub-model. The encoding sub-model in the target detection model is used to encode multiple images to be detected and the three-dimensional position information of the multiple images to be detected, and the multiple comprehensive encoding features corresponding to the multiple images to be detected are obtained, including: Using the image feature extraction backbone network to perform feature extraction on the multiple images to be detected, so as to obtain multiple image features to be detected corresponding to the multiple images to be detected; Using the three-dimensional position embedding sub-model to embed features of the three-dimensional position information of the multiple images to be detected, so as to obtain a plurality of three-dimensional position embedding features corresponding to the multiple images to be detected; and The multiple image features to be detected and the multiple three-dimensional position embedded features are feature fused to obtain the multiple comprehensive coding features.

11. The method according to claim 8, further comprising: The attribute sub-model in the target detection model is used to perform attribute prediction on the multiple decoded features to obtain attribute prediction results, wherein the attribute prediction results indicate attributes of the targets detected in the multiple images to be detected based on the multiple decoded features.

12. The method according to claim 11, wherein: The target detection model is used to detect vehicles in the multiple images to be detected, and the attributes include the vehicle model, color, taillight status, occlusion status and truncation status of the corresponding target.

13. A training device for a target detection model, comprising: A first encoding unit is configured to encode a plurality of sample images and three-dimensional position information of the plurality of sample images by using an encoding sub-model in a deep learning model to obtain a plurality of comprehensive encoding features corresponding to the plurality of sample images, wherein the plurality of sample images are obtained by photographing a surrounding environment based on multiple perspectives, and the three-dimensional position information of the plurality of sample images includes three-dimensional world coordinates of a plurality of points on rays corresponding to pixels in the plurality of sample images, and the three-dimensional position information is obtained by converting a camera space into a three-dimensional world space based on internal and external parameters of a camera corresponding to a perspective of the sample image where the point is located; A first decoding unit is configured to use a decoding sub-model in the deep learning model to decode a plurality of target query features to be trained based on the plurality of comprehensive coding features to obtain a plurality of decoding features corresponding one-to-one to the plurality of target query features; A first three-dimensional object prediction unit is configured to perform three-dimensional object detection on the plurality of decoded features using a three-dimensional object detection sub-model in the deep learning model to obtain a three-dimensional object detection result; a first two-dimensional target prediction unit, configured to perform two-dimensional target detection on the plurality of decoded features using a two-dimensional target detection sub-model in a deep learning model to obtain a two-dimensional target detection result, wherein the two-dimensional target detection result includes a two-dimensional classification result, and the two-dimensional classification result indicates a sample image in which at least one predicted target detected by the plurality of decoded features in the plurality of sample images is respectively located; as well as The training unit is configured to train the deep learning model and the multiple target query features based on the three-dimensional target detection result, the two-dimensional target detection result and the real label of the target detection corresponding to each of the multiple decoding features to obtain a target detection model and multiple trained target query features.

14. The device according to claim 13, wherein: The three-dimensional object detection result indicates a predicted three-dimensional detection box based on the three-dimensional world coordinates of the at least one predicted target, and the at least one predicted target is located in the sample image of the first part of the multiple sample images. The two-dimensional object detection result includes a two-dimensional regression result, and the two-dimensional regression result indicates a predicted two-dimensional detection box of the at least one predicted target in the corresponding first part of the sample image.

15. The device according to claim 14, wherein: The real label includes a real three-dimensional detection box of at least one real target in the surrounding environment, a sample image of a second part including the at least one real target among the multiple sample images, and a real two-dimensional detection box of the at least one real target in the sample image of the second part.

16. The device according to claim 15, wherein: The training unit comprises: a matching subunit, configured to match the at least one predicted target with the at least one real target; a 3D regression loss determination subunit, configured to determine a 3D regression loss based on a real 3D detection frame of the at least one real target and a predicted 3D detection frame of a predicted target matching the at least one real target; A two-dimensional classification loss determination subunit, configured to determine a two-dimensional classification loss based on a second portion of sample images corresponding to the at least one real target and a first portion of sample images corresponding to a predicted target matching the at least one real target; a two-dimensional regression loss determination subunit, configured to determine a two-dimensional regression loss based on a real two-dimensional detection frame of the at least one real target and a predicted two-dimensional detection frame of a predicted target matching the at least one real target; and A training subunit is configured to train the deep learning model and the multiple target query features based on the three-dimensional regression loss, the two-dimensional classification loss, and the two-dimensional regression loss.

17. The device according to claim 13, wherein: The encoding sub-model includes an image feature extraction backbone network and a three-dimensional position embedding sub-model, and the first encoding unit includes: A first feature extraction subunit is configured to perform feature extraction on the multiple sample images using the image feature extraction backbone network to obtain multiple sample image features corresponding to the multiple sample images; a first feature embedding subunit configured to perform feature embedding on the three-dimensional position information of the plurality of sample images using the three-dimensional position embedding submodel to obtain a plurality of three-dimensional position embedding features corresponding to the plurality of sample images; and The first feature fusion subunit is configured to perform feature fusion on the multiple sample image features and the multiple three-dimensional position embedding features to obtain the multiple comprehensive coding features.

18. The apparatus according to claim 13, further comprising: The first attribute prediction unit is configured to use the attribute prediction sub-model in the deep learning model to perform attribute prediction on the multiple decoding features to obtain attribute prediction results, wherein the attribute prediction results indicate predicted attributes of the prediction targets detected in the multiple sample images based on the multiple decoding features.

19. The device according to claim 18, wherein: The deep learning model and the target detection model are used to detect vehicles in the multiple sample images, and the predicted attributes include prediction results of the vehicle model, color, taillight status, occlusion status, and truncation status of the corresponding predicted target.

20. A target detection device, comprising: A second encoding unit is configured to encode a plurality of images to be detected and three-dimensional position information of the plurality of images to be detected using an encoding sub-model in the target detection model to obtain a plurality of comprehensive encoding features corresponding to the plurality of images to be detected, wherein the plurality of images to be detected are obtained by photographing the surrounding environment based on multiple perspectives, and the three-dimensional position information of the plurality of images to be detected includes three-dimensional world coordinates of a plurality of points on rays corresponding to pixels in the plurality of images to be detected, and the three-dimensional position information is obtained by converting a camera space into a three-dimensional world space based on internal and external parameters of a camera corresponding to the perspective of the image to be detected where the point is located; A second decoding unit is configured to decode the trained multiple target query features based on the multiple comprehensive encoding features using a decoding sub-model in the target detection model to obtain multiple decoding features corresponding to the multiple target query features one by one; a second three-dimensional object prediction unit, configured to perform three-dimensional object detection on the plurality of decoded features using a three-dimensional object detection sub-model in the object detection model to obtain a three-dimensional object detection result; as well as The second two-dimensional target prediction unit is configured to perform two-dimensional target detection on the multiple decoded features using a two-dimensional target detection sub-model in the target detection model to obtain a two-dimensional target detection result, wherein the two-dimensional target detection result includes a two-dimensional classification result, and the two-dimensional classification result indicates the image to be detected where at least one target detected by the multiple decoding features in the multiple images to be detected is located.

21. The device according to claim 20, wherein: The three-dimensional target detection result indicates a three-dimensional detection box of the at least one target based on three-dimensional world coordinates, and the at least one target is located in the first part of the images to be detected among the multiple images to be detected. The two-dimensional target detection result includes a two-dimensional regression result, and the two-dimensional regression result indicates a two-dimensional detection box of the at least one target in the corresponding first part of the images to be detected.

22. The device according to claim 20, wherein: The encoding sub-model includes an image feature extraction backbone network and a three-dimensional position embedding sub-model, and the second encoding unit includes: A second feature extraction subunit is configured to perform feature extraction on the multiple images to be detected using the image feature extraction backbone network to obtain multiple image features to be detected corresponding to the multiple images to be detected; a second feature embedding subunit, configured to perform feature embedding on the three-dimensional position information of the plurality of images to be detected using the three-dimensional position embedding submodel, so as to obtain a plurality of three-dimensional position embedding features corresponding to the plurality of images to be detected; and The second feature fusion subunit is configured to perform feature fusion on the multiple image features to be detected and the multiple three-dimensional position embedding features to obtain the multiple comprehensive coding features.

23. The apparatus according to claim 20, further comprising: The second attribute prediction unit is configured to use the attribute sub-model in the target detection model to perform attribute prediction on the multiple decoding features to obtain attribute prediction results, wherein the attribute prediction results indicate the attributes of the targets detected in the multiple images to be detected based on the multiple decoding features.

24. The device according to claim 23, wherein: The target detection model is used to detect vehicles in the multiple images to be detected, and the attributes include the vehicle model, color, taillight status, occlusion status and truncation status of the corresponding target.

25. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images