Target detection method, target detection model training method, device and vehicle

Abstract

This disclosure provides an object detection method, an object detection model training method, an apparatus, and a vehicle, relating to the field of artificial intelligence technology, and particularly to the fields of computer vision and deep learning technology. The implementation scheme includes: acquiring feature information of the target object; and based on the feature information, using a classification head to detect the category attribute of the target object, and using a regression head to detect the spatial attribute of the target object, wherein the spatial attribute includes at least one of orientation attribute, position attribute, and size attribute; wherein at least one of the classification head and the regression head performs detection based on a set of difference measurement functions, and the set of difference measurement functions is characterized by the Reni difference metric to indicate at least one difference measurement function.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision and deep learning technology, and specifically to a target detection method, a target detection model training method, a target detection device, a device for training a target detection model, an electronic device, a computer-readable storage medium, a computer program product, and a vehicle. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] In computer vision technology, object detection has a wide range of application scenarios. Image segmentation, object tracking, key point detection, etc. usually rely on object detection. Object detection is generally achieved through a trained object detection model. In the training of the object detection model, the prediction ability of the model is very sensitive to the loss function used in the neural network training process. Under different data and different network structures, choosing different loss functions often leads to changes in the prediction ability of the model.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a target detection method, a target detection model training method, a target detection device, a device for training a target detection model, an electronic device, a computer-readable storage medium, a computer program product, and a vehicle.

[0006] According to one aspect of the present disclosure, a target detection method is provided, comprising: acquiring feature information of a target object; and based on the feature information, detecting category attributes of the target object using a classification head, and detecting spatial attributes of the target object using a regression head, wherein the spatial attributes include at least one of orientation attributes, position attributes, and size attributes; wherein at least one of the classification head and the regression head performs the detection based on a set of difference metric functions, and the set of difference metric functions are characterized by a Reine difference metric to indicate at least one difference metric function.

[0007] According to another aspect of the present disclosure, a method for training a target detection model is provided, comprising: obtaining sample feature information and real feature information of a sample target object, wherein the real feature information includes real category attributes and real spatial attributes, and the real spatial attributes include at least one of the real orientation attributes, real position attributes, and real size attributes of the sample target object; based on the sample feature information, using a classification head to predict the predicted category attributes of the sample target object, and using a regression head to predict the predicted spatial attributes of the sample target object, and the predicted spatial attributes include at least one of the predicted orientation attributes, predicted position attributes, and predicted size attributes of the sample target object predicted by the regression head. At least one of the classification head and the regression head performs the prediction based on a set of difference metric functions, and the set of difference metric functions is characterized by the Reine difference metric to indicate at least one difference metric function; determining whether the predicted category attributes are consistent with the real category attributes and whether the predicted spatial attributes are consistent with the real spatial attributes; and in response to determining that the predicted category attributes are consistent with the real category attributes and the predicted spatial attributes are consistent with the real spatial attributes, obtaining a trained target detection model.

[0008] According to another aspect of the present disclosure, a target detection device is provided, comprising: an acquisition module configured to acquire feature information of a target object; and a processing module configured to detect a category attribute of the target object using a classification head based on the feature information, and to detect a spatial attribute of the target object using a regression head, wherein the spatial attribute includes at least one of an orientation attribute, a position attribute, and a size attribute. At least one of the classification head and the regression head performs detection based on a set of difference metric functions, and the set of difference metric functions is characterized by a Reine difference metric to indicate at least one difference metric function.

[0009] According to another aspect of the present disclosure, a training device for a target detection model is provided, comprising: an information acquisition module, configured to acquire sample feature information and real feature information of a sample target object, wherein the real feature information includes real category attributes and real space attributes, and the real space attributes include at least one of the real orientation attributes, real position attributes, and real size attributes of the sample target object. A sample prediction module, configured to predict the predicted category attributes of the sample target object using a classification head based on the sample feature information, and to predict the predicted space attributes of the sample target object using a regression head, wherein the predicted space attributes include at least one of the predicted orientation attributes, predicted position attributes, and predicted size attributes of the sample target object predicted by the regression head, and at least one of the classification head and the regression head predicts based on a set of difference metric functions, and a set of difference metric functions are characterized by the Reine difference metric to indicate at least one difference metric function. A prediction result determination module, configured to determine whether the predicted category attributes are consistent with the real category attributes and whether the predicted space attributes are consistent with the real space attributes. A model acquisition module, configured to obtain a trained target detection model in response to determining that the predicted category attributes are consistent with the real category attributes and the predicted space attributes are consistent with the real space attributes.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target detection method or training method provided above in the present disclosure.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to enable a computer to execute the target detection method or training method provided above in the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the target detection method or training method provided above in the present disclosure.

[0013] According to another aspect of the present disclosure, a motor vehicle is provided, comprising the electronic device provided above.

[0014] According to one or more embodiments of the present disclosure, a set of difference metric functions are introduced, and these difference metric functions have a unified expression under the characterization of the Raney difference metric, and different difference metric functions can be conveniently switched by setting different parameters.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0018] Figure 2 A schematic diagram of a target detection method according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram showing a feasible solution space of the Raney difference metric in a target detection method according to an embodiment of the present disclosure is shown;

[0020] Figure 4 A schematic diagram of the structure of a target detection model according to an embodiment of the present disclosure is shown;

[0021] Figure 5 A schematic diagram showing a method for training a target detection model according to an embodiment of the present disclosure is shown;

[0022] Figure 6 shows a structural block diagram of a target detection device 600 according to an embodiment of the present disclosure;

[0023] Figure 7 shows a structural block diagram of a target detection device 700 according to an embodiment of the present disclosure;

[0024] Figure 8 shows a structural block diagram of a target detection device 800 according to an embodiment of the present disclosure;

[0025] Fig. 9 A structural block diagram of a training device 900 for a target detection model according to an embodiment of the present disclosure is shown;

[0026] Fig.10 A structural block diagram of a training device 1000 for a target detection model according to an embodiment of the present disclosure is shown;

[0027] Fig.11A structural block diagram of a training device 1100 for a target detection model according to an embodiment of the present disclosure is shown;

[0028] Fig.12 A structural block diagram of a training device 1200 for a target detection model according to an embodiment of the present disclosure is shown;

[0029] Fig.13 A structural block diagram of an exemplary electronic device 1300 that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0032] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0033] Existing target detection algorithms connect different prediction branch heads in series to predict different 3D attributes, and the prediction branch heads include classification heads and regression heads. The classification head is used to predict the probability that the target belongs to a certain category, that is, to predict the category attribute, and the network is usually optimized based on the loss function of the probability distribution metric, such as optimizing the cross entropy loss function (Cross Entropy, CE). The regression head is used to predict the spatial attributes of multiple categories such as the target's orientation angle attribute (direction head), 3D position attribute and size attribute. The network is usually optimized based on the loss function of the distance metric, such as the mean square error (MSE).

[0034] The predictive ability of the model is very sensitive to the loss function used in the neural network training process. Different data and different network structures will often lead to changes in the predictive ability of the model if different loss functions are selected.

[0035] In response to the above technical problems, the present disclosure provides a target detection method and a training method for a target detection model.

[0036] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0037] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0038] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0039] Server 120 may run one or more services or software applications that enable methods for object detection or methods for training object detection models. In some embodiments, server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0040] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0041] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0042] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0043] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0044] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0045] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0046] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0047] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0048] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0049] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0050] Figure 2FIG. 1 is a schematic diagram showing a target detection method according to an embodiment of the present disclosure. Figure 2 As shown, the target detection method includes: step S201 to obtain feature information of the target object; and step S202 to detect the category attribute of the target object using a classification head based on the feature information, and to detect the spatial attribute of the target object using a regression head, wherein the spatial attribute includes at least one of an orientation attribute, a position attribute, and a size attribute. It is understandable that the number of spatial attributes can be less or more, and does not only include orientation attributes, position attributes, and size attributes. The number of regression heads corresponds to the number of spatial attributes, and the content detected by the regression head corresponding to each spatial attribute is the same, including the offset of the target along the XY axis, the height information in the Z axis, the size information of the target (including length, width, and height information), the sine information and cosine information of the target yaw angle, and the speed of the target along the XY axis, etc., which are not specifically limited in the present disclosure.

[0051] At least one of the classification head and the regression head performs detection based on a set of difference metric functions, and the set of difference metric functions is characterized by Renyi Divergence to indicate at least one difference metric function.

[0052] The general expression of the Raney difference metric is as follows:

[0053]

[0054] It introduces an α-order parameter, where p is the probability distribution of the true value and q is the probability distribution of the predicted value. When α takes different values, D α There will be different difference metric functions.

[0055] Exemplarily, when α takes n different values, that is, a set of difference metric functions includes difference metric functions of n different dimensions, each difference metric function corresponds to a feasible solution space, and when the n feasible solution spaces are superimposed together, their intersection can be obtained.

[0056] Figure 3 FIG. 1 is a schematic diagram showing a feasible solution space of the Raney difference metric in a target detection method according to an embodiment of the present disclosure. Taking a three-classification problem as an example, Figure 3As shown in the figure, the horizontal and vertical axes represent the predicted values ​​of the target detection model in a binary classification problem. When the values ​​of α are 1 / 2, 1, 2 and +∞, a set of difference metric functions includes the four functions of Half, Shannon, Collision and Min in the figure, that is, the same set of difference metric functions includes four different dimensional difference metric functions. Each closed area 301, 302, 303, 304 in the picture represents the feasible solution space of the corresponding loss function. When the four feasible solution ranges are superimposed together, their intersection 305 can be obtained. During the training process, each of the difference metric functions in this set of difference metric functions looks at the supervision information of the true value from a different divergent angle, and obtains a suitable observation point for the learning of the target detection prediction value.

[0057] The present invention introduces the same set of difference metric functions, which adopt a unified expression formula to combine different metric functions, so as to obtain a more compact parameter feasible solution space. At the same time, these difference metric functions can be conveniently switched by setting different α-order parameters, thereby improving the generalization ability of the target detection model under different data and different network structures.

[0058] Various aspects of the target detection method according to the embodiments of the present disclosure are further described below.

[0059] According to some embodiments, the classification head performs detection based on a first set of difference metric functions, the regression head performs detection based on a second set of difference metric functions, and at least one difference metric function included in the first set of difference metric functions is the same as at least one difference metric function included in the second set of difference metric functions.

[0060] Usually, in classification problems, a metric function based on probability distribution is used to convert the similarity between samples into the possibility of random events. The more common one is KL loss. In regression problems, a metric function based on distance is used to map the input data to a feature space based on distance measurement. The more common one is L2 loss.

[0061] For infinitesimal quantities x, it can be proved that L2 loss is a tight lower bound of KL loss. The proof process is as follows:

[0062]

[0063] definition:

[0064] f(α)=D α -(α / 2*x 2 +α / 36*(1+5α-5α 2 )*x 4 ),

[0065] And add the definition:

[0066]

[0067] Since both double and quadratic limits exist, the limit signs are interchangeable, so:

[0068]

[0069] In conclusion,

[0070]

[0071] Among them, o(x 2 ) is x 2 Higher-order infinitesimals.

[0072] After normalizing the predicted value and the true value, the regression head is measured using the same difference metric function as the classification head. After unifying the difference metric function, the learning of multiple tasks can be better balanced and the impact of different function value ranges can be avoided.

[0073] According to some embodiments, at least one of the classification head and the regression head is pre-trained based on a label that has undergone a smoothing operation, wherein the label is used to identify the true value of the target object, the true value includes a true category attribute and a real space attribute, and the smoothing operation is used to add uniform noise to at least one of the true category attribute and the real space attribute, and obtain the true value after label smoothing.

[0074] The label data after label smoothing solves the problem of only considering the loss of the correct label position (the position where the one-hot label is 1) in the original identification training sample, while ignoring the loss of the wrong label position (the position where the one-hot label is 0). It avoids the model from being too confident about the correct label, makes the output value difference of the predicted positive and negative samples not too large, avoids overfitting, and thus improves the generalization ability of the model.

[0075] According to some embodiments, the loss function used by the classification head or regression head in pre-training is the sum of the products of each of the at least one difference metric function and the corresponding first weight, and the first weight is used to represent the weight of each of the at least one difference metric function in a group of difference metric functions. That is, for the same group of difference metric functions, when the number of difference metric functions in the same group of difference metric functions is greater than or equal to 2, the weight can be allocated to each difference metric function based on a multi-objective optimization method. Figure 3 To further illustrate the example in, the loss function for a single detection head (which can be a classification head or a regression head) is expressed as:

[0076] L i (θ) = a i1 D 1 / 2(θ)+a i2 D1(θ)+a i3 D2(θ)+a i4 D ∞ (θ)

[0077] Among them, D 1 / 2 (θ) is the Half difference metric function, D1(θ) is the Shannon difference metric function, D2(θ) is the Collision difference metric function, and D ∞ (θ) is the Min difference metric function, a i1 、a i2 、a i3 and a i4 are the first weights of the four difference metric functions respectively.

[0078] According to some embodiments, the loss function of the classification head and the loss function of the regression head have respective second weights, and the second weights are learned in pre-training using a multi-objective balancing strategy based on the number of classification heads and regression heads.

[0079] By introducing a second weight to balance the weight of the loss function between the classification head and the regression head, Figure 3 To further illustrate the example in, the total loss function of the entire target detection model is expressed as:

[0080] L(θ)=b1L1(θ)+b2L2(θ)+b3L3(θ)+b4L4(θ)

[0081] Among them, L1(θ), L2(θ), L3(θ) and L4(θ) respectively correspond to the loss functions of the detection head for detecting the category attribute, orientation attribute, position attribute and size attribute of the target object, and b1, b2, b3 and b4 are the second weights of these four loss functions respectively.

[0082] In pre-training, target detection uses a total loss function, which is the sum of the loss function of the classification head and the loss function of the regression head. Figure 3 To further illustrate the example in, the total loss function of the entire target detection model is expressed as:

[0083] L(θ)=b1L1(θ)+b2L2(θ)+b3L3(θ)+b4L4(θ)

[0084] =a 11 bD α1 (θ)+a 12 bD α2 (θ)+a 13 bD α3 (θ)+a 14 bD α4(θ)+

[0085] a 21 b2D α5 (θ)+a 22 bD α6 (θ)+a 23 b2D α7 (θ)+a 24 b2D α8 (θ)+a 31 3D α9 (θ)+

[0086] a 32 3D α10 (θ)+a 33 3D α11 (θ)+a 34 3D α12 (θ)+a 41 bD α13 (θ)+

[0087] a 42 bD α14 (θ)+a 43 bD α15 (θ)+a 44 bD α16 (θ)

[0088] According to some embodiments, a total loss function is used in the pre-training, and the total loss function is the sum of the products of the at least one difference metric function in the loss function used by the classification head or the regression head in the pre-training and the corresponding third weight, and the third weight is used to represent the weight of each of the at least one difference metric function in the total loss function. That is, the third weight λ is introduced k To balance the weights of the loss functions of different tasks, we can use λ k Understood as the difference measurement function D αk The weight in the total loss function. Therefore, the above formula can also be rewritten as:

[0089] L(θ)=λ1D α1 (θ)+λ2D α2 (θ)+…+λ K D αK (θ)

[0090] In order to better balance the objectives of the classification task and the regression task, the third weight λ k It can exist independently without relying on the product of the first weight and the second weight, and use a weight learning strategy based on Pareto optimality to perform multi-objective balance optimization.

[0091] The multi-task learning process is abstracted into the following inner and outer double-layer optimization problem:

[0092]

[0093] Among them, the third weight λ of the inner layer optimization k Dynamic adjustments are made during the optimization process, and the outer optimization adjusts the model parameters θ according to the weighted sum loss function.

[0094] According to some embodiments, the feature information includes bird's-eye-view BEV feature information, which is feature information obtained based on bird's-eye-view perception (BEV Perception). It is a natural description to unify and represent different perspectives under the bird's-eye view, which can facilitate subsequent planning and control tasks. At the same time, the target object under the bird's-eye view does not have the scale and occlusion problems under the image perspective.

[0095] Wherein, acquiring characteristic information of the target object includes acquiring information data associated with the target object; and generating bird's-eye view BEV characteristic information based on the information data associated with the target object.

[0096] Exemplarily, a bird's-eye view BEV feature conversion algorithm (Lift, Splat, Shoot, LSS) can be used to construct a bird's-eye view BEV spatial feature based on pure visual / image information for exemplary explanation, wherein Lift explicitly estimates the depth distribution of feature points after image plane downsampling of the images captured by each camera, and obtains a view cone containing image features, that is, obtains depth information, and extracts the features of pixels in the 2D image in the 3D space with the vehicle body coordinates as the origin; Splat distributes the view cones of all cameras to the bird's-eye view BEV grid in combination with the camera's internal and external parameters, and performs voxel pooling calculations on multiple view cone points in each grid to form a bird's-eye view BEV feature map.

[0097] According to some embodiments, based on information data associated with a target object, generating a bird's-eye view BEV feature information includes: extracting multiple feature information at different scales from the information data associated with the target object; fusing the multiple feature information at different scales to obtain multi-scale fused feature information; and converting the multi-scale fused feature information to generate a bird's-eye view BEV feature information.

[0098] A multi-scale feature extractor (Feature Pyramid Network, FPN) can be used to extract multiple feature information at different scales from information data associated with the target object and to fuse multiple feature information at different scales. The multi-scale feature extraction based on pure visual / image information is also exemplified. The multi-scale fusion uses the multi-scale (resolution) feature map obtained in the skeleton network as input, and the maximum scale feature map is deconvolved to the target scale and then added to the input feature map at this scale to obtain a fused feature map at this scale. The fused feature map is then deconvolved to obtain the fused feature map of the next scale. By analogy, multiple multi-scale fused feature maps can be obtained.

[0099] For each frame image, N feature maps are input (for example, N=3), and N multi-scale fusion feature maps are output accordingly. The sizes of the N input feature maps are different from each other, and the sizes of the N input feature maps are the same as the sizes of the N output multi-scale fusion feature maps, that is, the sizes of the input and output remain unchanged. In a similar manner to the above, the multi-scale fusion feature map of each image in the N surround camera images is obtained respectively. The value of N can be set according to actual needs, and the present disclosure does not make any special restrictions.

[0100] According to some embodiments, the information data associated with the target object includes at least one of image data of the target object acquired based on at least one first sensor and point cloud data of the target object acquired based on at least one second sensor.

[0101] Exemplarily, the first sensor can be different types of cameras such as visual cameras and infrared cameras, and the cameras can be installed in the front, rear or other positions of the vehicle. The visual camera can capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, image data of target objects such as traffic light indications, intersection conditions, and operating status of other vehicles can be obtained. The infrared camera can capture image data containing the target object under night vision conditions.

[0102] The second sensor may be at least one of an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR), and different sensors may provide different detection accuracy and range. The millimeter wave radar may be installed in front of, behind, or at other locations of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. The laser radar may be installed in front of, behind, or at other locations of the vehicle to detect object edges and shape information for object recognition and tracking. Due to the Doppler effect, the radar device may also measure changes in the speed of the vehicle and moving objects.

[0103] Exemplarily, data from at least the first sensor and data from the second sensor may be fused at the original data layer. Taking a visual camera and a lidar as an example, depth information from the lidar may be used to help the visual camera enhance the characteristics to a bird's-eye view BEV space.

[0104] Exemplarily, at least the data of the first sensor and the data of the second sensor may be converted to generate corresponding bird's-eye view BEV feature information, and then a plurality of different bird's-eye view BEV feature information (such as camera bird's-eye view BEV features and lidar bird's-eye view BEV features) may be fused to form multi-sensor fused bird's-eye view BEV feature information. Currently, autonomous driving vehicles are usually equipped with different types of sensors, and the strengths of each sensor can be used to complement each other. In the bird's-eye view BEV space, feature information of different perspectives, different time series (frames), and different modalities (sensors) can be easily fused to output more accurate bird's-eye view BEV space features, thereby better achieving autonomous driving perception tasks.

[0105] For example, since the LiDAR data only contains short-term effective measurements of the ground plane (usually about 30 meters for a rotating 32-beam LiDAR), many parts of the LiDAR bird's-eye view BEV features are encoded as blank space. Compared with LiDAR, the visible area in the data of the visual camera is usually larger. Therefore, for different bird's-eye view BEV feature information, the fused BEV features can be obtained by parallel strategies (Add), series feature fusion (Concat), or weighted summation by obtaining weighted coefficients.

[0106] Figure 4 FIG. 2 shows a schematic diagram of a target detection model according to an embodiment of the present disclosure. Figure 4 As shown, the target detection method uses the target detection model 400, which includes a skeleton network 410, a multi-scale feature extractor 420, a bird's-eye view BEV feature converters 430 and 431, a bird's-eye view BEV feature fuser 440, a classification head 450 and a regression head 460.

[0107] The feature information of the target object may include pure visual / image information and LiDAR point cloud information. It is understandable that the feature information of the target object may also include the original information obtained by other sensors, or the feature information obtained by each sensor after feature extraction and feature fusion, such as Figure 4 As shown, the laser radar point cloud information can be directly input into the bird's-eye view BEV feature converter 431 for bird's-eye view BEV spatial change, and the laser radar point cloud information can also be input into the bird's-eye view BEV feature converter 431 for bird's-eye view BEV spatial change after feature extraction and fusion.

[0108] The pure visual image information passes through the skeleton network 410, which is responsible for extracting visual features based on the input pure visual / image information, where visual features refer to feature maps of different scales. The skeleton network 410 can have different forms, such as a multi-dimensional hybrid model scaling network (EfficientNet). For each image from a visual camera, the pure visual / image information is input into the skeleton network 410, thereby obtaining multiple feature maps of different scales.

[0109] The multi-scale feature extractor 420 fuses the features extracted by the skeleton network 410 at different scales to make the learned features more robust. For each frame of image, N feature maps are input to the multi-scale feature extractor 420, and the multi-scale feature extractor 420 will output N multi-scale fused feature maps accordingly. The sizes of the N input feature maps are different from each other, and the sizes of the N input feature maps are the same as the sizes of the N output multi-scale fused feature maps, that is, the sizes of the input and output remain unchanged. In a similar manner to the above, the multi-scale fused feature maps of each image in the N surround camera images are obtained respectively.

[0110] The bird's-eye view BEV feature converters 430 and 431 transfer the feature information of different types of target objects into the bird's-eye view BEV space to form different types of bird's-eye view BEV feature maps.

[0111] The bird's-eye view BEV feature fuser 440 fuses different types of bird's-eye view BEV feature maps. Since the lidar data only contains short-term effective measurements of the ground plane (usually around 30 meters for a rotating 32-beam LiDAR), many parts of the lidar bird's-eye view BEV feature are encoded as blank space. Compared with lidar, the visible area in the data of a visual camera is usually larger. Therefore, for different bird's-eye view BEV feature information, the fused BEV feature can be obtained by Add, Concat, or by obtaining a weighted coefficient for weighted summation.

[0112] The classification head 450 and the regression head 460 perform target detection based on the fused bird's-eye view BEV features. The classification head 450 is used to detect the category attributes of the target object, and the regression head 460 is used to detect the spatial attributes of the target object, wherein the spatial attributes include at least one of orientation attributes, position attributes, and size attributes.

[0113] Figure 5 FIG. 1 is a schematic diagram showing a method for training a target detection model according to an embodiment of the present disclosure. Figure 5 As shown in the figure, the training method of the target detection model includes:

[0114] Step S501, obtaining sample feature information and real feature information of the sample target object, wherein the real feature information includes real category attributes and real space attributes, and the real space attributes include at least one of real orientation attributes, real position attributes, and real size attributes of the sample target object.

[0115] It is understandable that the number of real space attributes can be less or more, and does not only include real orientation attributes, real position attributes and real size attributes. The number of real space attributes corresponds to the number of regression heads, and the content detected by the regression head corresponding to each real space attribute is the same, including the offset of the target along the XY axis, the height information in the Z axis direction, the size information of the target (including length, width and height information), the sine information and cosine information of the target yaw angle, and the speed of the target along the XY axis, etc., which are not specifically limited in the present disclosure.

[0116] Step S502, based on the sample feature information, use the classification head to predict the predicted category attribute of the sample target object, and use the regression head to predict the predicted spatial attribute of the sample target object, the predicted spatial attribute includes at least one of the predicted orientation attribute, predicted position attribute and predicted size attribute of the sample target object predicted by the regression head.

[0117] At least one of the classification head and the regression head makes predictions based on a set of difference metric functions, and the set of difference metric functions is characterized by a Reiney difference metric to indicate at least one difference metric function.

[0118] The general expression of the Raney difference metric is as follows:

[0119]

[0120] It introduces an α-order parameter, where p is the probability distribution of the true value and q is the probability distribution of the predicted value. When α takes different values, D α There will be different difference metric functions.

[0121] Exemplarily, when α takes n different values, that is, a set of difference metric functions includes difference metric functions of n different dimensions, each difference metric function corresponds to a feasible solution space, and when the n feasible solution spaces are superimposed together, their intersection can be obtained.

[0122] Step S503 determines whether the predicted category attribute is consistent with the true category attribute and whether the predicted space attribute is consistent with the true space attribute.

[0123] Exemplarily, such consistency determination is achieved based on limiting the maximum degree of difference between the probability distribution of predicted values ​​(including predicted category attributes and predicted space attributes) and the probability distribution of true values ​​(including true category attributes and true space attributes) (ie, the learning target).

[0124] Step S504 responds to determining that the predicted category attributes are consistent with the true category attributes and the predicted space attributes are consistent with the true space attributes, that is, by minimizing the difference measurement function, the probability distribution of the predicted value (including the predicted category attributes and the predicted space attributes) gradually approaches the probability distribution of the true value (including the true category attributes and the true space attributes), thereby obtaining a trained target detection model.

[0125] During the training process, each of this set of difference metric functions looks at the supervision information of the true value from a different divergent perspective, obtaining a suitable observation point for learning the target detection prediction value.

[0126] The present invention introduces the same set of difference metric functions, which adopt a unified expression formula to combine different metric functions, so as to obtain a more compact parameter feasible solution space. At the same time, these difference metric functions can be conveniently switched by setting different α-order parameters, thereby improving the generalization ability of the target detection model under different data and different network structures.

[0127] The following further describes various aspects of the training method of the target detection model according to the embodiments of the present disclosure.

[0128] According to some embodiments, the classification head performs detection based on a first set of difference metric functions, the regression head performs detection based on a second set of difference metric functions, and at least one difference metric function included in the first set of difference metric functions is the same as at least one difference metric function included in the second set of difference metric functions.

[0129] Usually, in classification problems, a metric function based on probability distribution is used to convert the similarity between samples into the possibility of random events. The more common one is KL loss. In regression problems, a metric function based on distance is used to map the input data to a feature space based on distance measurement. The more common one is L2 loss.

[0130] After normalizing the predicted value and the true value, the regression head is measured using the same difference metric function as the classification head. After unifying the difference metric function, the learning of multiple tasks can be better balanced and the impact of different function value ranges can be avoided.

[0131] According to some embodiments, the real feature information is represented by a label, which is used to identify the real category attributes and the real space attributes of the sample target object, and at least one of the classification head and the regression head is trained based on the label after smoothing operation, wherein the smoothing operation is used to add uniform noise to at least one of the real category attributes and the real space attributes, and obtain the real value after label smoothing.

[0132] The label data after label smoothing solves the problem of only considering the loss of the correct label position (the position where the one-hot label is 1) in the original identification training sample, while ignoring the loss of the wrong label position (the position where the one-hot label is 0). It avoids the model from being too confident about the correct label, makes the output value difference of the predicted positive and negative samples not too large, avoids overfitting, and thus improves the generalization ability of the model.

[0133] According to some embodiments, the loss function used in the training of the classification head or the regression head is the sum of the products of each of the at least one difference metric function and the corresponding first weight, and the first weight is used to represent the weight of each of the at least one difference metric function in a group of difference metric functions. That is, for the same group of difference metric functions, when the number of difference metric functions in the same group of difference metric functions is greater than or equal to 2, the weight can be allocated to each difference metric function based on a multi-objective optimization method. Taking the same group of difference metric functions including four difference metric functions with different α values ​​as an example, the loss function of a single detection head (which can be a classification head or a regression head) is expressed as:

[0134] L i (θ) = a i1 D α1 (θ)+a i2 D α2 (θ)+a i3 D α3 (θ)+a i4 D α4 (θ)

[0135] Among them, D α1 (θ), D α2 (θ), D α3 (θ) and D α4 (θ) are the difference measurement functions with four different α values, a i1 、a i2 、a i3 and a i4 are the first weights of the four difference metric functions respectively.

[0136] According to some embodiments, the loss function of the classification head and the loss function of the regression head have respective second weights, and the second weights are learned in training using a multi-objective balancing strategy based on the number of classification heads and regression heads.

[0137] By introducing the second weight to balance the weight of the loss function between the classification head and the regression head, taking the same set of difference metric functions including four difference metric functions with different α values ​​as an example, the total loss function of the entire target detection model is expressed as:

[0138] L(θ)=b1L1(θ)+b2L2(θ)+b3L3(θ)+b4L4(θ)

[0139] Among them, L1(θ), L2(θ), L3(θ) and L4(θ) respectively correspond to the loss functions of the detection head for detecting the category attribute, orientation attribute, position attribute and size attribute of the target object, and b1, b2, b3 and b4 are the second weights of these four loss functions respectively.

[0140] The training of target detection uses a total loss function, which is the sum of the loss function of the classification head and the loss function of the regression head. Taking the same set of difference metric functions including four difference metric functions with different α values ​​as an example, the total loss function of the entire target detection model is expressed as:

[0141] L(θ)=b1L1(θ)+b2L2(θ)+b3L3(θ)+b4L4(θ)

[0142] =a 11 bD α1 (θ)+a 12 bD α2 (θ)+a 13 bD α3 (θ)+a 14 bD α4 (θ)+

[0143] a 21 b2D α5 (θ)+a 22 bD α6 (θ)+a 23 b2D α7 (θ)+a 24 b2D α8 (θ)+a 31 3D α9 (θ)+

[0144] a 32 3D α10 (θ)+a 33 3D α11 (θ)+a 34 3D α12 (θ)+a 41 bD α13 (θ)+

[0145] a 42 bD α14 (θ)+a 43 bD α15 (θ)+a 44 bD α16 (θ)

[0146] According to some embodiments, a total loss function is used in training, and the total loss function is the sum of the products of at least one difference metric function in the loss function used by the classification head or the regression head in pre-training and the corresponding third weight, and the third weight is used to represent the weight of each of the at least one difference metric function in the total loss function. That is, the third weight λ is introduced k To balance the weights of the loss functions of different tasks, we can use λ k Understood as the difference measurement function D αk The weight in the total loss function. Therefore, the above formula can also be rewritten as:

[0147] L(θ)=λ1D α1 (θ)+λ2D α2 (θ)+…+λ K D αK (θ)

[0148] In order to better balance the objectives of the classification task and the regression task, the third weight λ k It can exist independently without relying on the product of the first weight and the second weight, and use a weight learning strategy based on Pareto optimality to perform multi-objective balance optimization.

[0149] The multi-task learning process is abstracted into the following inner and outer double-layer optimization problem:

[0150]

[0151] Among them, the third weight λ of the inner layer optimization k Dynamic adjustments are made during the optimization process, and the outer optimization adjusts the model parameters θ according to the weighted sum loss function.

[0152] According to another aspect of the present disclosure, a target detection device is also provided.

[0153] Figure 6 FIG. 6 shows a structural block diagram of a target detection device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the target detection device 600 includes: an acquisition module 610 and a processing module 620 .

[0154] The acquisition module 610 is configured to acquire feature information of the target object. The processing module 620 is configured to detect the category attribute of the target object using the classification head based on the feature information, and detect the spatial attribute of the target object using the regression head, wherein the spatial attribute includes at least one of an orientation attribute, a position attribute, and a size attribute. At least one of the classification head and the regression head performs detection based on a set of difference metric functions, and the set of difference metric functions is characterized by a Reine difference metric to indicate at least one difference metric function.

[0155] During the training process, each of this set of difference metric functions looks at the supervision information of the true value from a different divergent perspective, obtaining a suitable observation point for learning the target detection prediction value.

[0156] The present invention introduces the same set of difference metric functions, which adopt a unified expression formula to combine different metric functions, so as to obtain a more compact parameter feasible solution space. At the same time, these difference metric functions can be conveniently switched by setting different α-order parameters, thereby improving the generalization ability of the target detection model under different data and different network structures.

[0157] According to some embodiments, the classification head performs detection based on a first set of difference metric functions, the regression head performs detection based on a second set of difference metric functions, and at least one difference metric function included in the first set of difference metric functions is the same as at least one difference metric function included in the second set of difference metric functions.

[0158] Usually, in classification problems, a metric function based on probability distribution is used to convert the similarity between samples into the possibility of random events. The more common one is KL loss. In regression problems, a metric function based on distance is used to map the input data to a feature space based on distance measurement. The more common one is L2 loss.

[0159] After normalizing the predicted value and the true value, the regression head is measured using the same difference metric function as the classification head. After unifying the difference metric function, the learning of multiple tasks can be better balanced and the impact of different function value ranges can be avoided.

[0160] According to some embodiments, at least one of the classification head and the regression head is pre-trained based on a label that has undergone a smoothing operation, wherein the label is used to identify the true value of the target object, the true value includes a true category attribute and a real space attribute, and the smoothing operation is used to add uniform noise to at least one of the true category attribute and the real space attribute, and obtain the true value after label smoothing.

[0161] The label data after label smoothing solves the problem of only considering the loss of the correct label position (the position where the one-hot label is 1) in the original identification training sample, while ignoring the loss of the wrong label position (the position where the one-hot label is 0). It avoids the model from being too confident about the correct label, makes the output value difference of the predicted positive and negative samples not too large, avoids overfitting, and thus improves the generalization ability of the model.

[0162] According to some embodiments, the loss function used by the classification head or the regression head in pre-training is the sum of the products of each of the at least one difference metric function and the corresponding first weight, and the first weight is used to represent the weight of each of the at least one difference metric function in a group of difference metric functions. That is, for the same group of difference metric functions, when the number of difference metric functions in the same group of difference metric functions is greater than or equal to 2, the weight can be allocated to each difference metric function based on a multi-objective optimization method.

[0163] According to some embodiments, the loss function of the classification head and the loss function of the regression head have respective second weights, and the second weights are learned in pre-training using a multi-objective balancing strategy based on the number of classification heads and regression heads.

[0164] In pre-training, object detection uses a total loss function, which is the sum of the loss function of the classification head and the loss function of the regression head.

[0165] According to some embodiments, a total loss function is used in pre-training, and the total loss function is the sum of the products of at least one difference metric function in the loss function used by the classification head or the regression head in pre-training and the corresponding third weight, and the third weight is used to represent the weight of each of the at least one difference metric function in the total loss function. That is, the third weight λ is introduced k To balance the weights of the loss functions of different tasks, we can use λ k Understood as the difference measurement function D αk The weight in the total loss function. Therefore, the total loss function of the entire target detection model is expressed as:

[0166] L(θ)=λ1D α1 (θ)+λ2D α2 (θ)+...+λ K D αK (θ)

[0167] In order to better balance the objectives of the classification task and the regression task, the third weight λ k It can exist independently without relying on the product of the first weight and the second weight, and use a weight learning strategy based on Pareto optimality to perform multi-objective balance optimization.

[0168] Figure 7 FIG. 8 is a block diagram of a target detection device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the target detection device 700 includes: an acquisition module 710 and a processing module 720 .

[0169] The feature information includes bird's-eye view BEV feature information, wherein the acquisition module 710 includes a correlation information acquisition module 711 and a bird's-eye view BEV feature generation module 712 .

[0170] The associated information acquisition module 711 is configured to acquire information data associated with the target object; and the bird's-eye view BEV feature generation module 712 is configured to generate bird's-eye view BEV feature information based on the information data associated with the target object. The bird's-eye view BEV feature information is feature information obtained based on the perception under the bird's-eye view. It is a natural description to unify and represent different perspectives under the bird's-eye view, which can facilitate subsequent planning and control tasks; at the same time, the target object under the bird's-eye view does not have the scale and occlusion problems under the image perspective.

[0171] Figure 8 FIG. 8 is a block diagram showing a structure of a target detection device 800 according to an embodiment of the present disclosure. Figure 8 As shown, the target detection device 800 includes: an acquisition module 810 and a processing module 820 , and the acquisition module 810 includes a correlation information acquisition module 811 and a bird's-eye view BEV feature generation module 812 .

[0172] The bird's-eye view BEV feature generation module 812 includes a multi-scale extraction submodule 8121, a multi-scale fusion submodule 8122, and a multi-scale conversion submodule 8123. The multi-scale extraction submodule 8121 is configured to extract multiple feature information at different scales from the information data associated with the target object; the multi-scale fusion submodule 8122 is configured to fuse multiple feature information at different scales to obtain multi-scale fused feature information; and the multi-scale conversion submodule 8123 is configured to convert the multi-scale fused feature information to generate the bird's-eye view BEV feature information.

[0173] According to some embodiments, the information data associated with the target object includes at least one of image data of the target object acquired based on at least one first sensor and point cloud data of the target object acquired based on at least one second sensor.

[0174] Exemplarily, the first sensor can be different types of cameras such as visual cameras and infrared cameras, and the cameras can be installed in the front, rear or other positions of the vehicle. The visual camera can capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, image data of target objects such as traffic light indications, intersection conditions, and operating status of other vehicles can be obtained. The infrared camera can capture image data containing the target object under night vision conditions.

[0175] The second sensor may be at least one of an ultrasonic sensor, a millimeter wave radar, and a laser radar. Different sensors may provide different detection accuracy and range. The millimeter wave radar may be installed in front of, behind, or at other locations of the vehicle to measure the distance of an object outside the vehicle from the vehicle using the characteristics of electromagnetic waves. The laser radar may be installed in front of, behind, or at other locations of the vehicle to detect the edge and shape information of an object, thereby identifying and tracking the object. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0176] Fig. 9 FIG. 9 is a block diagram showing a training device 900 for a target detection model according to an embodiment of the present disclosure. Fig. 9 As shown, the training device 900 for the target detection model includes: an information acquisition module 910, a sample prediction module 920, a prediction result determination module 930, and a model acquisition module 940.

[0177] The information acquisition module 910 is configured to acquire sample feature information and real feature information of the sample target object, wherein the real feature information includes real category attributes and real space attributes, and the real space attributes include at least one of the real orientation attributes, real position attributes and real size attributes of the sample target object.

[0178] The sample prediction module 920 is configured to obtain the predicted category attribute of the sample target object based on the sample feature information using the classification head prediction, and to obtain the predicted spatial attribute of the sample target object using the regression head prediction, wherein the predicted spatial attribute includes at least one of the predicted orientation attribute, predicted position attribute and predicted size attribute of the sample target object obtained by the regression head prediction.

[0179] At least one of the classification head and the regression head performs detection based on a set of difference metric functions, and the set of difference metric functions is characterized by a Reiney difference metric to indicate at least one difference metric function.

[0180] The prediction result determination module 930 is configured to determine whether the predicted category attribute is consistent with the true category attribute and whether the predicted space attribute is consistent with the true space attribute.

[0181] The model acquisition module 940 is configured to obtain a trained target detection model in response to determining that the predicted category attribute is consistent with the true category attribute and the predicted space attribute is consistent with the true space attribute.

[0182] During the training process, each of this set of difference metric functions looks at the supervision information of the true value from a different divergent perspective, obtaining a suitable observation point for learning the target detection prediction value.

[0183] The present invention introduces the same set of difference metric functions, which adopt a unified expression formula to combine different metric functions, so as to obtain a more compact parameter feasible solution space. At the same time, these difference metric functions can be conveniently switched by setting different α-order parameters, thereby improving the generalization ability of the target detection model under different data and different network structures.

[0184] According to some embodiments, the classification head performs detection based on a first set of difference metric functions, the regression head performs detection based on a second set of difference metric functions, and at least one difference metric function included in the first set of difference metric functions is the same as at least one difference metric function included in the second set of difference metric functions.

[0185] Usually, in classification problems, a metric function based on probability distribution is used to convert the similarity between samples into the possibility of random events. The more common one is KL loss. In regression problems, a metric function based on distance is used to map the input data to a feature space based on distance measurement. The more common one is L2 loss.

[0186] After normalizing the predicted value and the true value, the regression head is measured using the same difference metric function as the classification head. After unifying the difference metric function, the learning of multiple tasks can be better balanced and the impact of different function value ranges can be avoided.

[0187] According to some embodiments, the real feature information is represented by a label, and the label is used to identify the real category attribute and the real space attribute of the sample target object. At least one of the classification head and the regression head is trained based on the label after the smoothing operation, wherein the label is used to identify the real value of the target object, and the real value includes the real category attribute and the real space attribute, and the smoothing operation is used to add uniform noise to at least one of the real category attribute and the real space attribute, and obtain the real value after the label is smoothed. That is, at least one of the classification head and the regression head is trained based on the label after the smoothing operation.

[0188] The label data after label smoothing solves the problem of only considering the loss of the correct label position (the position where the one-hot label is 1) in the original identification training sample, while ignoring the loss of the wrong label position (the position where the one-hot label is 0). It avoids the model from being too confident about the correct label, makes the output value difference of the predicted positive and negative samples not too large, avoids overfitting, and thus improves the generalization ability of the model.

[0189] Fig.10 FIG. 1 shows a structural block diagram of a training device 1000 for a target detection model according to an embodiment of the present disclosure. Fig.10 As shown, the training device 1000 of the target detection model includes: an information acquisition module 1010, a sample prediction module 1020, a prediction result determination module 1030, a model acquisition module 1040 and a first weight determination module 1050. The configurations of the information acquisition module 1010, the sample prediction module 1020, the prediction result determination module 1030 and the model acquisition module 1040 may be the same as those in the above-mentioned embodiment.

[0190] The first weight determination module 1050 is configured to determine the first weight of at least one difference metric function in a group of difference metric functions, wherein the loss function used by the classification head or the regression head in training is the sum of the products of at least one difference metric function and the corresponding first weight. That is, for the same group of difference metric functions, when the number of difference metric functions in the same group of difference metric functions is greater than or equal to 2, the weight can be allocated to each difference metric function based on a multi-objective optimization method.

[0191] Fig.11 FIG. 1 shows a structural block diagram of a training device 1100 for a target detection model according to an embodiment of the present disclosure. Fig.11 As shown, the training device 1100 of the target detection model includes: an information acquisition module 1110, a sample prediction module 1120, a prediction result determination module 1130, a model acquisition module 1140, a first weight determination module 1150, and a second weight determination module 1160. The configurations of the information acquisition module 1110, the sample prediction module 1120, the prediction result determination module 1130, the model acquisition module 1140, and the first weight determination module 1150 may be the same as those in the aforementioned embodiment.

[0192] The second weight determination module 1160 is configured to determine the second weights of the loss function of the classification head and the loss function of the regression head, respectively, where the second weights are learned in training using a multi-objective balancing strategy based on the number of classification heads and regression heads.

[0193] According to some embodiments, during pre-training, object detection uses a total loss function, which is the sum of the loss function of the classification head and the loss function of the regression head.

[0194] Fig.12 FIG. 1 shows a structural block diagram of a training device 1200 for a target detection model according to an embodiment of the present disclosure. Fig.12 As shown, the training device 1200 for the target detection model includes: an information acquisition module 1210, a sample prediction module 1220, a prediction result determination module 1230, a model acquisition module 1240, and a third weight module 1270.

[0195] The total loss function is used in training, and the training device 1200 also includes: a third weight determination module 1270 is configured to determine the weight of each of the at least one difference metric function in the total loss function; the total loss function is the sum of the products of each of the at least one difference metric function in the loss function used by the classification head or the regression head in training and the corresponding third weight.

[0196] By introducing the third weight λ k To balance the weights of the loss functions of different tasks, we can use λ k Understood as the difference measurement function D αk The weight in the total loss function. Therefore, the total loss function of the entire target detection model is expressed as:

[0197] L(θ)=λ1D α1 (θ)+λ2D α2 (θ)+…+λ K D αK (θ)

[0198] In order to better balance the objectives of the classification task and the regression task, the third weight λ k It can exist independently without relying on the product of the first weight and the second weight, and use a weight learning strategy based on Pareto optimality to perform multi-objective balance optimization.

[0199] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, a computer program product, and a motor vehicle are also provided.

[0200] refer to Fig.13, a block diagram of an electronic device 1300 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0201] like Fig.13 As shown, the device 1300 includes a computing unit 1310, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1320 or a computer program loaded from a storage unit 1380 into a random access memory (RAM) 1330. In the RAM 1330, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1310, the ROM 1320, and the RAM 1330 are connected to each other via a bus 1340. An input / output (I / O) interface 1350 is also connected to the bus 1340.

[0202] A plurality of components in the device 1300 are connected to the I / O interface 1350, including: an input unit 1360, an output unit 1370, a storage unit 1380, and a communication unit 1390. The input unit 1360 may be any type of device capable of inputting information to the device 1300, the input unit 1360 may receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1370 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1380 may include but is not limited to a disk and an optical disk. The communication unit 1390 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0203] The computing unit 1310 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1310 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1310 performs the various methods and processes described above, such as the target detection method or training method mentioned in the aforementioned embodiment. For example, in some embodiments, the target detection method or training method mentioned in the aforementioned embodiment may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1380. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1320 and / or the communication unit 1390. When the computer program is loaded into the RAM 1330 and executed by the computing unit 1310, one or more steps of the target detection method or training method described above may be performed. Alternatively, in other embodiments, the computing unit 1310 may be configured to execute the target detection method or training method mentioned in the aforementioned embodiments in any other appropriate manner (for example, by means of firmware).

[0204] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0205] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0206] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0207] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0208] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0209] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network.

[0210] The relationship between client and server is generated by a computer program that creates a client-server relationship. The server can be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0211] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0212] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A target detection method, comprising: Obtain characteristic information of the target object; as well as Based on the feature information, a classification head is used to detect a category attribute of the target object, and a regression head is used to detect a spatial attribute of the target object, wherein the spatial attribute includes at least one of an orientation attribute, a position attribute, and a size attribute. wherein at least one of the classification head and the regression head performs the detection based on a set of difference metric functions, wherein the set of difference metric functions is characterized by a Raney difference metric to indicate at least one difference metric function, At least one of the classification head and the regression head is pre-trained based on a label that has undergone a smoothing operation, wherein the label is used to identify a true category attribute and a true spatial attribute of the target object, and the smoothing operation is used to add uniform noise to at least one of the true category attribute and the true spatial attribute, The loss function used by the classification head or the regression head in the pre-training is the sum of the products of each of the at least one difference metric function and a corresponding first weight, and the first weight is used to represent the weight of each of the at least one difference metric function in the set of difference metric functions, The loss function of the classification head and the loss function of the regression head have respective second weights, and the second weights are learned in the pre-training using a multi-objective balance strategy based on the number of the classification heads and the regression heads.

2. The method according to claim 1, wherein: The classification head performs the detection based on a first set of difference metric functions, the regression head performs the detection based on a second set of difference metric functions, and the at least one difference metric function included in the first set of difference metric functions is the same as the at least one difference metric function included in the second set of difference metric functions.

3. The method according to claim 1, wherein: A total loss function is used in the pre-training, and the total loss function is the sum of the products of the at least one difference metric function in the loss function used by the classification head or the regression head in the pre-training and the corresponding third weight, and the third weight is used to represent the weight of each of the at least one difference metric function in the total loss function.

4. The method according to any one of claims 1 to 3, wherein: The characteristic information includes bird's-eye view BEV characteristic information, wherein the acquiring characteristic information of the target object includes: Acquiring information data associated with the target object; and The bird's eye view BEV feature information is generated based on information data associated with the target object.

5. The method according to claim 4, wherein: The generating the bird's-eye view BEV feature information based on the information data associated with the target object includes: Extracting a plurality of feature information at different scales from information data associated with the target object; fusing the plurality of feature information at different scales to obtain multi-scale fused feature information; and The multi-scale fusion feature information is converted to generate the bird's-eye view BEV feature information.

6. The method according to claim 4, wherein: The information data associated with the target object includes at least one of image data of the target object acquired based on at least one first sensor and point cloud data of the target object acquired based on at least one second sensor.

7. A method for training a target detection model, comprising: Acquire sample feature information and real feature information of the sample target object, wherein the real feature information includes a real category attribute and a real space attribute, and the real space attribute includes at least one of a real orientation attribute, a real position attribute, and a real size attribute of the sample target object; Based on the sample feature information, a predicted category attribute of the sample target object is obtained by using a classification head for prediction, and a predicted spatial attribute of the sample target object is obtained by using a regression head for prediction, wherein the predicted spatial attribute includes at least one of a predicted orientation attribute, a predicted position attribute, and a predicted size attribute of the sample target object obtained by the regression head for prediction, wherein at least one of the classification head and the regression head performs the prediction based on a set of difference metric functions, and the set of difference metric functions is characterized by a Raney difference metric to indicate at least one difference metric function; determining whether the predicted category attribute is consistent with the true category attribute and whether the predicted spatial attribute is consistent with the true spatial attribute; and In response to determining that the predicted category attribute is consistent with the true category attribute and the predicted spatial attribute is consistent with the true spatial attribute, obtaining a trained object detection model, The real feature information is represented by a label, and the label is used to identify the real category attribute and the real space attribute of the sample target object. At least one of the classification head and the regression head performs the training based on the label after a smoothing operation, and the smoothing operation is used to add uniform noise to at least one of the real category attribute and the real space attribute. The loss function used by the classification head or the regression head in the training is the sum of the products of each of the at least one difference metric function and a corresponding first weight, and the first weight is used to represent the weight of each of the at least one difference metric function in the set of difference metric functions, The loss function of the classification head and the loss function of the regression head have respective second weights, and the second weights are learned in the training using a multi-objective balance strategy based on the number of the classification heads and the regression heads.

8. The method according to claim 7, wherein: The classification head performs the detection based on a first set of difference metric functions, the regression head performs the detection based on a second set of difference metric functions, and the at least one difference metric function included in the first set of difference metric functions is the same as the at least one difference metric function included in the second set of difference metric functions.

9. The method according to claim 7, wherein: A total loss function is used in the training, and the total loss function is the sum of the products of each of the at least one difference metric function in the loss function used by the classification head or the regression head in the training and the corresponding third weight, and the third weight is used to represent the weight of each of the at least one difference metric function in the total loss function.

10. A target detection device, comprising: An acquisition module, configured to acquire feature information of a target object; as well as a processing module configured to detect a category attribute of the target object using a classification head based on the feature information, and to detect a spatial attribute of the target object using a regression head, wherein the spatial attribute includes at least one of an orientation attribute, a position attribute, and a size attribute; wherein at least one of the classification head and the regression head performs the detection based on a set of difference metric functions, wherein the set of difference metric functions is characterized by a Raney difference metric to indicate at least one difference metric function, At least one of the classification head and the regression head is pre-trained based on a label that has undergone a smoothing operation, wherein the label is used to identify a true category attribute and a true spatial attribute of the target object, and the smoothing operation is used to add uniform noise to at least one of the true category attribute and the true spatial attribute, The loss function used by the classification head or the regression head in the pre-training is the sum of the products of each of the at least one difference metric function and a corresponding first weight, and the first weight is used to represent the weight of each of the at least one difference metric function in the set of difference metric functions, The loss function of the classification head and the loss function of the regression head have respective second weights, and the second weights are learned in the pre-training using a multi-objective balance strategy based on the number of the classification heads and the regression heads.

11. The device according to claim 10, wherein: The classification head performs the detection based on a first set of difference metric functions, the regression head performs the detection based on a second set of difference metric functions, and the at least one difference metric function included in the first set of difference metric functions is the same as the at least one difference metric function included in the second set of difference metric functions.

12. The device according to claim 10, wherein: A total loss function is used in the pre-training, and the total loss function is the sum of the products of the at least one difference metric function in the loss function used by the classification head or the regression head in the pre-training and the corresponding third weight, and the third weight is used to represent the weight of each of the at least one difference metric function in the total loss function.

13. The device according to any one of claims 10 to 12, wherein: The characteristic information includes bird's-eye view BEV characteristic information, wherein the acquisition module includes: an associated information acquisition module, configured to acquire information data associated with the target object; and The bird's-eye view BEV feature generation module is configured to generate the bird's-eye view BEV feature information based on the information data associated with the target object.

14. The device according to claim 13, wherein: The bird's eye view BEV feature generation module includes: A multi-scale extraction submodule, configured to extract a plurality of feature information at different scales from the information data associated with the target object; a multi-scale fusion submodule, configured to fuse the plurality of feature information of different scales to obtain multi-scale fused feature information; and The multi-scale conversion submodule is configured to convert the multi-scale fusion feature information to generate the bird's-eye view BEV feature information.

15. The device according to claim 13, wherein: The information data associated with the target object includes at least one of image data of the target object acquired based on at least one first sensor and point cloud data of the target object acquired based on at least one second sensor.

16. A training device for a target detection model, comprising: an information acquisition module, configured to acquire sample feature information and real feature information of the sample target object, wherein the real feature information includes a real category attribute and a real space attribute, and the real space attribute includes at least one of a real orientation attribute, a real position attribute, and a real size attribute of the sample target object; and The sample prediction module is configured to predict, based on the sample feature information, a predicted category attribute of the sample target object using a classification head, and predict a predicted spatial attribute of the sample target object using a regression head, wherein the predicted spatial attribute includes at least one of a predicted orientation attribute, a predicted position attribute, and a predicted size attribute of the sample target object predicted by the regression head, wherein at least one of the classification head and the regression head performs the detection based on a set of difference metric functions, and the set of difference metric functions is characterized by a Raney difference metric to indicate at least one difference metric function; a prediction result determination module, configured to determine whether the predicted category attribute is consistent with the true category attribute and whether the predicted space attribute is consistent with the true space attribute; and A model acquisition module is configured to obtain a trained object detection model in response to determining that the predicted category attribute is consistent with the true category attribute and the predicted space attribute is consistent with the true space attribute, The real feature information is represented by a label, and the label is used to identify the real category attribute and the real space attribute of the sample target object. Wherein, the device further comprises: a label smoothing module configured to perform a smoothing operation on the label, wherein the smoothing operation is used to add uniform noise to at least one of the true category attribute and the true spatial attribute, Wherein, the device further comprises: A first weight determination module is configured to determine a first weight of each of the at least one difference metric function in the set of difference metric functions, The loss function used by the classification head or the regression head in the training is the sum of the products of each of the at least one difference metric function and the corresponding first weight, Wherein, the device further comprises: The second weight determination module is configured to determine the second weights of the loss function of the classification head and the loss function of the regression head respectively, and the second weights are learned in the training using a multi-objective balance strategy based on the number of the classification heads and the regression heads.

17. The device according to claim 16, wherein: The classification head performs the detection based on a first set of difference metric functions, the regression head performs the detection based on a second set of difference metric functions, and the at least one difference metric function included in the first set of difference metric functions is the same as the at least one difference metric function included in the second set of difference metric functions.

18. The device according to claim 16, wherein: Using a total loss function in the training, the apparatus further comprises: A third weight determination module, configured to determine a weight of each of the at least one difference metric function in the total loss function; The total loss function is the sum of products of each of the at least one difference metric function in the loss function used by the classification head or the regression head in the training and the corresponding third weight.

19. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target detection method described in any one of claims 1-6 or the training method of the target detection model described in any one of claims 7-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the target detection method according to any one of claims 1-6 or the training method of the target detection model according to any one of claims 7-9.

21. A computer program product comprising a computer program, wherein: When executed by a processor, the computer program implements the target detection method described in any one of claims 1 to 6 or the training method of the target detection model described in any one of claims 7 to 9.

22. A motor vehicle comprising the electronic device as claimed in claim 19.

Images