Training method of target detection model, map generation method, device and equipment

Abstract

The method comprises the following steps: acquiring training input data and corresponding first label data, the training input data comprising training image data and / or training point cloud data, and the first label data comprising ordered point sets corresponding to a first number of instances in the training input data, wherein each ordered point set comprises a target number of coordinate points in a first coordinate system; training a pre-established target detection network based on the training input data, the first label data, a point-to-point loss function and a direction loss function, to obtain a target detection model. The target detection model obtained by the method can accurately and effectively predict the ordered point sets corresponding to each instance, realizes coordinate point level prediction of the instance, and greatly improves the accuracy of the prediction result compared with the existing instance box level prediction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to automatic driving technology, and in particular to a target detection model training method, a map generation method, apparatus and device. BACKGROUND

[0002] In an automatic driving scenario, it is usually necessary to utilize a vehicle-mounted surround view camera and / or a radar to perceive road elements (such as lane lines, zebra crossings, road edges, drivable areas, and the like) for online map generation. In the related art, a target detection model is usually used to predict map instance boxes corresponding to various elements, to obtain map instance box positions, and then generate a map based on the map instance box positions. However, the generated map has low precision. SUMMARY

[0003] To solve the above technical problems such as low precision of a map generated based on map instance box positions, the present disclosure is proposed. Embodiments of the present disclosure provide a target detection model training method, a map generation method, apparatus and device.

[0004] According to an aspect of an embodiment of the present disclosure, a target detection model training method is provided, including: obtaining training input data and corresponding first label data, the training input data including training image data and / or training point cloud data, the first label data including ordered point sets corresponding to a first number of instances in the training input data, the ordered point set including a target number of coordinate points in a first coordinate system; training a pre-established target detection network based on the training input data, the first label data, a point-to-point loss function and a direction loss function, to obtain a target detection model, the point-to-point loss function being used to determine a point-to-point loss of training instance point sets output by the target detection network relative to ordered point sets of instances in the first label data, and the direction loss function being used to determine a loss of directions between points in the training instance point sets relative to directions between points in the ordered point sets of instances in the first label data.

[0005] According to another aspect of an embodiment of the present disclosure, a map generation method is provided, including: obtaining first image data and / or first point cloud data of at least one perspective; based on the first image data and / or the first point cloud data, using a pre-trained target detection model to obtain target instance ordered point sets, the target detection model being obtained by the target detection model training method according to any one of the above embodiments, the target instance ordered point sets including ordered point sets corresponding to a first number of instances, and the ordered point set including a target number of coordinate points in a first coordinate system; and generating a map based on the target instance ordered point sets.

[0006] According to still another aspect of the embodiments of the present disclosure, a device for training a target detection model is provided, comprising: a first obtaining module configured to obtain training input data and corresponding first label data, the training input data comprising training image data and / or training point cloud data, the first label data comprising ordered point sets corresponding to a first number of instances in the training input data, each ordered point set comprising a target number of coordinate points in a first coordinate system; and a first processing module configured to train a pre-established target detection network based on the training input data, the first label data, a point-to-point loss function and a direction loss function, to obtain a target detection model, the point-to-point loss function being configured to determine a point-to-point loss of training instance point sets output by the target detection network relative to the ordered point sets of instances in the first label data, and the direction loss function being configured to determine a loss of directions between points in the training instance point sets relative to directions between points in the ordered point sets of instances in the first label data.

[0007] According to still another aspect of the embodiments of the present disclosure, a device for generating a map is provided, comprising: a second obtaining module configured to obtain first image data and / or first point cloud data of at least one view angle; and a second processing module configured to obtain target instance ordered point sets based on the first image data and / or the first point cloud data using a pre-trained target detection model, the target detection model being obtained by the training method of the target detection model according to any one of the above embodiments, and each target instance ordered point set comprising a first number of ordered point sets corresponding to instances, each ordered point set comprising a target number of coordinate points in a first coordinate system.

[0008] According to still another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute the training method of the target detection model according to any one of the above embodiments of the present disclosure, or the computer program being configured to execute the generation method of the map according to any one of the above embodiments of the present disclosure.

[0009] According to still another aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory configured to store executable instructions of the processor; and the processor configured to read the executable instructions from the memory and execute the instructions to implement the training method of the target detection model according to any one of the above embodiments of the present disclosure.

[0010] According to still another aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory configured to store executable instructions of the processor; and the processor configured to read the executable instructions from the memory and execute the instructions to implement the generation method of the map according to any one of the above embodiments of the present disclosure.

[0011] The training method of the target detection model, the map generation method, the device and the equipment provided by the above-mentioned embodiments of the present disclosure can effectively improve the precision by using the ordered point set corresponding to the instance as the label, combining the point-to-point loss and the direction loss to train the pre-established target detection network, and obtaining the target detection model capable of predicting the ordered point set of the instance for the image data and / or the point cloud data, that is, realizing the prediction at the coordinate point level of the map element, relative to the existing prediction at the instance box level.

[0012] The technical solutions of the present disclosure will be described in further detail below by means of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description of the embodiments of the present disclosure taken in conjunction with the accompanying drawings. The accompanying drawings are provided to assist in understanding the embodiments of the present disclosure and constitute a part of the specification. The accompanying drawings together with the present disclosure are used to explain the present disclosure and do not constitute a limitation on the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0014] Figure 1 is an exemplary application scenario of the training method of the target detection model provided by the present disclosure;

[0015] Figure 2 is a flowchart of the training method of the target detection model provided by an exemplary embodiment of the present disclosure;

[0016] Figure 3 is a flowchart of step 202 provided by an exemplary embodiment of the present disclosure;

[0017] Figure 4 is a structural diagram of the target detection network provided by an exemplary embodiment of the present disclosure;

[0018] Figure 5 is a flowchart of step 202 provided by another exemplary embodiment of the present disclosure;

[0019] Figure 6 is a flowchart of step 2021 provided by an exemplary embodiment of the present disclosure;

[0020] Figure 7 is a structural diagram of the decoder network provided by an exemplary embodiment of the present disclosure;

[0021] Figure 8 is a principle diagram of Deformable DETR provided by an exemplary embodiment of the present disclosure;

[0022] Figure 9is a schematic diagram of a determination principle of a training instance point set provided by another example embodiment of the present disclosure.

[0023] Figure 10 is a schematic diagram of step 202 provided by another example embodiment of the present disclosure.

[0024] Figure 11 is a schematic diagram of a prediction network of a prediction type provided by an example embodiment of the present disclosure.

[0025] Figure 12 is a schematic diagram of step 2024 provided by an example embodiment of the present disclosure.

[0026] Figure 13 is a schematic diagram of a generation method of a map provided by an example embodiment of the present disclosure.

[0027] Figure 14 is a structural schematic diagram of a training device of a target detection model provided by an example embodiment of the present disclosure.

[0028] Figure 15 is a structural schematic diagram of a first processing module 502 provided by an example embodiment of the present disclosure.

[0029] Figure 16 is a structural schematic diagram of a second processing unit 5022 provided by an example embodiment of the present disclosure.

[0030] Figure 17 is a structural schematic diagram of a first processing unit 5021 provided by an example embodiment of the present disclosure.

[0031] Figure 18 is a structural schematic diagram of a first processing module 502 provided by another example embodiment of the present disclosure.

[0032] Figure 19 is a structural schematic diagram of a generation device of a map provided by an example embodiment of the present disclosure.

[0033] Figure 20 is a structural schematic diagram of an electronic device according to an application embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0035] It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0036] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor indicate their logical order.

[0037] It should also be understood that in the embodiments of the present disclosure, "multiple" can mean two or more, and "at least one" can mean one, two or more.

[0038] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, it can be understood as one or more in general, without explicit limitation or in the context of the opposite indication.

[0039] In addition, the term "and / or" in the present disclosure is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0040] It should also be understood that the description of the embodiments of the present disclosure emphasizes the differences between the embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0041] At the same time, it should be understood that in order to facilitate the description, the size of each part shown in the drawings is not drawn according to the actual proportional relationship.

[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application or uses.

[0043] The techniques, methods and devices known to those skilled in the relevant art can not be discussed in detail, but in appropriate cases, the techniques, methods and devices should be considered as part of the specification.

[0044] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0045] The disclosed embodiments can be applied to terminal devices, computer systems, servers, and other electronic devices, which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations that can be suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computers, server computers, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputers, mainframe computers, and distributed cloud computing technology environments that include any of the above systems, and the like.

[0046] Terminal devices, computer systems, servers, and other electronic devices can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, that perform particular tasks or implement particular abstract data types. Computer systems / servers can be practiced in distributed cloud-computing environments with remote processing devices that are linked through a communications network. In a distributed cloud-computing environment, program modules can be located in local or remote computer system storage media including memory storage devices.

[0047] Summary of the Disclosure

[0048] In the process of implementing the present disclosure, the inventors found that in the automatic driving scenario, it is usually necessary to utilize the on-board surround view camera and / or radar to perceive road elements (such as lane lines, zebra crossings, road edges, drivable areas, and the like) for the generation of an online map. In the related art, a target detection model is usually used to predict the map instance frame corresponding to each type of element, to obtain the map instance frame position, and then generate a map based on the map instance frame position. However, the generated map has low precision.

[0049] Example Summary

[0050] Figure 1 is one exemplary application scenario of the training method of the target detection model provided by the present disclosure.

[0051] In the automatic driving scene, by using the training method of the target detection model of the present disclosure, the pre-acquired image data can be used as the training image data, the pre-acquired point cloud data can be used as the training point cloud data, the training input data can be formed, the label corresponding to the training image data and the label corresponding to the training point cloud data can be used as the first label data for training of the target detection model, the network output of the target detection model is an ordered point set corresponding to each of the first quantity of instances, that is, each instance corresponds to an ordered point set, and each ordered point set can include a target quantity of coordinate points in a first coordinate system. The first coordinate system can be a coordinate system corresponding to a bird's eye view. The instance can be a representation of various elements on the road in the image or the point cloud, such as each lane line, zebra crossing, arrow, road edge, and drivable area in the image. Each element is an instance, that is, each element in the image or the point cloud can correspond to one or more instances, and the corresponding ordered point set of each instance can be predicted, which can fit the element corresponding to the instance, such as a lane line instance, and the ordered point set includes three coordinate points, which can fit a lane line. In the training process, the network parameters of the target detection model are adjusted based on the point-to-point loss function and the direction loss function, so that the target detection model obtained by training can effectively detect the ordered point set corresponding to each instance. Since the target detection model obtained by training predicts the coordinate points of the instance, the accuracy of the prediction result can be greatly improved compared with the prediction of the existing instance frame. Further, the target detection model obtained by training is deployed to a map generation device on a vehicle-mounted computing platform of an automatic driving vehicle for online mapping of the automatic driving vehicle, which can effectively improve the accuracy of the generated map.

[0052] In actual applications, the training method of the target detection model of the present disclosure is not limited to the automatic driving scene, but can also be applied to any other implementable scene according to actual needs.

[0053] Example Method

[0054] Figure 2 is a flowchart of the training method of the target detection model provided by an exemplary embodiment of the present disclosure. The present embodiment can be applied on electronic devices, such as servers, terminals, and the like, as shown in Figure 2 , including the following steps:

[0055] In step 201, training input data and corresponding first label data are obtained, the training input data includes training image data and / or training point cloud data, and the first label data includes an ordered point set corresponding to each of the first quantity of instances in the training input data. The ordered point set includes a target quantity of coordinate points in a first coordinate system.

[0056] The training image data and the training point cloud data can be obtained based on a vehicle-mounted surround-view camera and a radar. For example, a special collection vehicle provided with a surround-view camera and a radar drives on a road to collect road environment images and road point cloud data around the collection vehicle as the training image data and the training point cloud data. The first label data is obtained by sequentially labeling instances in the training image data and / or the training point cloud data. Since the first label data includes coordinate points in a first coordinate system, the first coordinate system can be a coordinate system corresponding to a bird's-eye view, the training image data is data in an image coordinate system, and the training point cloud data is data in a radar coordinate system, the labeling result can be converted to the first coordinate system based on camera parameters and radar parameters to obtain corresponding first label data. The specific conversion principle is not described herein. The instances can be representations of various elements on the road in images or point clouds, such as each lane line, zebra crossing, arrow, road edge, and drivable area in the image. Each element in the image or point cloud can correspond to one or more instances, and each instance corresponds to an ordered point set, which can fit the element corresponding to the instance. For example, the lane line instance, the ordered point set includes three coordinate points, and a lane line can be fitted through the three coordinate points. The first number and the target number can be set according to actual requirements.

[0057] In step 202, the pre-established target detection network is trained based on the training input data, the first label data, a point-to-point loss function, and a direction loss function to obtain a target detection model. The point-to-point loss function is used to determine a point-to-point loss of a training instance point set output by the target detection network relative to an ordered point set of an instance in the first label data. The direction loss function is used to determine a loss of a direction between points in the training instance point set relative to a direction between points in the ordered point set of the instance in the first label data.

[0058] The target detection network can be set according to actual needs, such as a deformable detection transformer (Deformable DEtection TRansformer, Deformable DETR for short) detection network and other implementable detection networks. The point-to-point loss function and the direction loss function can adopt any implementable loss function, such as an L1 loss function for the point-to-point loss function, which is also known as least absolute deviation (LAD) or least absolute error (LAE), which minimizes the sum of the absolute difference between the target value (in this disclosure, the label value of the annotation) and the estimated value (in this disclosure, the output value of the target detection network), and a cosine similarity loss function of the direction vectors of two adjacent points for the direction loss function. In the training process, the network parameters of the target detection model are adjusted based on the point-to-point loss function and the direction loss function, so that the target detection model obtained by training can effectively detect the ordered point set corresponding to each instance. The point-to-point loss determined by the point-to-point loss function is used to supervise the point-level prediction result of the target detection network, so that the target detection network can accurately predict the points of the instance. The direction loss determined by the direction loss function is used to supervise the order of the points, so that the target detection network can predict a more accurate ordered point set. Since the target detection model obtained by training predicts the ordered coordinate points of the instance, the prediction accuracy of the prediction result can be greatly improved compared with the existing instance frame prediction.

[0059] The training method of the target detection model provided in this embodiment uses the point-to-point loss determined based on the point-to-point loss function and the direction loss determined based on the direction loss function to supervise the points and the order of the points of the instance point set output by the target detection network, so that the target detection model obtained by training can accurately and effectively predict the ordered point set corresponding to each instance, realizing the coordinate point-level prediction of the instance. Compared with the existing instance frame-level prediction, the target detection model of this disclosure greatly improves the prediction accuracy, and further effectively improves the map accuracy when used for map generation.

[0060] In one optional example, Figure 3 is a flowchart of step 202 provided in an example embodiment of this disclosure. In this example, step 202 can specifically include the following steps:

[0061] Step 2021, obtaining a training instance point set based on training input data and a target detection network.

[0062] The target detection network can be set according to actual needs to input only training image data, input only training point cloud data, or input both training image data and training point cloud data.

[0063] For example, Figure 4 This is a schematic diagram of the structure of an object detection network provided in an exemplary embodiment of this disclosure. In this example, the object detection network includes a feature extraction network, an encoder network, a decoder network, and a prediction head network. The feature extraction network may include a first feature extraction network and / or a second feature extraction network, depending on actual needs. The first feature extraction network is used for feature extraction from training image data to obtain training image features, and the second feature extraction network is used for extraction from training point cloud data to obtain training point cloud features. The encoder network is used to encode the training image features and / or training point cloud features to obtain a training feature map in a first coordinate system. The decoder network is used to decode the training feature map to obtain a training decoding result. The prediction head network is used to predict a set of training instance points based on the training decoding result. For example, the prediction head network may be a linear neural network such as an MLP (Multilayer Perceptron, or feedforward neural network), and the specific configuration can be determined according to actual needs. The training input data is used as input to the object detection network, and after inference by the object detection network, the output set of training instance points can be obtained.

[0064] Different ordered point sets can be used for different instances. For example, for lane line instances, an open-loop ordered point set can be used, where the starting and ending points of the ordered point set are not the same point, and the fitted result is a line segment. Another example is the drivable area instance, where the ordered point set is a polygonal point set, forming a closed-loop ordered point set, and the fitted result is a closed-loop polygon. The specific settings can be configured according to actual needs. The target number of coordinate points in the ordered point sets for different instances can be the same or different. For example, a lane line instance can use an ordered point set with 3 coordinate points, while a zebra crossing instance can use an ordered point set with 5 coordinate points; there is no specific limitation.

[0065] Step 2022: Determine the first loss based on the training instance point set, the first label data, and the point-to-point loss function.

[0066] After obtaining the training instance point set, the point set of each instance in the training instance point set can be compared point-to-point with the ordered point set of that instance in the first label data. The absolute value of the difference between the point-to-point values ​​is used as the loss for that point, thus obtaining the loss for each point in each instance. Based on the losses of each point in each instance, the overall point-to-point loss of the network is determined as the first loss. For example, the losses of each point in each instance can be summed to obtain the first loss. The specific settings can be configured according to actual needs.

[0067] Step 2023: Determine the second loss based on the training instance point set, the first label data, and the orientation loss function.

[0068] The direction loss function is used to determine the loss of the direction between points in the training instance point set relative to the direction between points in the ordered point set of the instance in the first label data, which can be determined based on the cosine similarity of the direction vectors between two points.

[0069] For example, for two adjacent points in an instance in the training instance point set, a first direction vector of the two adjacent points is determined based on the coordinate values of the two adjacent points, a second direction vector of the two adjacent points is determined based on the coordinate values of the two points corresponding to the two adjacent points in the first label data, and the cosine similarity of the two direction vectors is determined based on the first direction vector and the second direction vector. The direction loss of the entire network is determined based on the cosine similarity of the two adjacent points in each instance as the second loss, which can be set according to actual needs.

[0070] Steps 2022 and 2023 are not in chronological order.

[0071] In step 2024, the network parameters of the target detection network are adjusted based on the first loss and the second loss until the first loss and the second loss meet the preset condition, and the target detection model is obtained.

[0072] The first loss and the second loss can be weighted and summed by a preset weight as a comprehensive loss for adjusting the network parameters. The preset condition can be set according to actual needs. The adjustment of the network parameters can be realized by any implementable optimizer, such as the Adam training optimizer. The Adam training optimizer absorbs the advantages of the gradient descent algorithm of adaptive learning rate (Adagrad) and the momentum gradient descent algorithm, which can adapt to sparse gradient (i.e. natural language and computer vision problems) and can alleviate the problem of gradient shock. The specific principle is not described again.

[0073] Figure 5 is a flowchart of step 202 provided by another exemplary embodiment of the present disclosure.

[0074] In an optional example, step 2022 of determining the first loss based on the training instance point set, the first label data, and the point-to-point loss function includes:

[0075] In step 20221, for each instance, based on the ordered point set corresponding to the instance in the first label data, the correspondence between each point in the ordered point set and the point of the instance in the training instance point set is determined in different orders of the ordered point set, respectively, to obtain the point-to-point relationship corresponding to each order.

[0076] Wherein, different orders of the ordered point set refer to orders taking different end points in the ordered point set as starting points, such as the ordered point set of a line segment like a lane line, including three ordered coordinate points A1, A2, A3, wherein there are two end points A1 and A3, and different orders of the ordered point set include two orders, one is A1-A2-A3, and the other is A3-A2-A1, and the adjacent relationship of any two coordinate points in the different orders is unchanged. For example, for the ordered point set of a polygon like a drivable area or a zebra crossing, including ordered coordinate points B1-B5, wherein B5 can be equal to B1 to represent a polygon, or can be represented by other symbols to indicate that the ordered point set corresponds to a polygon, and the first and last points need to be connected to form a closed loop during fitting, which can be set according to actual needs as long as it can be distinguished from the ordered point set of a line segment. It also provides a basis for determining the point-to-point relationship of different orders. For the ordered point set B1-B5, taking B5 not equal to B1 as an example, since each coordinate point is a vertex of the polygon and can be taken as a starting point, the ordered point set has five starting points, and in combination with the direction, the ordered point set corresponds to 10 orders, including B1-B5 and reverse order B5-B1, B2-B3-B4-B5-B1 and its reverse order, B3-B4-B5-B1-B2 and its reverse order, B4-B5-B1-B2-B3 and its reverse order, B5-B1-B2-B3-B4 and its reverse order. By adjusting the order of the coordinate points in the ordered point set in the training instance point set or the first label data, the point-to-point relationship corresponding to the different orders in the corresponding instances in the two is determined. For example, the ordered point set of an instance in the training instance point set is C1-C5, and the valid point set label of the instance in the first label data is D1-D5. After arranging D1-D5 according to the above 10 different orders of B1-B5, they are respectively corresponding to C1-C5, that is, the point-to-point relationship corresponding to different orders is formed. For example, C1-C5 corresponds to D5-D1 in order, and the specific principle is not repeated here.

[0077] In step 20222, based on the point-to-point relationship corresponding to each order respectively, the point-to-point loss corresponding to each order respectively is determined.

[0078] Wherein, after determining the point-to-point relationship corresponding to each order respectively, the point-to-point loss is obtained based on the point-to-point loss function.

[0079] In step 20223, the order with the minimum point-to-point loss is taken as the target order of the instance.

[0080] Wherein, after determining the point-to-point loss corresponding to each order respectively, the minimum point-to-point loss and the order of the minimum point-to-point loss are determined, and the order is taken as the target order of the ordered point set of the instance, which is used for determining the overall point-to-point loss of the network subsequently.

[0081] Step 20224, taking the point-to-point loss corresponding to the target order as the target point-to-point loss of the instance.

[0082] Since the training instance point set includes the ordered point set corresponding to the first number of instances respectively, when the first number is multiple, each instance determines the corresponding target order and the corresponding target point-to-point loss.

[0083] Step 20225, determining the first loss based on the target point-to-point loss of each instance.

[0084] Specifically, the first loss of the entire network is determined by integrating the target point-to-point loss of each instance.

[0085] The present disclosure determines the smallest point-to-point loss for the determination of the entire network point-to-point loss by considering various possible orders of the ordered point set during the training process, so that the target detection network can simulate the optimal starting point of the instance and the order of the corresponding instance, further improving the model performance and the accuracy of the prediction result.

[0086] In an optional example, step 2023 of determining the second loss based on the training instance point set, the first label data, and the direction loss function includes:

[0087] Step 20231, determining the second loss based on the training instance point set, the first label data, the target order corresponding to each instance, and the direction loss function.

[0088] Since the second loss is a direction loss, it involves the directionality of adjacent two points, therefore, when the point-to-point loss adopts the point-to-point relationship of the target order, the direction loss also needs to be determined based on the direction vector of the adjacent two coordinate points determined by the point-to-point relationship of the target order, to ensure the consistency of the predicted point-to-point direction and the label direction.

[0089] Figure 6 The flowchart of step 2021 provided by an exemplary embodiment of the present disclosure is shown in the figure.

[0090] In an optional example, the training input data further includes initial query features and initial reference points, the initial query features include a target number of initial features corresponding to the first number of instances respectively, and the initial reference points include reference coordinate points corresponding to each initial feature; the target detection network is a detection network based on a deformable detection transformer; correspondingly, step 2021 of obtaining the training instance point set based on the training input data and the target detection network includes:

[0091] Step 20211, performing feature extraction on the training image data based on the first feature extraction network in the target detection network to obtain the first training image feature.

[0092] Wherein the deformable DETR is a detection transformer improved from the DETR, which uses a multi-scale deformable attention module instead of the attention module in the DETR to process features, solves the problems of high computational complexity and slow convergence of the DETR, and is the first pure end-to-end target detector, which fully integrates the convolutional neural network (CNN) and the transformer, and realizes target detection by means of the powerful modeling capability of the transformer. The initial query feature can be a randomly initialized query feature, and the initial query feature is used in the decoder attention operation of the target detection network. The initial reference point can be a set of initial reference coordinate points corresponding to each instance, and the first feature extraction network can adopt any implementable feature extraction network, such as a convolutional neural network as the feature extraction network, which can be set according to actual needs.

[0093] Step 20212, performing feature extraction on the training point cloud data based on the second feature extraction network in the target detection network to obtain first training point cloud features.

[0094] Wherein the second feature extraction network can adopt any implementable feature extraction network, such as a convolutional neural network as the feature extraction network, which can be set according to actual needs.

[0095] Wherein the steps 20211 and 20212 are not in a specific order.

[0096] Step 20213, encoding the first training image features and / or the first training point cloud features based on the encoder network in the target detection network to obtain target training feature maps in the first coordinate system.

[0097] Wherein the encoder network includes at least one encoder, and the encoder network converts the first training image features and the first training point cloud features to the first coordinate system through encoding to obtain corresponding target training feature maps.

[0098] Step 20214, obtaining a training decoding result based on the target training feature map, the initial query feature, the initial reference point, and the decoder network in the target detection network, wherein the decoder network includes at least one decoder.

[0099] Wherein the training decoding result is a decoding result obtained by decoding through at least one decoder. The decoder network continuously updates the initial query feature based on the initial reference point and the target training feature map to obtain the training decoding result.

[0100] Step 20215, determining a training instance point set based on the training decoding result.

[0101] The training instance point set is obtained by continuously updating the initial reference points based on the decoding results of each decoder. That is, after each decoder obtains a decoding result, the offset of each reference point corresponding to the decoding result is predicted based on the decoding result. Taking the first decoder as an example, the offset of each initial reference point corresponding to the decoding result of the first decoder is predicted, each offset is added to the corresponding initial reference point, and the updated reference point is obtained as the output reference point corresponding to the first decoder. After the second decoder is decoded, the offset of each output reference point of the first decoder corresponding to the decoding result is predicted based on the decoding result, and is added to each output reference point of the first decoder to obtain the output reference point corresponding to the second decoder. Similarly, the output reference point corresponding to the last decoder is taken as the training instance point set.

[0102] Exemplarily, Figure 7 is a structural schematic diagram of a decoder network provided by an example embodiment of the present disclosure. In the example, the decoder network includes N decoders, the initial query feature includes three initial features corresponding to two instances (instance 1 and instance 2) respectively, the initial reference point includes reference coordinate points corresponding to the two instances respectively, the training decoding result is obtained by decoding the N decoders, and then the training instance point set is obtained based on the training decoding result.

[0103] Exemplarily, Figure 8 is a principle schematic diagram of Deformable DETR provided by an example embodiment of the present disclosure. In the example, taking the first decoder as an example, Query Feature represents the initial query feature, Reference Point represents the initial reference point, Input Feature Map represents the target training feature map, the variable attention module of the decoder only focuses on a part of the range near the reference point each time, and does not consider the resolution of the entire feature map. Head represents the attention head, m represents the mth attention head, Attention Weights (Amqk) represents the attention weight, W' represents the linear layer, Aggregate represents the aggregation operation of the attention weight and the key point in the value vector Values, Softmax represents the activation function, and Output represents the decoding result. Compared with DETR, Deformable DETR only collects the main feature points near the reference point, so for each query vector Query, there are only a very small number of key vectors (Key). The initial query feature is predicted through the linear layer to obtain the feature offset (Sampling Offsets) Δp m x represents the encoding of the key vector (KEY) in the attention operation, Linear represents the linear layer, Aggregate represents the aggregation operation of the attention weight and the key point in the value vector Values, Softmax represents the activation function, and Output represents the decoding result. Compared with DETR, Deformable DETR only collects the main feature points near the reference point, so for each query vector Query, there are only a very small number of key vectors (Key). The initial query feature is predicted through the linear layer to obtain the feature offset (Sampling Offsets) Δp mqk , the feature offset (Sampling Offsets) Δp mqkIn each attention head (such as Head1), 3 feature offsets are predicted, represented by 3 arrows respectively. The feature offset refers to the position offset of the key point collected in the value vector relative to the initial reference point. The input feature map obtains the value vector Values through the linear layer. Each attention head obtains the corresponding value vector. The feature offset is used to extract sparse values (i.e. the key points in the above Values) from the value vector Values near the initial reference point. The Query Feature is obtained through the linear layer and the Softmax, and the attention weight AttentionWeights(Amqk) is obtained. The AttentionWeights(Amqk) and the sparse values are aggregated, for example, the values of the 3 key points (stacked 3 green blocks) extracted from the value vector of Head1 are weighted and summed through the 3 weights 0.5, 0.3 and 0.2 of Head1 in AttentionWeights(Amqk), to obtain the attention result corresponding to the attention head Head1. Similarly, the attention result corresponding to each attention head (Aggregated Sampled Values) can be obtained. The Aggregated Sampled Values are obtained through the linear layer to obtain the decoding result (Output). Alternatively, the Output can be added to the Query Feature through the residual connection, and the added result is taken as the decoding result. The specific setting can be made according to the actual demand. The specific principle of Deformable DETR will not be repeated here.

[0104] The present disclosure realizes the coordinate point level prediction of the instance through the target detection model based on Deformable DETR. Compared with the existing prediction using segmentation combined with post-processing or autoregressive prediction, the prediction accuracy can be greatly improved. Moreover, the target detection model based on Deformable DETR uses deformable convolution, and only collects the main feature points near the reference point when performing attention operation, greatly reducing the calculation amount, thereby effectively improving the prediction speed.

[0105] In an optional example, step 20214, which obtains the training decoding result based on the target training feature map, initial query features, initial reference points, and the decoder network in the target detection network, includes: for each decoder in the decoder network, obtaining the decoding result of the decoder based on the target training feature map and the input query features and input reference points corresponding to that decoder, wherein the input query features and input reference points corresponding to the first decoder are the initial query features and initial reference points, respectively, and the input query features corresponding to any other decoder other than the first decoder are the decoding results of the previous decoder of that other decoder, and the input reference points of that other decoder are the output reference points determined based on the decoding results of the previous decoder; and using the decoding result of the last decoder as the training decoding result.

[0106] In an optional example, after obtaining the decoding result of each decoder in the decoder network based on the target training feature map and the input query features and input reference points corresponding to that decoder, the method further includes: determining a first offset corresponding to the decoder based on the decoding result of the decoder and the offset prediction network corresponding to the decoder; determining an output reference point corresponding to the decoder based on the first offset and the input reference point corresponding to the decoder; and correspondingly, determining a training instance point set based on the training decoding result, including: using the output reference point corresponding to the last decoder determined based on the training decoding result as the training instance point set.

[0107] For example, Figure 9 This is a schematic diagram illustrating the principle of determining the training instance point set provided in another exemplary embodiment of this disclosure. Each decoder corresponds to an offset prediction network, which predicts the first offset of the reference point based on the decoding result of the decoder and adds it to the input reference point corresponding to the decoder. The input reference point of decoder 1 is the initial reference point, and the input reference point of decoder i (i = 2, 3, ..., N) is the output reference point of decoder i-1. By continuously fine-tuning the reference points through training, an accurate instance point set is obtained.

[0108] Figure 10 This is a flowchart illustrating step 202 provided in another exemplary embodiment of this disclosure.

[0109] In an optional example, the first label data also includes the type labels corresponding to each instance in the training input data; after obtaining the training decoding results based on the target training feature map, initial query features, initial reference points, and the decoder network in the target detection network in step 20214, the data further includes:

[0110] Step 20216: Based on the training decoding results, determine the training type results, which include the prediction type corresponding to each instance.

[0111] Wherein, the type label of the instance is the real type of each instance obtained by pre-labeling, such as lane line, road edge, zebra crossing, arrow, drivable area, etc. The predicted type corresponding to the instance refers to the element type to which the instance belongs predicted by the target detection network, and the element type can include lane line, road edge, zebra crossing, arrow, drivable area, etc. For example, it is predicted that a certain instance belongs to a lane line.

[0112] Exemplarily, Figure 11 is a schematic diagram of a prediction network for predicting types provided by an exemplary embodiment of the present disclosure. Wherein, the decoder N decodes to obtain a training decoding result, and the type prediction network is used to predict to obtain a training type result. The type prediction network can be a prediction network based on a feedforward neural network, which can be set according to actual needs. The type prediction network, each offset prediction network, and each reference point updating network can be collectively referred to as a prediction head network.

[0113] Step 20217, determining a type loss based on the training type result and the type label in the first label data.

[0114] Wherein, the type loss can be determined based on a preset type loss function, and the type loss function can adopt any implementable loss function, such as a focal loss function, which can be set according to actual needs.

[0115] Correspondingly, step 2024 adjusts the network parameters of the target detection network based on the first loss and the second loss until the first loss and the second loss meet the preset condition, and obtains the target detection model, including:

[0116] Step 20241, determining a comprehensive loss based on the first loss, the second loss, the type loss, and a preset weight.

[0117] Wherein, the preset weight can be set according to actual needs, such as the weights of the first loss l1, the second loss l2, and the type loss l3 can be set as λ1, λ2, and λ3 respectively, and the comprehensive loss can be represented as:

[0118] L = λ1*l1 + λ2*l2 + λ3*l3

[0119] Exemplarily, λ1, λ2, and λ3 can be set as 5, 0.1, and 2 respectively, which is not limited in detail.

[0120] Step 20242, adjusting the network parameters of the target detection network based on the comprehensive loss until the comprehensive loss meets the preset condition, and obtaining the target detection model.

[0121] The present disclosure further combines the type loss, the point-to-point loss, and the direction loss to comprehensively adjust the network parameters, thereby further improving the performance of the target detection model and the accuracy of the prediction result.

[0122] In an optional example, Figure 12 is a flowchart of step 2024 provided by an example embodiment of the present disclosure. In this example, the step 2024 adjusts the network parameters of the target detection network based on the first loss and the second loss until the first loss and the second loss meet the preset condition, and obtains the target detection model, including:

[0123] Step 20241a, based on the first loss and the second loss, determine the comprehensive loss.

[0124] Wherein, the first loss and the second loss can be weighted and summed according to a certain proportion weight to obtain the comprehensive loss, and the specific principle is referred to the foregoing content, which will not be repeated here.

[0125] Step 20242a, based on the comprehensive loss, adjust the network parameters of the target detection network until the comprehensive loss meets the preset condition, and obtain the target detection model.

[0126] The specific network parameter adjustment principle is referred to the foregoing content, which will not be repeated here.

[0127] The training method of the target detection model of the present disclosure, through the instance and the hierarchical prediction mode of the corresponding ordered point set, combines the point-to-point loss, the direction loss and the type loss for model training, so that the obtained target detection model can more accurately predict the ordered point set of the instance, greatly improve the prediction accuracy, and combine the deformable DETR network to make the attention operation of the target detection model in the inference process only focus on the feature interaction of the neighboring points around the reference point, greatly reduce the calculation complexity, thereby effectively reducing the calculation amount and improving the prediction efficiency. And the query vector of the target detection model of the present disclosure is point level, which is more flexible than the existing instance box level.

[0128] The above embodiments or optional examples of the present disclosure can be implemented alone or in any combination without conflict, which can be set according to actual needs, and the present disclosure is not limited.

[0129] Figure 13 is a flowchart of the map generation method provided by an example embodiment of the present disclosure. The present embodiment can be applied on an electronic device, such as a vehicle-mounted computing platform. As shown in Figure 13 , including the following steps:

[0130] Step 301, obtaining first image data and / or first point cloud data of at least one view angle.

[0131] The first image data can be image data of a current frame collected in real time by at least one camera arranged on the vehicle during driving of the vehicle, and the first point cloud data can be point cloud data of a current frame collected in real time by a radar arranged on the vehicle during driving of the vehicle.

[0132] In step 302, based on the first image data and / or the first point cloud data, a target detection model obtained through pre-training is used to obtain a target instance ordered point set.

[0133] The target detection model is obtained through the training method of the target detection model provided in any one of the above embodiments or optional examples, and the target instance ordered point set includes a first number of instance corresponding ordered point sets, and each ordered point set includes a target number of coordinate points in a first coordinate system.

[0134] The specific input data required by the target detection model can be set and obtained through training according to actual needs, and can support image data or point cloud data, or simultaneously support image data and point cloud data, for specific reference to the foregoing embodiments. The specific inference principle of the target detection model is described in the foregoing embodiments, and will not be described here.

[0135] In step 303, a map is generated based on the target instance ordered point set.

[0136] The target instance ordered point set is a coordinate point set in a first coordinate system (such as a coordinate system corresponding to a bird's-eye view), and a corresponding map element, such as a lane line, a zebra crossing, or a road edge, can be obtained by fitting the ordered point set of each instance in the target instance point set. The fitting result of each instance can be used as a local road map around the current position of the vehicle.

[0137] In actual applications, the target instance ordered point set can also be converted to a global coordinate system (such as a world coordinate system) through coordinate conversion, so that a global road map can be generated according to a region growing method, which can be set according to actual needs.

[0138] The map generation method of the present disclosure realizes prediction of coordinate points of map instances based on a target detection model, which can greatly improve the accuracy of the map compared with the existing frame-level prediction of map instances.

[0139] Any method (including the training method of the target detection model and the map generation method) provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capability, including but not limited to: terminal devices and servers, etc. Alternatively, any method provided in the embodiments of the present disclosure can be executed by a processor, such as a processor executing any method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. The following will not be described here.

[0140] Example Device

[0141] Figure 14 is a structural schematic diagram of a device for training a target detection model provided by an example embodiment of the present disclosure. The device of this embodiment can be used to implement the training method embodiments of the corresponding target detection model of the present disclosure, as shown in Figure 14 The device shown includes a first acquisition module 501 and a first processing module 502.

[0142] The first acquisition module 501 is configured to acquire training input data and corresponding first label data, the training input data including training image data and / or training point cloud data, and the first label data including ordered point sets corresponding to a first number of instances in the training input data, each ordered point set including a target number of coordinate points in a first coordinate system. The first processing module 502 is configured to train a pre-established target detection network based on the training input data, the first label data, a point-to-point loss function, and a direction loss function acquired by the first acquisition module 501, to obtain a target detection model. The point-to-point loss function is used to determine a point-to-point loss of training instance point sets output by the target detection network relative to the ordered point sets of the instances in the first label data. The direction loss function is used to determine a loss of directions between points in the training instance point sets relative to the directions between points in the ordered point sets of the instances in the first label data.

[0143] In an optional example, Figure 15 is a structural schematic diagram of the first processing module 502 provided by an example embodiment of the present disclosure. In this example, the first processing module 502 includes a first processing unit 5021, a second processing unit 5022, a third processing unit 5023, and a fourth processing unit 5024.

[0144] The first processing unit 5021 is configured to obtain training instance point sets based on the training input data and the target detection network. The second processing unit 5022 is configured to determine a first loss based on the training instance point sets obtained by the first processing unit 5021, the first label data, and the point-to-point loss function. The third processing unit 5023 is configured to determine a second loss based on the training instance point sets obtained by the first processing unit 5021, the first label data, and the direction loss function. The fourth processing unit 5024 is configured to adjust network parameters of the target detection network based on the first loss and the second loss until the first loss and the second loss meet a preset condition, to obtain the target detection model.

[0145] In an optional example, Figure 16is a structural schematic diagram of the second processing unit 5022 provided by an exemplary embodiment of the present disclosure. In the present example, the second processing unit 5022 includes a first determination sub-unit 50221, a second determination sub-unit 50222, a third determination sub-unit 50223, a fourth determination sub-unit 50224, and a fifth determination sub-unit 50225.

[0146] The first determination sub-unit 50221 is configured to, for each instance, determine a point-to-point relationship between each point in the ordered point set corresponding to the instance in the first label data and the point of the instance in the training instance point set in different orders of the ordered point set respectively, to obtain a point-to-point relationship corresponding to each order respectively; the second determination sub-unit 50222 is configured to determine a point-to-point loss corresponding to each order respectively based on the point-to-point relationship corresponding to each order respectively; the third determination sub-unit 50223 is configured to determine an order with the minimum point-to-point loss as a target order of the instance; the fourth determination sub-unit 50224 is configured to determine a point-to-point loss corresponding to the target order as a target point-to-point loss of the instance; and the fifth determination sub-unit 50225 is configured to determine the first loss based on the target point-to-point loss of each instance.

[0147] In an optional example, the third processing unit 5023 is specifically configured to determine the second loss based on the training instance point set, the first label data, the target order corresponding to each instance respectively, and a direction loss function.

[0148] Figure 17 is a structural schematic diagram of the first processing unit 5021 provided by an exemplary embodiment of the present disclosure.

[0149] In an optional example, the training input data further includes initial query features and an initial reference point, the initial query features include a target number of initial features corresponding to a first number of instances respectively, and the initial reference point includes a reference coordinate point corresponding to each initial feature respectively; the target detection network is a detection network based on a deformable detection transformer; and the first processing unit 5021 includes a first feature extraction sub-unit 50211, a second feature extraction sub-unit 50212, an encoding sub-unit 50213, a decoding sub-unit 50214, and a first processing sub-unit 50215.

[0150] The first feature extraction subunit 50211 is configured to perform feature extraction on the training image data based on a first feature extraction network in the target detection network to obtain first training image features; the second feature extraction subunit 50212 is configured to perform feature extraction on the training point cloud data based on a second feature extraction network in the target detection network to obtain first training point cloud features; the encoding subunit 50213 is configured to encode the first training image features and / or the first training point cloud features based on an encoder network in the target detection network to obtain a target training feature map in a first coordinate system; the decoding subunit 50214 is configured to obtain a training decoding result based on the target training feature map, an initial query feature, an initial reference point, and a decoder network in the target detection network, the decoder network including at least one decoder; and the first processing subunit 50215 is configured to determine a training instance point set based on the training decoding result.

[0151] In an optional example, the decoding subunit 50214 is specifically configured to: for each decoder in the decoder network, obtain a decoding result of the decoder based on the target training feature map and input query features and input reference points corresponding to the decoder, wherein the input query features and the input reference points corresponding to a first decoder are the initial query feature and the initial reference point respectively, the input query features corresponding to any other decoder except the first decoder are decoding results of previous decoders of the other decoder, and the input reference points of the other decoder are output reference points determined based on the decoding results of the previous decoders; and take the decoding result of a last decoder as the training decoding result.

[0152] In an optional example, the first processing unit 5021 further includes an offset prediction subunit 50216 and a second processing subunit 50217.

[0153] The offset prediction subunit 50216 is configured to determine a first offset corresponding to the decoder based on the decoding result of the decoder and an offset prediction network corresponding to the decoder; the second processing subunit 50217 is configured to determine an output reference point corresponding to the decoder based on the first offset and the input reference point corresponding to the decoder; and correspondingly, the first processing subunit 50215 is specifically configured to take the output reference point corresponding to a last decoder determined based on the training decoding result as the training instance point set.

[0154] Figure 18 FIG. 2 is a structural schematic diagram of the first processing module 502 according to another example embodiment of the present disclosure.

[0155] In an optional example, the first label data further includes type labels corresponding to respective instances in the training input data; and the first processing module 502 further includes:

[0156] The fifth processing unit 5025 is configured to determine a training type result based on the training decoding result, the training type result including a prediction type corresponding to each instance respectively; the sixth processing unit 5026 is configured to determine a type loss based on the training type result and the type label in the first label data; correspondingly, the fourth processing unit 5024 includes: a third processing subunit 50241 configured to determine a comprehensive loss based on the first loss, the second loss, the type loss, and a preset weight; and a fourth processing subunit 50242 configured to adjust the network parameters of the target detection network based on the comprehensive loss until the comprehensive loss meets a preset condition, and obtain the target detection model.

[0157] In an optional example, the fourth processing unit 5024 is specifically configured to: determine a comprehensive loss based on the first loss and the second loss; and adjust the network parameters of the target detection network based on the comprehensive loss until the comprehensive loss meets a preset condition, and obtain the target detection model.

[0158] Figure 19 is a structural schematic diagram of a map generation apparatus provided by an example embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding map generation method embodiments of the present disclosure, such as the apparatus shown in Figure 19 The apparatus includes a second acquisition module 601, a second processing module 602, and a third processing module 603.

[0159] The second acquisition module 601 is configured to acquire first image data and / or first point cloud data of at least one view angle; the second processing module 602 is configured to obtain a target instance ordered point set based on the first image data and / or the first point cloud data acquired by the second acquisition module 601, and using a target detection model obtained through pre-training, the target detection model being obtained through the training method of the target detection model of any one of the above embodiments or the optional example, the target instance ordered point set including ordered point sets corresponding to a first number of instances respectively, and each ordered point set including a target number of coordinate points in a first coordinate system; and the third processing module 603 is configured to generate a map based on the target instance ordered point set obtained by the second processing module 602.

[0160] Example Electronic Device

[0161] The present disclosure also provides an electronic device, which includes a memory configured to store a computer program;

[0162] A processor is configured to execute the computer program stored in the memory, and when the computer program is executed, the training method of the target detection model according to any one of the above embodiments of the present disclosure is implemented.

[0163] The present disclosure also provides an electronic device, which includes a memory configured to store a computer program;

[0164] a processor configured to execute a computer program stored in the memory, and the computer program, when executed, implements the method for generating a map according to any one of the embodiments of the present disclosure.

[0165] Figure 20 is a structural schematic diagram of an application embodiment of the electronic device of the present disclosure. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0166] The processor 11 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 10 to perform desired functions.

[0167] The memory 12 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 11 can execute the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, and the like can also be stored in the computer-readable storage media.

[0168] In one example, the electronic device 10 can further include an input device 13 and an output device 14, which are interconnected by a bus system and / or other forms of connection mechanism (not shown).

[0169] For example, the input device 13 can be the microphone or microphone array described above, for capturing input signals of a sound source.

[0170] In addition, the input device 13 can also include, for example, a keyboard, a mouse, and the like.

[0171] The output device 14 can output various information to the outside, including determined distance information, direction information, and the like. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0172] Of course, in order to simplify, Figure 20 In FIG. 1, only some of the components in the electronic device 10 related to the present disclosure are shown, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 10 can also include any other appropriate components according to specific application circumstances.

[0173] Example Computer Program Product and Computer-Readable Storage Medium

[0174] In addition to the method and the device described above, an embodiment of the present disclosure can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of the present specification.

[0175] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the “C” language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

[0176] In addition, an embodiment of the present disclosure can also be a computer readable storage medium, which stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of the present specification.

[0177] The computer readable storage medium can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0178] The basic principles of the present disclosure are described above in combination with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present disclosure. In addition, the above-mentioned specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above-mentioned details do not limit the present disclosure to the must-use of the above-mentioned specific details.

[0179] The various embodiments described in this specification are intended to be illustrative only and in no way limit the scope of the application. One skilled in the art will readily recognize from the disclosure herein, possible variations of the embodiments described herein. For example, the principles and applications are useful in systems other than the systems of choice or the application of choice to accomplish measurements of the performance of various aspects. Therefore, the application as claimed is intended to be as broad as possible and will encompass all changes and modifications of the application herein disclosed and equivalents thereof within the scope of the application. Accordingly, the specification and drawings are to be regarded in an illustrative manner and results to be achieved are intended to be within the scope of the application.

[0180] The block diagrams of devices, apparatuses, equipment, systems referred to in the present disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, configurations shown in the block diagrams are required. As will be realized by one skilled in the art, the devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include," "contain," "have," etc., are to be construed as open-ended terms meaning "including, but not limited to," and are to be interpreted in the same manner. The words "or" and "and" as used herein are to be interpreted as the word "and / or" and are to be interpreted in the same manner. The word "such as" as used herein is to be interpreted as the phrase "such as but not limited to" and is to be interpreted in the same manner.

[0181] The methods and apparatuses of the present disclosure can be implemented in a number of ways. For example, the methods and apparatuses of the present disclosure can be implemented using software, hardware, firmware, or any combination of these. The order of any steps of the methods described above is merely exemplary and the steps of the methods of the present disclosure are not limited to the order described above unless otherwise specifically stated. Furthermore, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, which includes machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the methods according to the present disclosure.

[0182] It is also important to note that the devices, equipment and methods of the present disclosure can be embodied in a variety of ways. These are to be considered as equivalents of what is described herein.

[0183] The above description of the disclosed aspects is intended to be illustrative only and not limiting of the scope of the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0184] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although the various example aspects and embodiments have been described herein with regard to particular aspects and embodiments, those skilled in the art will recognize that certain modifications, changes, substitutions, additions and sub-combinations can be made without departing from the spirit of the disclosure.

Claims

1. A method for training an object detection model, comprising: Acquire training input data and corresponding first label data. The training input data includes training image data and / or training point cloud data. The first label data includes ordered point sets corresponding to a first number of instances in the training input data. The ordered point sets include a target number of coordinate points in a first coordinate system. Based on the training input data, the first label data, the point-to-point loss function, and the orientation loss function, a pre-established target detection network is trained to obtain a target detection model. The point-to-point loss function is used to determine the point-to-point loss of the training instance point set output by the target detection network relative to the ordered point set of instances in the first label data. The orientation loss function is used to determine the loss of the orientation between points in the training instance point set relative to the orientation between points in the ordered point set of instances in the first label data.

2. The method according to claim 1, wherein, The step of training a pre-established target detection network based on the training input data, the first label data, the point-to-point loss function, and the orientation loss function to obtain a target detection model includes: Based on the training input data and the target detection network, the training instance point set is obtained; Based on the training instance point set, the first label data, and the point-to-point loss function, a first loss is determined; Based on the training instance point set, the first label data, and the orientation loss function, a second loss is determined; Based on the first loss and the second loss, the network parameters of the target detection network are adjusted until the first loss and the second loss meet preset conditions to obtain the target detection model.

3. The method according to claim 2, wherein, The step of determining the first loss based on the training instance point set, the first label data, and the point-to-point loss function includes: For each instance, based on the ordered point set corresponding to the instance in the first label data, the correspondence between each point in the ordered point set and the points of the instance in the training instance point set is determined according to different orders of the ordered point set, so as to obtain the point-to-point relationship corresponding to each order. Based on the point-to-point relationships corresponding to each sequence, determine the point-to-point loss corresponding to each sequence; The order that minimizes point-to-point loss is taken as the target order for this instance; The point-to-point loss corresponding to the target sequence is used as the target point-to-point loss for this instance; The first loss is determined based on the target point-to-point loss of each of the aforementioned instances.

4. The method according to claim 3, wherein, The step of determining the second loss based on the training instance point set, the first label data, and the orientation loss function includes: The second loss is determined based on the training instance point set, the first label data, the target order corresponding to each instance, and the direction loss function.

5. The method according to claim 2, wherein, The training input data also includes initial query features and initial reference points. The initial query features include initial features corresponding to the target number of the first number of instances. The initial reference points include reference coordinate points corresponding to each of the initial features. The target detection network is a detection network based on a deformable detection transformer. The step of obtaining the training instance point set based on the training input data and the target detection network includes: Based on the first feature extraction network in the target detection network, feature extraction is performed on the training image data to obtain the first training image features; Based on the second feature extraction network in the target detection network, feature extraction is performed on the training point cloud data to obtain the first training point cloud features; The encoder network in the target detection network encodes the features of the first training image and / or the features of the first training point cloud to obtain the target training feature map in the first coordinate system. Based on the target training feature map, the initial query features, the initial reference point, and the decoder network in the target detection network, a training decoding result is obtained, wherein the decoder network includes at least one decoder; Based on the training decoding results, the training instance point set is determined.

6. The method according to claim 5, wherein, The step of obtaining training and decoding results based on the target training feature map, the initial query features, the initial reference point, and the decoder network in the target detection network includes: For each decoder in the decoder network, the decoding result of the decoder is obtained based on the target training feature map and the input query feature and input reference point corresponding to the decoder. The input query feature and input reference point corresponding to the first decoder are the initial query feature and the initial reference point, respectively. The input query feature corresponding to any other decoder other than the first decoder is the decoding result of the previous decoder of the other decoder, and the input reference point of the other decoder is the output reference point determined based on the decoding result of the previous decoder. The decoding result of the last decoder is used as the training decoding result.

7. The method according to claim 6, wherein, After obtaining the decoding result of each decoder in the decoder network based on the target training feature map and the corresponding input query features and input reference points, the method further includes: Based on the decoding result of the decoder and the offset prediction network corresponding to the decoder, the first offset corresponding to the decoder is determined; Based on the first offset and the input reference point corresponding to the decoder, the output reference point corresponding to the decoder is determined; Determining the training instance point set based on the training decoding results includes: The output reference point corresponding to the last decoder determined based on the training decoding result is used as the training instance point set.

8. The method according to claim 5, wherein, The first label data also includes type labels corresponding to each instance in the training input data; After obtaining the training and decoding results based on the target training feature map, the initial query features, the initial reference point, and the decoder network in the target detection network, the process further includes: Based on the training decoding results, the training type results are determined, including the prediction type corresponding to each instance. Based on the training type results and the type labels in the first label data, determine the type loss; The step of adjusting the network parameters of the target detection network based on the first loss and the second loss until the first loss and the second loss meet preset conditions to obtain the target detection model includes: Based on the first loss, the second loss, the type loss, and the preset weights, a comprehensive loss is determined; Based on the comprehensive loss, the network parameters of the target detection network are adjusted until the comprehensive loss meets the preset condition, thereby obtaining the target detection model.

9. The method according to claim 2, wherein, The step of adjusting the network parameters of the target detection network based on the first loss and the second loss until the first loss and the second loss meet preset conditions to obtain the target detection model includes: Based on the first loss and the second loss, determine the comprehensive loss; Based on the comprehensive loss, the network parameters of the target detection network are adjusted until the comprehensive loss meets the preset condition, thereby obtaining the target detection model.

10. A method for generating a map, comprising: Acquire first image data and / or first point cloud data from at least one viewpoint; Based on the first image data and / or the first point cloud data, a pre-trained target detection model is used to obtain an ordered set of target instances. The target detection model is obtained by the training method of the target detection model as described in any one of claims 1-9. The ordered set of target instances includes ordered set of points corresponding to a first number of instances. The ordered set of points includes a target number of coordinate points in a first coordinate system. A map is generated based on the ordered set of points of the target instance.

11. A training device for an object detection model, comprising: The first acquisition module is used to acquire training input data and corresponding first label data. The training input data includes training image data and / or training point cloud data. The first label data includes ordered point sets corresponding to a first number of instances in the training input data. The ordered point sets include a target number of coordinate points in a first coordinate system. The first processing module is used to train a pre-established target detection network based on the training input data, the first label data, the point-to-point loss function, and the orientation loss function to obtain a target detection model. The point-to-point loss function is used to determine the point-to-point loss of the training instance point set output by the target detection network relative to the ordered point set of instances in the first label data. The orientation loss function is used to determine the loss of the orientation between points in the training instance point set relative to the orientation between points in the ordered point set of instances in the first label data.

12. A map generating apparatus, comprising: The second acquisition module is used to acquire first image data and / or first point cloud data from at least one viewpoint; The second processing module is used to obtain an ordered set of target instances based on the first image data and / or the first point cloud data by using a pre-trained target detection model. The target detection model is obtained by training the target detection model as described in any one of claims 1-9. The ordered set of target instances includes ordered set of points corresponding to a first number of instances. The ordered set of points includes a target number of coordinate points in a first coordinate system. The third processing module is used to generate a map based on the ordered point set of the target instance.

13. A computer-readable storage medium storing a computer program for performing a training method for a target detection model according to any one of claims 1-9; or, the computer program for performing a map generation method according to claim 10.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the training method of the target detection model according to any one of claims 1-9.

15. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the map generation method of claim 10.

Images