Target detection model training method and device, target detection method and device

Abstract

The present disclosure provides a target detection model training method and device, and a target detection method and device, which can be applied to technical fields such as intelligent driving, autonomous driving, and environmental perception. The target detection model training method includes: using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images to train a first basic network to obtain a trained single-frame detection network; inputting at least one set of historical scene images into the single-frame detection network, outputting at least one set of bird's-eye view feature sample matrices associated with the at least one set of historical scene images, and storing the at least one set of bird's-eye view feature sample matrices in a database; and training a second basic network using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with the at least one set of historical scene images to obtain a trained multi-frame fusion network.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as intelligent driving, autonomous driving, and environmental perception, and specifically to a target detection model training method and device, a target detection method and device, equipment, media, and program products. Background Art

[0002] In the field of autonomous driving, accurate prediction of the position and pose of objects around the vehicle is required in various scenarios to facilitate decision-making on the vehicle side. 3D object detection algorithms based on video streams have been widely applied. For example, these algorithms can utilize a certain number of previous frames for multi-frame fusion prediction, effectively mining temporal features for more accurate detection results.

[0003] In implementing the concepts of this disclosure, the inventors discovered at least the following issues with related technologies: While a larger number of consecutive frames can more effectively mine temporal information and produce more accurate results, this also exponentially increases the algorithm's graphics memory, memory, and computational costs. Limited by existing computing hardware, model training cannot be completed using a very large number of consecutive frames, and detection models trained with fewer consecutive frames often perform poorly. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a target detection model training method and device, a target detection method and device, equipment, medium and program product.

[0005] One aspect of the present disclosure provides a method for training an object detection model, comprising:

[0006] training a first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images to obtain a trained single-frame detection network, wherein each set of historical scene images is continuously collected at predetermined time intervals, and the historical scene images are used to represent the historical driving environment of the target vehicle;

[0007] Inputting at least one set of historical scene images into a single-frame detection network, outputting at least one set of bird's-eye view feature sample matrices associated with the at least one set of historical scene images, and storing the at least one set of bird's-eye view feature sample matrices in a database;

[0008] A second basic network is trained using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each set of historical scene images. The trained single-frame detection network and the multi-frame fusion network constitute a target detection model.

[0009] According to an embodiment of the present disclosure, training the second basic network using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images includes:

[0010] Performing feature enhancement processing on at least one set of bird's-eye view feature sample matrices to obtain at least one set of feature enhanced sample data associated with at least one set of historical scene images;

[0011] Inputting the feature-enhanced sample data into the second basic network, and outputting the first predicted pose information of the target sample object in the target historical scene image;

[0012] The first pose error between the target historical pose information and the first predicted pose information is calculated until the first pose error is less than a first preset threshold, thereby obtaining a trained multi-frame fusion network.

[0013] According to an embodiment of the present disclosure, wherein:

[0014] The feature enhancement processing of the bird's-eye view feature sample matrix includes: performing a predetermined type of element transformation on multiple elements in the bird's-eye view feature sample matrix, and the predetermined type of element transformation includes one of the following: flipping the positions of multiple elements based on a predetermined coordinate axis, rotating the positions of multiple elements based on a predetermined center point, and scaling the values ​​of multiple elements according to a predetermined ratio.

[0015] According to an embodiment of the present disclosure, training the first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images includes:

[0016] Inputting at least one set of historical scene images into a first basic network, and outputting at least one set of bird's-eye view feature training matrices associated with the at least one set of historical scene images;

[0017] Inputting at least one set of bird's-eye view feature training matrices into a pose result output network, and outputting second predicted pose information of the target sample object in each set of historical scene images;

[0018] A second pose error between the second predicted pose information and the historical pose information is calculated until the second pose error is less than a second preset threshold, thereby obtaining a trained single-frame detection network.

[0019] According to an embodiment of the present disclosure, the first basic network includes a two-dimensional feature extraction layer and a three-dimensional feature conversion mapping layer;

[0020] Inputting at least one set of historical scene images into a first basic network and outputting at least one set of bird's-eye view feature training matrices associated with the at least one set of historical scene images comprises:

[0021] Input the historical scene image into the two-dimensional feature extraction layer, and output the two-dimensional image scene features of the historical scene image;

[0022] The two-dimensional image scene features of the historical scene image are input into the three-dimensional feature conversion mapping layer, and the bird's-eye view feature training matrix associated with the historical scene image is output.

[0023] According to an embodiment of the present disclosure, the historical posture information of the target sample object includes: the historical position coordinates of the center point of the target sample object, the three-dimensional size of the target sample object, and the historical direction angle of the target sample object, wherein the historical direction angle is used to characterize: the angle between the first historical driving direction of the target sample object and the second historical driving direction of the target vehicle.

[0024] Another aspect of the present disclosure provides a target detection method, comprising:

[0025] Input the target scene image into the single-frame detection network and output the target bird's-eye view feature matrix associated with the target scene image;

[0026] Acquiring a plurality of previous bird's-eye view feature matrices corresponding to a plurality of frames of previous scene images, wherein the plurality of frames of previous scene images are continuously acquired at predetermined time intervals, the acquisition time of the target scene image is later than the acquisition time of the plurality of frames of previous scene images, and the previous bird's-eye view feature matrices are obtained by processing the previous scene images using a single-frame detection network;

[0027] The target bird's-eye view feature matrix and multiple previous bird's-eye view feature matrices are input into a multi-frame fusion network to output target predicted pose information of the target object in the target scene image, wherein the single-frame detection network and the multi-frame fusion network are trained using the above-mentioned target detection model training method.

[0028] According to an embodiment of the present disclosure, the method further includes: storing the target bird's-eye view feature matrix in a database.

[0029] Another aspect of the present disclosure provides a target detection model training device, comprising:

[0030] a first training module, configured to train a first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images to obtain a trained single-frame detection network, wherein each set of historical scene images is continuously collected at predetermined time intervals, and the historical scene images are used to represent the historical driving environment of the target vehicle;

[0031] an output module, configured to input at least one set of historical scene images into a single-frame detection network, output at least one set of bird's-eye view feature sample matrices associated with the at least one set of historical scene images, and store the at least one set of bird's-eye view feature sample matrices in a database;

[0032] The second training module is used to train a second basic network using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each set of historical scene images. The trained single-frame detection network and multi-frame fusion network constitute a target detection model.

[0033] Another aspect of the present disclosure provides an object detection device, comprising:

[0034] A single-frame detection module is used to input the target scene image into the single-frame detection network and output a target bird's-eye view feature matrix associated with the target scene image;

[0035] an acquisition module, configured to acquire a plurality of previous bird's-eye view feature matrices corresponding to a plurality of frames of previous scene images, wherein the plurality of frames of previous scene images are acquired continuously at predetermined time intervals, the acquisition time of the target scene image is later than the acquisition time of the plurality of frames of previous scene images, and the previous bird's-eye view feature matrices are acquired by processing the previous scene images using a single-frame detection network;

[0036] The multi-frame fusion module is used to input the target bird's-eye view feature matrix and multiple previous bird's-eye view feature matrices into the multi-frame fusion network, and output the target predicted pose information of the target object in the target scene image, wherein the single-frame detection network and the multi-frame fusion network are trained using the above-mentioned target detection model training method.

[0037] Another aspect of the present disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned target detection model training method or target detection method.

[0038] Another aspect of the present disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned target detection model training method or target detection method.

[0039] Another aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned target detection model training method or target detection method when executed by a processor.

[0040] According to an embodiment of the present disclosure, through the above-mentioned two-stage multi-frame fusion training strategy, a single-frame detection network is obtained in the first stage of training, which is used to mine single-frame features, calculate and store bird's-eye view features; and a single-frame fusion network is obtained in the second stage of training, which is used to fuse and mine the bird's-eye view features of multi-frame inputs, obtain the final output result, and complete the training of the overall model. Due to the adoption of a staged training strategy, there is no need to introduce a multi-frame processing strategy in the first stage, which avoids the exponential increase in the model's computational load and video memory usage. After the single-frame detection network training is completed, forward reasoning is performed on the images of the training data set to output the bird's-eye view features corresponding to each frame of the image. The saved bird's-eye view features will be used as the input of the second-stage multi-frame fusion network to mine multi-frame temporal features. By adopting a strategy of isolating single-frame and multi-frame training, the backbone network calculation of single-frame forward reasoning can be omitted in the multi-frame training stage, reducing video memory and computational load. At the same time, during the multi-frame fusion network training phase, since there is no need to calculate the bird's-eye view features (the bird's-eye view features output in the first phase are pre-stored), a larger number of consecutive frames can be used for model training, so that more consecutive frames can be used in the second phase to improve model performance, and the trained model prediction effect is better. It can be seen that the training method of the embodiment of the present disclosure can reduce the amount of calculation, avoid excessive consumption of computing resources, and increase the number of consecutive frames used by the model, unleashing the advantages of the multi-frame fusion strategy, making it easier to achieve excellent model performance under limited computing hardware conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0042] Figure 1 Schematically illustrates an application scenario diagram of the target detection model training method, target detection method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0043] Figure 2 The following schematically shows a flow chart of a method for training a target detection model according to an embodiment of the present disclosure;

[0044] Figure 3 The following schematically shows a principle diagram of a target detection model training method according to an embodiment of the present disclosure;

[0045] Figure 4 The following schematically shows a flow chart of a target detection method according to an embodiment of the present disclosure;

[0046] Figure 5 The following schematically shows a principle diagram of a target detection method according to an embodiment of the present disclosure;

[0047] Figure 6The following schematically shows a structural block diagram of a target detection model training device according to an embodiment of the present disclosure;

[0048] Figure 7 The following schematically shows a structural block diagram of an object detection device according to an embodiment of the present disclosure;

[0049] Figure 8 A block diagram of an electronic device suitable for implementing a target detection model training method or a target detection method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0050] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0051] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0052] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0053] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0054] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of the data involved (for example, including but not limited to user personal information) all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.

[0055] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0056] The present invention provides a method for training a target detection model, including:

[0057] training a first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images to obtain a trained single-frame detection network, wherein each set of historical scene images is continuously collected at predetermined time intervals, and the historical scene images are used to represent the historical driving environment of the target vehicle;

[0058] Inputting at least one set of historical scene images into a single-frame detection network, outputting at least one set of bird's-eye view feature sample matrices associated with the at least one set of historical scene images, and storing the at least one set of bird's-eye view feature sample matrices in a database;

[0059] A second basic network is trained using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each set of historical scene images. The trained single-frame detection network and the multi-frame fusion network constitute a target detection model.

[0060] Figure 1 Schematically illustrates an application scenario diagram of the target detection model training method, target detection method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0061] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a vehicle 101 and a server 102. The vehicle 101 and the server 102 may communicate via a network, which may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0062] Vehicle 101 is equipped with an autonomous driving module for guiding the vehicle to achieve autonomous driving. Vehicle 101 is also equipped with an image acquisition device, such as a camera, for capturing images of the surrounding environment while vehicle 101 is driving. A source radar may also be installed to collect point cloud data of the surrounding environment of vehicle 101.

[0063] Server 102 can be a server that provides various services, for example, it can be a background management server, which can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0064] In the application scenario of the disclosed embodiment, vehicle 101 captures video stream scene images (multiple consecutive frames) of the surrounding environment during historical driving using the image acquisition device installed therein. These images can serve as sample data for subsequent model training. Model training can be performed on server 102. Server 102 acquires sample scene images and can train an object detection model based on the acquired video stream images by executing the model training method of the disclosed embodiment.

[0065] After the model is trained, it can be used to assist vehicle 101 in achieving autonomous driving. For example, during the autonomous driving process, vehicle 101 will capture video stream scene images (continuous frames) of the surrounding environment through the image acquisition device installed therein, and send the scene images to server 102. After receiving the scene images, server 102 can input the scene images into a trained target detection model based on user requests, output the pose information of the obstacle through the target detection model and return it to the autonomous driving module in vehicle 101, so that the autonomous driving module in vehicle 101 can assist vehicle 101 in achieving autonomous driving based on the pose information of the obstacles identified in the surrounding environment.

[0066] Figure 2 The flowchart of the target detection model training method according to an embodiment of the present disclosure is schematically shown.

[0067] like Figure 2 As shown, the target detection model training method of this embodiment includes operations S201 to S203.

[0068] In operation S201, a first basic network is trained using at least one group of historical scene images and historical pose information of target sample objects in each group of historical scene images to obtain a trained single-frame detection network, wherein each group of historical scene images is continuously collected at a predetermined time interval, and the historical scene images are used to characterize the historical driving environment of the target vehicle, and the historical driving environment of the target vehicle includes at least one sample object (such as a pedestrian, a bicycle, a motor vehicle, or other movable obstacles).

[0069] In operation S202, at least one set of historical scene images is input into a single-frame detection network, at least one set of bird's-eye view feature sample matrices associated with the at least one set of historical scene images is output, and the at least one set of bird's-eye view feature sample matrices is stored in a database.

[0070] In operation S203, a second basic network is trained using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each set of historical scene images. The trained single-frame detection network and the multi-frame fusion network constitute a target detection model. The target historical scene image can be the latest image captured in each set of historical scene images, i.e., the last image in the set. The target historical pose information of the target sample object in the target historical scene image can be used as a label for training the second basic network.

[0071] According to an embodiment of the present disclosure, the target vehicle may be the unmanned vehicle of current concern. During driving, the unmanned vehicle needs to obtain the posture information of objects in the surrounding environment, and realize automatic driving of the unmanned vehicle by combining the posture information of the objects with other perception information (such as traffic light status, road indication information, etc.).

[0072] The target vehicle may be equipped with an image acquisition device, such as a camera, for capturing images of the surrounding driving environment while the vehicle is in motion. The model trained using the training method described above in the embodiment of the present disclosure is used to detect targets in the driving environment images to obtain the object's position and posture information. It should be noted that the training method described above in the embodiment of the present disclosure is based on a video stream, that is, a method for processing multiple frames of scene images. That is, each group of historical scene images is continuously acquired by the image acquisition device at predetermined time intervals. Different groups of historical scene images may be associated with the same type of driving scene or different types of driving scenes. For example, the first group of historical scene images is 10 frames of images continuously acquired while the target vehicle was driving on a rural road on January 3, the second group of historical scene images is 10 frames of images continuously acquired while the target vehicle was driving on the same rural road on January 4, the third group of historical scene images is 10 frames of images continuously acquired while the target vehicle was driving on an urban road on January 4, and so on.

[0073] The historical pose information of the target sample objects in each set of historical scene images is used as a label for each set of historical scene images and is used to train the first basic network to obtain a single-frame detection network. The historical pose information of the target sample objects includes: the historical position coordinates (Location) of the target sample object's center point; the target sample object's three-dimensional dimensions (Dimension, such as the length, width, and height of the object's bounding box); and the target sample object's historical yaw angle (Yaw). The historical yaw angle is used to represent the angle between the target sample object's first historical driving direction and the target vehicle's second historical driving direction. These three sets of pose parameters can fully describe the pose information of a rigid target object in a 3D scene.

[0074] In the object detection model trained in the disclosed embodiments, during information detection, the model first extracts 2D features from the image signal. These 2D features are then mapped to a bird's-eye view (BEV) perspective through 3D projection and voxel pooling, yielding more intuitive BEV features. This model then utilizes a certain number of previous frames for multi-frame fusion prediction, effectively mining temporal features and achieving more accurate detection results. Specifically, by simultaneously mining features from a certain number of consecutive frames, a multi-frame BEV feature map is generated, which is then concatenated and passed through a subsequent network to yield the final output.

[0075] During model training, the greater the number of consecutive frames used, the more effectively the temporal information can be mined, and the better the prediction effect of the trained model. However, in conventional training strategies, in order to ensure the calculation of gradients during training, in the multi-frame fusion training stage, it is necessary to first calculate the BEV features of multiple previous frames (directly loading historical BEV features cannot reflect the gradient, and real-time inference calculation is required during training), and then perform fusion training with the current frame. Therefore, the amount of calculation is large. As the number of consecutive frames increases, the algorithm's video memory, memory, and computing costs also increase exponentially. Under the limitations of existing computing hardware, it is impossible to complete model training using a large number of consecutive frames, and the detection model trained with a smaller number of consecutive frames often performs poorly, limiting the performance of the multi-frame fusion 3D detection algorithm.

[0076] Based on this, in order to alleviate the limitation of computing resources, the embodiments of the present disclosure utilize a larger number of consecutive frames for model training and propose a two-stage multi-frame fusion training strategy: that is, a strategy of isolating single-frame training from multi-frame training.

[0077] Specifically, the first stage includes operations S201 and S202 described above. First, a single-frame detection network is trained using at least one set of historical scene images and their labels (historical pose information of the target sample objects). The trained network then outputs intermediate results: bird's-eye view feature sample matrix features (BEV features), which are stored in a database.

[0078] The second stage is the above-mentioned operation S203, which directly reads at least one set of bird's-eye view feature sample matrix features (BEV features) and data labels (target historical pose information of target sample objects in target historical scene images) from the database to train the multi-frame fusion network.

[0079] The trained single-frame detection network and multi-frame fusion network constitute the final target detection model, which is used to detect object pose information.

[0080] According to an embodiment of the present disclosure, through the above-mentioned two-stage multi-frame fusion training strategy, the first stage of training is performed in operations S201 and S202 to obtain a single-frame detection network for mining single-frame features and for calculating and storing BEV feature maps; and the second stage of training is performed through the above-mentioned operation S203 to obtain a single-frame fusion network for fusing and mining multi-frame input BEV features to obtain the final output result, thereby completing the training of the overall model.

[0081] Because a phased training strategy is employed, there's no need to introduce a multi-frame processing strategy in the first phase, avoiding a doubling of model computational overhead and memory usage. After the single-frame detection network is trained, forward inference is performed on the images in the training dataset, outputting the BEV features corresponding to each frame. These stored BEV features serve as input to the second-phase multi-frame fusion network for multi-frame temporal feature mining. By separating single-frame from multi-frame training, the backbone network computations for single-frame forward inference can be eliminated during the multi-frame training phase, reducing both memory usage and computational overhead. Furthermore, since BEV feature computation is not required during the multi-frame fusion network training phase (the BEV features output from the first phase are pre-stored), a larger number of consecutive frames can be used for model training. This allows for improved model performance and better predictions in the second phase using more consecutive frames. Thus, the training method of the disclosed embodiments reduces computational overhead, avoids excessive consumption of computing resources, and increases the number of consecutive frames used by the model, unleashing the advantages of a multi-frame fusion strategy and making it easier to achieve excellent model performance even with limited computing hardware.

[0082] According to an embodiment of the present disclosure, specifically, in the second stage, training the second basic network using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images includes the following operations 11 to 13.

[0083] Operation 11: Perform feature enhancement processing on at least one set of bird's-eye view feature sample matrices to obtain at least one set of feature-enhanced sample data associated with at least one set of historical scene images.

[0084] Operation 12: input the feature-enhanced sample data into the second basic network, and output the first predicted pose information of the target sample object in the target historical scene image.

[0085] Operation 13, calculate the first pose error between the target historical pose information and the first predicted pose information, and tune the network parameters based on the first pose error, and iterate until the first pose error is less than a first preset threshold, thereby obtaining a trained multi-frame fusion network.

[0086] According to an embodiment of the present disclosure, the second basic network is trained by directly reading the BEV features output from the first stage in the second stage. Since BEV features have a higher degree of information abstraction than the original input image, directly extracting features from them often leads to overfitting, resulting in reduced model performance. To this end, the embodiment of the present disclosure proposes a supporting training strategy that performs BEV perspective data enhancement on multi-frame input BEV features to alleviate overfitting and improve model performance. Specifically, feature enhancement processing is first performed on the bird's-eye view feature sample matrix to obtain feature-enhanced sample data, which is then used to train the second basic network.

[0087] According to the embodiments of the present disclosure, by proposing a matching model training strategy and introducing a data enhancement strategy, model training can converge quickly and obtain stronger generalization capabilities, solve the overfitting problem, and achieve the best model training effect under existing computing resources.

[0088] According to an embodiment of the present disclosure, further, performing BEV feature enhancement processing on the bird's-eye view feature sample matrix may include: performing a predetermined type of element transformation on multiple elements in the bird's-eye view feature sample matrix.

[0089] Specifically, the predetermined type of element transformation may include:

[0090] The positions of multiple elements are flipped based on the predetermined coordinate axis, for example, the BEV view is probabilistically flipped along the X and Y axes respectively.

[0091] Rotate multiple elements around a predetermined center point, such as the probabilistic random rotation of BEV views.

[0092] Scaling the values ​​of multiple elements according to a predetermined ratio, such as probabilistic random scaling of BEV views.

[0093] Among them, each enhancement strategy (predetermined type of element transformation) needs to implement corresponding geometric transformation of the target elements associated with the target object, which is reflected in the change of the label value of the target element to improve the generalization ability of the model.

[0094] Figure 3 The schematic diagram shows a principle diagram of the target detection model training method according to an embodiment of the present disclosure.

[0095] The following combination Figure 3 , an exemplary description of the target detection model training method of the embodiment of the present disclosure is given.

[0096] like Figure 3As shown, in the first stage of model training, the first basic network is trained using historical scene images and historical pose information (sample images + labels) of target sample objects in historical scene images.

[0097] According to the embodiments of the present disclosure, in the first phase of training, a pose result output network is added to complete model training. It should be noted that due to the phased and isolated training strategy adopted in the embodiments of the present disclosure, the pose result output network is required to assist in completing the training process in the first phase. After model training is completed, the pose result output network is no longer required during the test application process.

[0098] The first stage of training specifically includes the following operations 21 to 23:

[0099] Operation 21: input at least one set of historical scene images into a first basic network, and output at least one set of bird's-eye view feature training matrices associated with the at least one set of historical scene images.

[0100] like Figure 3 As shown, the first basic network includes a two-dimensional feature extraction layer and a three-dimensional feature conversion mapping layer. Based on this network structure, the bird's-eye view feature training matrix output by the first basic network includes:

[0101] First, the historical scene image is input into the two-dimensional feature extraction layer, and the two-dimensional feature extraction layer extracts the two-dimensional features of the image and outputs the two-dimensional image scene features of the historical scene image;

[0102] The 2D scene features of the historical scene images are then input into a 3D feature conversion mapping layer, which then outputs a bird's-eye view feature training matrix associated with the historical scene images. Specifically, the 2D scene features are first converted to 3D features, transforming the extracted 2D image features into 3D camera cone spatial features. The 3D scene features are then mapped to a bird's-eye view (BEV) perspective. Voxel pooling is used to construct BEV spatial features from the camera cone features and the mapped 3D camera cone space, resulting in more intuitive BEV features.

[0103] Operation 22: inputting at least one set of bird's-eye view feature training matrices into a pose result output network, and outputting second predicted pose information of the target sample objects in each set of historical scene images.

[0104] Operation 23 calculates the second pose error between the second predicted pose information and the historical pose information, and tunes the network parameters based on the second pose error. This is repeated until the second pose error is less than a second preset threshold, thereby obtaining a trained single-frame detection network.

[0105] like Figure 3As shown in the figure, after completing the first stage of training, multiple single-frame sample images in the training samples are input into the trained single-frame detection network in turn, and the single-frame detection network is used to output the intermediate results: bird's-eye view feature sample matrix features (multiple BEV view features corresponding to multiple single-frame images one by one), and stored in the database.

[0106] like Figure 3 As shown, during the second stage of training, at least one set of bird's-eye view feature matrix samples (multiple single-frame BEV features) is directly read from the database and combined with data labels (the target historical pose information of the target sample object in the target historical scene image) to train the multi-frame fusion network. During training, a strategy for feature enhancement processing of the BEV features is adopted. The specific training methods for this stage can be found in the description of the second stage training methods in the previous embodiment and will not be repeated here.

[0107] Another aspect of the present disclosure provides a target detection method, Figure 4 The flowchart of the target detection method according to the embodiment of the present disclosure is schematically shown. Figure 5 The principle diagram of the target detection method according to the embodiment of the present disclosure is schematically shown. Figure 4 、 Figure 5 The method of the embodiment of the present disclosure is described. The single-frame detection network and the multi-frame fusion network used in the target detection method are trained using the target detection model training method of the aforementioned embodiment.

[0108] like Figure 4 As shown, the target detection method of this embodiment includes operations S401 to S403.

[0109] In operation S401, a target scene image is input into a single-frame detection network, and a target bird's-eye view feature matrix associated with the target scene image is output;

[0110] In operation S402, a plurality of previous bird's-eye view feature matrices corresponding to a plurality of frames of previous scene images are obtained, wherein the plurality of frames of previous scene images are continuously acquired at predetermined time intervals, the acquisition time of the target scene image is later than the acquisition time of the plurality of frames of previous scene images, and the previous bird's-eye view feature matrices are obtained by processing the previous scene images using a single-frame detection network;

[0111] In operation S403, the target bird's-eye view feature matrix and a plurality of previous bird's-eye view feature matrices are input into a multi-frame fusion network, and target predicted pose information of the target object in the target scene image is output.

[0112] After obtaining the target posture information of the target object, it can be returned to the automatic driving module in the target vehicle, so that the automatic driving module in the vehicle can realize the automatic driving of the unmanned vehicle through the posture information of the object combined with other perception information (such as traffic light status, road indication information, etc.).

[0113] According to the embodiments of the present disclosure, the above-mentioned object detection method is a processing strategy based on video streams (multi-frame images). The prediction results of each current frame are integrated with the prediction information of multiple previous frames, and the multi-frame temporal features are mined to obtain more accurate prediction results.

[0114] This target detection method is also divided into two prediction stages. The first stage is used to extract single-frame image information, and the second stage is used to fuse multi-frame features.

[0115] like Figure 5 As shown, in the first stage, for each current frame, operation S401 is performed, and a target bird's-eye view feature matrix associated with the target scene image is output through a single-frame detection network. Specifically, 2D features are first extracted from the image signal, and then the 2D features are mapped to a bird's-eye view (BEV) through 3D projection and voxel pooling operations to obtain a more intuitive BEV feature of the current frame.

[0116] like Figure 5 As shown, in the second stage, before performing feature fusion processing, first, through operation S402, multiple previous bird's-eye view feature matrices (previous frame BEV features) corresponding to multiple frames of previous scene images are obtained, and then, operation S403 is performed to fuse the current frame BEV features output in the first stage and the multiple previous frame BEV features obtained through a multi-frame fusion network to better mine the temporal features and output the target predicted pose information of the target object in the target scene image.

[0117] Among them, in the first stage, after obtaining the target bird's-eye view feature matrix (BEV feature) for each current frame, the BEV feature can be stored in the database for easy call in subsequent processing. Since the BEV features of each current frame are stored in the database in a timely manner, a large number of BEV feature results of previous frames are stored in the database over time. In the application stage of the model, since the model has been fully trained, there is no need to consider the requirements of calculating gradients. In the fusion processing stage of subsequent frames, directly calling the BEV feature results of the previous frame can also obtain better prediction results. In this way, the process of performing inference calculations for each previous frame is omitted, the consumption of computer resources is reduced, and the prediction results are quickly obtained. In scenarios with high timeliness requirements such as autonomous driving, this method has better practicality.

[0118] Based on the above target detection model training method, the present disclosure also provides a target detection model training device. Figure 6 The device is described in detail.

[0119] Figure 6 The structural block diagram of the target detection model training device according to an embodiment of the present disclosure is schematically shown.

[0120] like Figure 6 As shown, the target detection model training device 600 of this embodiment includes a first training module 601, an output module 602, and a second training module 603.

[0121] Among them, the first training module 601 is used to train the first basic network using at least one group of historical scene images and the historical pose information of the target sample objects in each group of historical scene images to obtain a trained single-frame detection network, wherein each group of historical scene images is continuously collected at a predetermined time interval, and the historical scene images are used to characterize the historical driving environment of the target vehicle.

[0122] The output module 602 is used to input at least one set of historical scene images into the single-frame detection network, output at least one set of bird's-eye view feature sample matrices associated with the at least one set of historical scene images, and store the at least one set of bird's-eye view feature sample matrices in the database.

[0123] The second training module 603 is used to train a second basic network using at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each set of historical scene images. The trained single-frame detection network and multi-frame fusion network constitute a target detection model.

[0124] According to an embodiment of the present disclosure, a two-stage multi-frame fusion training strategy can be implemented through the above-mentioned target detection model training device 600. The first stage of training is performed by the first training module 601 and the output module 602, and a single-frame detection network is obtained through training, which is used for mining single-frame features and for calculating and storing BEV feature maps; and the second stage of training is performed by the second training module 603, and a single-frame fusion network is obtained through training, which is used for fusing and mining multi-frame input BEV features to obtain the final output result, thereby completing the training of the overall model.

[0125] Due to the adoption of a phased training strategy, there is no need to introduce a multi-frame processing strategy during the first phase of execution through the first training module 601, thus avoiding an exponential increase in the model's computational load and video memory usage. After the single-frame detection network training is completed, the output module 602 performs forward reasoning on the images in the training data set, outputting the BEV features corresponding to each frame of the image. The saved BEV features are used as input to the second-phase multi-frame fusion network, and multi-frame temporal feature mining is performed through the second training module 603. By adopting a strategy of isolating single-frame from multi-frame training, the backbone network calculations for single-frame forward reasoning can be omitted during the multi-frame training phase, reducing video memory and computational load. At the same time, in the multi-frame fusion network training phase executed in the second training module 603, since there is no need to calculate the BEV features (the BEV features output in the first phase are pre-stored), a larger number of continuous frames can be used for model training, so that more continuous frames can be used in the second phase to improve the model performance. The trained model has better prediction effect, which can reduce the amount of calculation, avoid excessive consumption of computing resources, and increase the number of continuous frames used by the model, releasing the advantages of the multi-frame fusion strategy, and making it easier to achieve excellent model performance under limited computing hardware conditions.

[0126] According to an embodiment of the present disclosure, the second training module 603 includes a feature enhancement unit, a first output unit, and a first calculation unit.

[0127] Among them, the feature enhancement unit is used to perform feature enhancement processing on at least one group of bird's-eye view feature sample matrices to obtain at least one group of feature enhanced sample data associated with at least one group of historical scene images; the first output unit is used to input the feature enhanced sample data into the second basic network and output the first predicted pose information of the target sample object in the target historical scene image; the first calculation unit is used to calculate the first pose error between the target historical pose information and the first predicted pose information until the first pose error is less than the first preset threshold, thereby obtaining a trained multi-frame fusion network.

[0128] According to an embodiment of the present disclosure, the feature enhancement unit includes a feature enhancement sub-unit, which is used to perform a predetermined type of element transformation on multiple elements in the bird's-eye view feature sample matrix. The predetermined type of element transformation includes one of the following: flipping the position of multiple elements based on a predetermined coordinate axis, rotating the position of multiple elements based on a predetermined center point, and scaling the values ​​of multiple elements according to a predetermined ratio.

[0129] According to an embodiment of the present disclosure, the first training module 601 includes a second output unit, a third output unit, and a second calculation unit.

[0130] Among them, the second output unit is used to input at least one group of historical scene images into the first basic network, and output at least one group of bird's-eye view feature training matrices associated with at least one group of historical scene images; the third output unit is used to input at least one group of bird's-eye view feature training matrices into the pose result output network, and output the second predicted pose information of the target sample object in each group of historical scene images; the second calculation unit is used to calculate the second pose error between the second predicted pose information and the historical pose information until the second pose error is less than the second preset threshold, thereby obtaining a trained single-frame detection network.

[0131] According to an embodiment of the present disclosure, the first basic network includes a two-dimensional feature extraction layer and a three-dimensional feature conversion mapping layer.

[0132] The second output unit includes a two-dimensional feature extraction subunit and a feature conversion mapping subunit.

[0133] Among them, the two-dimensional feature extraction subunit is used to input the historical scene image into the two-dimensional feature extraction layer and output the two-dimensional image scene features of the historical scene image; the feature conversion mapping subunit is used to input the two-dimensional image scene features of the historical scene image into the three-dimensional feature conversion mapping layer and output the bird's-eye view feature training matrix associated with the historical scene image.

[0134] According to an embodiment of the present disclosure, the historical posture information of the target sample object includes: the historical position coordinates of the center point of the target sample object, the three-dimensional size of the target sample object, and the historical direction angle of the target sample object, wherein the historical direction angle is used to characterize: the angle between the first historical driving direction of the target sample object and the second historical driving direction of the target vehicle.

[0135] Figure 7 The following schematically shows a structural block diagram of an object detection device according to an embodiment of the present disclosure.

[0136] like Figure 7 As shown, the target detection device 700 of this embodiment includes a single-frame detection module 701 , an acquisition module 702 , and a multi-frame fusion module 703 .

[0137] The single-frame detection module 701 is configured to input a target scene image into a single-frame detection network and output a target bird's-eye view feature matrix associated with the target scene image.

[0138] Acquisition module 702 is used to acquire multiple previous bird's-eye view feature matrices corresponding to multiple frames of previous scene images, wherein the multiple frames of previous scene images are continuously acquired at predetermined time intervals, the acquisition time of the target scene image is later than the acquisition time of the multiple frames of previous scene images, and the previous bird's-eye view feature matrix is ​​obtained by processing the previous scene images through a single-frame detection network.

[0139] The multi-frame fusion module 703 is used to input the target bird's-eye view feature matrix and multiple previous bird's-eye view feature matrices into the multi-frame fusion network, and output the target predicted pose information of the target object in the target scene image, wherein the single-frame detection network and the multi-frame fusion network are trained using the above-mentioned target detection model training method.

[0140] According to an embodiment of the present disclosure, the above-mentioned device further includes a storage module for storing the target bird's-eye view feature matrix in a database.

[0141] According to an embodiment of the present disclosure, any multiple modules of the first training module 601, the output module 602, the second training module 603, or the single-frame detection module 701, the acquisition module 702, and the multi-frame fusion module 70 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first training module 601, the output module 602, the second training module 603, or the single-frame detection module 701, the acquisition module 702, and the multi-frame fusion module 70 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, the first training module 601, the output module 602, the second training module 603, or at least one of the single-frame detection module 701, the acquisition module 702, and the multi-frame fusion module 70 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0142] Figure 8 A block diagram of an electronic device suitable for implementing a target detection model training method or a target detection method according to an embodiment of the present disclosure is schematically shown.

[0143] like Figure 8As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0144] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0145] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage portion 808 including a hard disk; and a communication portion 809 including a network interface card such as a LAN card or a modem. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 810 as needed, so that a computer program read therefrom can be installed into the storage portion 808 as needed.

[0146] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0147] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0148] Embodiments of the present disclosure also include a computer program product, comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the object detection model training method or object detection method provided in the embodiments of the present disclosure.

[0149] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0150] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0151] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the processor 801, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0152] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0155] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for training a target detection model, comprising: Training a first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images to obtain a trained single-frame detection network, wherein each set of historical scene images is continuously acquired at predetermined time intervals, and the historical scene images are used to represent the historical driving environment of the target vehicle; Inputting at least one set of the historical scene images into the single-frame detection network, outputting at least one set of bird's-eye view feature sample matrices associated with the at least one set of the historical scene images, and storing the at least one set of bird's-eye view feature sample matrices in a database; A second basic network is trained using the at least one group of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one group of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each group of historical scene images, and the trained single-frame detection network and the multi-frame fusion network constitute a target detection model.

2. The method according to claim 1, wherein Training a second basic network using the at least one set of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one set of historical scene images includes: performing feature enhancement processing on the at least one set of bird's-eye view feature sample matrices to obtain at least one set of feature enhanced sample data associated with the at least one set of historical scene images; Inputting the feature-enhanced sample data into a second basic network, and outputting first predicted pose information of a target sample object in the target historical scene image; A first pose error between the target historical pose information and the first predicted pose information is calculated until the first pose error is less than a first preset threshold, thereby obtaining a trained multi-frame fusion network.

3. The method according to claim 2, wherein: The feature enhancement processing of the bird's-eye view feature sample matrix includes: performing a predetermined type of element transformation on multiple elements in the bird's-eye view feature sample matrix, and the predetermined type of element transformation includes one of the following: flipping the positions of the multiple elements based on a predetermined coordinate axis, rotating the positions of the multiple elements based on a predetermined center point, and scaling the values ​​of the multiple elements according to a predetermined ratio.

4. The method according to claim 1, wherein Training the first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of historical scene images includes: Inputting the at least one set of historical scene images into the first basic network, and outputting at least one set of bird's-eye view feature training matrices associated with the at least one set of historical scene images; Inputting the at least one set of bird's-eye view feature training matrices into a pose result output network, and outputting second predicted pose information of the target sample object in each set of the historical scene images; A second pose error between the second predicted pose information and the historical pose information is calculated until the second pose error is less than a second preset threshold, thereby obtaining a trained single-frame detection network.

5. The method according to claim 4, wherein The first basic network includes a two-dimensional feature extraction layer and a three-dimensional feature conversion mapping layer; Inputting the at least one set of historical scene images into the first basic network and outputting at least one set of bird's-eye view feature training matrices associated with the at least one set of historical scene images comprises: Inputting the historical scene image into the two-dimensional feature extraction layer, and outputting the two-dimensional image scene features of the historical scene image; The two-dimensional image scene features of the historical scene image are input into the three-dimensional feature conversion mapping layer, and a bird's-eye view feature training matrix associated with the historical scene image is output.

6. The method according to claim 1, wherein The historical pose information of the target sample object includes: the historical position coordinates of the center point of the target sample object, the three-dimensional size of the target sample object, and the historical direction angle of the target sample object, wherein the historical direction angle is used to represent: the angle between the first historical driving direction of the target sample object and the second historical driving direction of the target vehicle.

7. A target detection method, comprising: Inputting a target scene image into a single-frame detection network, and outputting a target bird's-eye view feature matrix associated with the target scene image; Acquiring a plurality of previous bird's-eye view feature matrices corresponding to a plurality of frames of previous scene images, wherein the plurality of frames of previous scene images are acquired continuously at predetermined time intervals, the acquisition time of the target scene image is later than the acquisition time of the plurality of frames of previous scene images, and the previous bird's-eye view feature matrices are acquired by processing the previous scene images using the single-frame detection network; The target bird's-eye view feature matrix and the multiple previous bird's-eye view feature matrices are input into a multi-frame fusion network, and target predicted pose information of the target object in the target scene image is output, wherein the single-frame detection network and the multi-frame fusion network are trained using the method described in any one of claims 1 to 6.

8. The method according to claim 7, further comprising: The target bird's-eye view feature matrix is ​​stored in a database.

9. A target detection model training device comprising: a first training module, configured to train a first basic network using at least one set of historical scene images and historical pose information of target sample objects in each set of the historical scene images, to obtain a trained single-frame detection network, wherein each set of the historical scene images is continuously acquired at predetermined time intervals, and the historical scene images are used to represent the historical driving environment of the target vehicle; an output module, configured to input at least one set of the historical scene images into the single-frame detection network, output at least one set of bird's-eye view feature sample matrices associated with the at least one set of the historical scene images, and store the at least one set of bird's-eye view feature sample matrices in a database; The second training module is used to train a second basic network using the at least one group of bird's-eye view feature sample matrices and at least one target historical pose information associated with at least one group of historical scene images to obtain a trained multi-frame fusion network, wherein each target historical pose information is associated with the target historical scene image in each group of historical scene images, and the trained single-frame detection network and the multi-frame fusion network constitute a target detection model.

10. A target detection device comprising: a single-frame detection module, configured to input a target scene image into a single-frame detection network and output a target bird's-eye view feature matrix associated with the target scene image; an acquisition module, configured to acquire a plurality of previous bird's-eye view feature matrices corresponding to a plurality of frames of previous scene images, wherein the plurality of frames of previous scene images are acquired continuously at predetermined time intervals, the acquisition time of the target scene image is later than the acquisition time of the plurality of frames of previous scene images, and the previous bird's-eye view feature matrices are acquired by processing the previous scene images using the single-frame detection network; A multi-frame fusion module is used to input the target bird's-eye view feature matrix and the multiple previous bird's-eye view feature matrices into a multi-frame fusion network, and output the target predicted pose information of the target object in the target scene image, wherein the single-frame detection network and the multi-frame fusion network are trained using the method described in any one of claims 1 to 6.

11. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 6 or 7 to 8.

12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 6 or 7 to 8.

13. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 or 7 to 8 is implemented.

Images