Monocular three-dimensional target detection method, system and device based on Lyapunov optimization theory and multi-task learning and storage medium

Abstract

The invention discloses a monocular three-dimensional target detection method, a monocular three-dimensional target detection system and monocular three-dimensional target detection equipment based on a Lyapunov optimization theory and multi-task learning, and a storage medium. The monocular three-dimensional target detection method comprises the following steps: inputting continuous frames of monocular images; constructing a multi-task monocular three-dimensional target detection model; based on the Lyapunov optimization theory, dynamically adjusting the loss weight of each subtask, and constructing a multi-task learning loss function; performing model training according to the multi-task learning loss function to obtain a trained monocular three-dimensional target detection model; and performing feature extraction and multi-task reasoning on the input continuous frame monocular image by using the trained model. The invention aims to provide monocular three-dimensional target detection based on 2D target detection and key point estimation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a monocular 3D target detection method, system, device, and storage medium based on Lyapunov optimization theory and multi-task learning. Background Technology

[0002] Monocular 3D object detection aims to predict the 3D position, size, and pose of objects in a scene from monocular images, and has important applications in fields such as autonomous driving and robot navigation. However, due to the lack of direct depth information in monocular images, 3D object detection faces multiple challenges, including ambiguity in depth estimation, object occlusion, scale variations, and environmental interference.

[0003] Several monocular 3D target detection schemes exist in the current technology. For example, CN115346210B proposes a monocular 3D target detection method based on pseudo-target generation and quality score evaluation. This method enhances training samples by generating pseudo-targets and combines them with a deep network to predict 3D information. CN115205654B discloses a monocular 3D target detection method based on keypoint constraints. This method improves detection accuracy by constructing a loss function based on the geometric consistency between predicted 2D keypoints and 3D positions.

[0004] However, the aforementioned existing technologies still have the following drawbacks: Fixed weight allocation in multi-task learning: Existing multi-task learning methods typically use fixed weights to combine the losses of each subtask, thus transmitting the multi-task loss function. in express Sub-tasks (including 2D object detection, 3D object detection and depth estimation in this invention). It is a task The weight, It is a task loss function, These are the parameters of the model. In traditional multi-task learning, It is fixed and difficult to dynamically adapt to the learning state of each task during training, which can easily lead to gradient conflicts or certain tasks dominating the training process.

[0005] Therefore, it is necessary to propose a monocular 3D object detection method that can dynamically adjust the learning weights of multiple tasks. Summary of the Invention

[0006] The purpose of this invention is to provide a monocular 3D target detection method, system, device, and storage medium based on Lyapunov optimization theory and multi-task learning. This method can dynamically adjust the multi-task learning weights.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows: A monocular 3D object detection method based on Lyapunov optimization theory and multi-task learning includes the following steps: S1: Input continuous frames of monocular images; S2: Construct a multi-task monocular 3D target detection model; S3: Based on Lyapunov optimization theory, dynamically adjust the loss weights of each sub-task to construct a multi-task learning loss function; S4: Train the model according to the multi-task learning loss function to obtain the trained monocular 3D target detection model; S5: Use the trained model to perform feature extraction and multi-task inference on the input continuous frame monocular images.

[0008] Furthermore, the dynamic adjustment of the loss weights of each subtask based on Lyapunov optimization theory in step S3 specifically includes: Construct a weighted error queue for each task. ; Define Lyapunov functions ; By minimizing Lyapunov drift By using the expected value of the weighted sum of task losses, we can derive the weight update rule.

[0009] Furthermore, the weight update rule is as follows: in, Let be the weight of task t. The learning rate of the neural network. Let be the loss function for task t.

[0010] Furthermore, the multi-task learning loss function in step S3 is: in, This represents the gradient similarity between tasks, used to measure the correlation between tasks.

[0011] Furthermore, the gradient similarity between the tasks: And optimize the task parameter sharing strategy based on the gradient similarity.

[0012] Further, the multi-task monocular three-dimensional target detection model in the S2 step comprises a parallel depth estimation head, a pseudo 3D detection head and a monocular 3D detection head, and the multi-task reasoning in the S5 is that the depth estimation head performs a depth estimation subtask and outputs a dense depth map; the pseudo 3D detection head performs a pseudo 3D target detection subtask and outputs a two-dimensional bounding box and a key point position; and the monocular 3D detection head performs a monocular 3D target detection subtask and outputs a three-dimensional bounding box.

[0013] Further, the multi-task monocular three-dimensional target detection model in the S2 step comprises a CNN and a Transformer hybrid feature extraction network, and the CNN and the Transformer hybrid feature extraction network are configured as follows: the input continuous frame monocular image is first subjected to a YOLOv7 neural network to output multi-scale features, the multi-scale features are input into a Transformer encoder and a decoder for fine feature extraction after being fused by a fusion module.

[0014] The monocular three-dimensional target detection system based on the Lyapunov optimization theory and the multi-task learning comprises: An image input module configured to input continuous frame monocular images; A model construction module configured to construct a multi-task monocular three-dimensional target detection model; A dynamic weight optimization module configured to dynamically adjust loss weights of each subtask based on the Lyapunov optimization theory and construct a multi-task learning loss function; A model training module configured to perform model training according to the multi-task learning loss function and obtain a trained monocular three-dimensional target detection model; A three-dimensional detection output module configured to call the trained model to perform feature extraction and multi-task reasoning on the input continuous frame monocular images.

[0015] An electronic device comprising a processor and a memory storing a computer program, wherein the processor implements the steps of the monocular three-dimensional target detection method based on the Lyapunov optimization theory and the multi-task learning when executing the computer program.

[0016] A computer readable storage medium storing a computer program, wherein the computer program implements the steps of the monocular three-dimensional target detection method based on the Lyapunov optimization theory and the multi-task learning when executed by a processor.

[0017] The monocular three-dimensional target detection method based on the Lyapunov optimization theory and the multi-task learning has the following advantages: (1) The application realizes a stable and self-adaptive multi-task training process, avoiding gradient conflicts; traditional multi-task learning uses fixed loss weights, which is difficult to balance the learning progress of different tasks and is prone to lead to gradient conflicts or dominance of some tasks in training. The application innovatively introduces Lyapunov optimization theory, dynamically adjusts the loss weights of each sub-task by constructing a weight error queue and a Lyapunov function. This mechanism can automatically stabilize the weight distribution according to the real-time state of each task (such as gradient variance) during training, minimize the negative interference between tasks, ensure smoother model convergence and higher training efficiency, and ultimately improve the overall performance and generalization ability of the model.

[0018] (2) The application introduces a task correlation constraint based on gradient similarity in the multi-task learning loss function, which can explicitly measure and optimize the parameter sharing strategy between different tasks. This not only improves the learning efficiency of the model, but also generates an inherent uncertainty measure in the learning process itself. The system can perceive the consistency between different task prediction results, providing additional confidence reference for subsequent autonomous driving decision modules (such as planning and control), helping to trigger more cautious safety strategies when there is ambiguity in perception results, reducing the risk of false positives or false negatives. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A monocular three-dimensional target detection model architecture provided by the application is shown in the figure; Figure 2 A hybrid feature extraction network architecture for monocular three-dimensional target detection provided by the application is shown in the figure; Figure 3 A monocular three-dimensional target detection result provided by the application is shown in the figure; Figure 4 A monocular three-dimensional target detection result provided by the application is shown in the figure; Figure 5 A monocular three-dimensional target detection result provided by the application is shown in the figure; Figure 6 A structural schematic diagram of an electronic device provided by the application is shown in the figure. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0021] Embodiment one As Figures 1 to 5The monocular 3D object detection method shown, based on Lyapunov optimization theory and multi-task learning, includes the following steps: S1: Input continuous frame monocular images, such as road scene images acquired by a vehicle-mounted monocular camera to obtain continuous frame monocular images, and store them in a time-series continuous frame manner. For example, input 2 adjacent frames in each batch, with a frame interval Δt=1 / 30≈33ms, denoted as frame t and frame t+1, to ensure the correlation of time series. S2: Construct a multi-task monocular 3D target detection model. The multi-task monocular 3D target detection model in this embodiment includes a CNN and Transformer hybrid feature extraction network and parallel depth estimation head, pseudo 3D detection head and monocular 3D detection head.

[0022] The CNN and Transformer hybrid feature extraction network architecture is as follows: the input continuous frame monocular images are processed by YOLOv7 to output multi-scale features; these multi-scale features come from different depths of the network: shallow features have high resolution and are rich in details such as edges and corners; the multi-scale features are fused by the multi-scale feature fusion module and then input into the Transformer encoder and decoder for further fine-grained feature extraction; the fine-grained features are used in parallel depth estimation head, pseudo 3D detection head and monocular 3D detection head to perform subtasks.

[0023] S3: Based on Lyapunov optimization theory, the loss weights of each sub-task are dynamically adjusted to construct a multi-task learning loss function; S3 is the key innovation of this invention, aiming to solve the training imbalance problem caused by fixed weights in multi-task learning, and the specific implementation is as follows: Construct a weighted error queue for each task. For each task (T = {depth estimation, pseudo-3D detection, monocular 3D detection}), in the k-th round of training, its gradient variance is calculated. As a measure of task instability, a virtual weighted error queue is constructed. To accumulate this instability: Define Lyapunov functions: It is used to measure the instability of all task queues.

[0024] By minimizing Lyapunov drift The expected value of the weighted sum of task losses: The goal of each training round is to minimize Lyapunov drift. The expected value of its weighted sum with the task loss is: where, control the stability of task weights, avoid drastic fluctuations, is the original loss of task t, control the goal of task optimization, and φ is a trade-off coefficient.

[0025] Derive the weight update rule: by solving the above optimization goal, the online update rule of task weight can be derived: where, is the weight of task t, is the learning rate of the neural network, is the loss function of task t. This rule makes the weight adaptively adjust according to the current instability of the task and the gradient direction of its loss to the parameters.

[0026] Construct the final multi-task loss function: combine dynamic weights and task correlation constraints, and the total loss function is: where, is the gradient cosine similarity of tasks and , which represents the gradient similarity used to measure the correlation between tasks. λ is the weight of the task correlation constraint. indicates that tasks and task are similar and should share more parameters, indicates that the tasks are independent and should reduce sharing to maintain individual expression, indicates that the gradient conflicts and the parameter sharing strategy needs to be optimized.

[0027] S4: training the monocular three-dimensional target detection model according to the multi-task learning loss function; The following is a specific embodiment of the training process: Rounds 1-50 (warm-up phase): fix the task weight w_t=1 / 3, only train the CNN+Transformer hybrid architecture and the three-task head, and make the model converge to the initial stable state; Rounds 51-300 (Lyapunov optimization phase): after each round of training, execute the weight update rule of step S3 to dynamically adjust , and calculate the multi-task loss , and update the model parameters through back propagation; Model saving: save the model weight every 20 rounds, and select the model with the highest "monocular 3D detection AP" on the KITTI validation set as the final training model.

[0028] S5: using the trained model to perform feature extraction and multi-task reasoning on the input continuous frame monocular image.

[0029] The multi-task reasoning in S5 is: The depth estimation head is usually composed of one or more convolutional layers, responsible for predicting the depth value of each pixel from the feature map, and the depth estimation head performs the depth estimation subtask and outputs a dense depth map, which is used to provide depth constraints for 3D detection; The pseudo 3D detection head actually performs the fusion task of 2D target detection and key point estimation, which predicts the 2D coordinates of the two-dimensional bounding box (class, center point, width and height) and the pre-defined three key points (for example, the bottom contact point of the target) based on the feature map, and these 2D detection results will be used to construct the pseudo 3D target in the subsequent and provide geometric priors, and the pseudo 3D detection head performs the pseudo 3D target detection subtask and outputs the two-dimensional bounding box and the key point position, which is used to generate the geometric constraints of the pseudo 3D target; The monocular 3D detection head directly regresses the complete 3D properties of the target based on the feature map, including the center (X, Y, Z in the camera coordinate system), size (length, width, height) and orientation angle of the 3D bounding box, as shown in Figures 3 to 5 The monocular 3D detection head performs the monocular 3D target detection subtask and outputs the three-dimensional bounding box.

[0030] The detection performance of the monocular three-dimensional target detection method based on Lyapunov optimization theory and multi-task learning of the present application on the validation set of the KITTI dataset, and the experimental data of the detection of the monocular three-dimensional target detection method based on Lyapunov optimization theory and multi-task learning of the present application and other algorithms on the validation set of the KITTI dataset are as follows: According to the experimental data, in the KITTI benchmark test, the 3D detection accuracy (AP 3D ) of the car class reaches 29.11 (easy target), 19.38 (medium difficulty target), and 17.54 (difficult target), which surpasses most fusion methods.

[0031] Embodiment Two The monocular three-dimensional target detection system based on Lyapunov optimization theory and multi-task learning comprises: An image input module for inputting continuous frame monocular images; A model construction module for constructing a multi-task monocular three-dimensional target detection model; A dynamic weight optimization module based on Lyapunov optimization theory, which dynamically adjusts the loss weights of each subtask and constructs a multi-task learning loss function; The model training module is used to train the model according to the multi-task learning loss function to obtain the trained monocular 3D object detection model. The 3D detection output module is used to call the trained model to perform feature extraction and multi-task inference on the input continuous frame monocular images.

[0032] Example 3 Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention, as shown below. Figure 3 As shown, the electronic device includes a processor 310, a memory 320, an input device 330, and an output device 340; the number of processors 310 in the computer device can be one or more. Figure 4 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, and output device 340 in the electronic device can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0033] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning in this embodiment of the invention (e.g., the receiving module 210, processing module 220, and indexing module 230 of the monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning). The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby realizing the aforementioned monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning.

[0034] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] Input device 330 can be used to receive input user identity information. Output device 340 may include display devices such as a display screen.

[0036] Example 4 The fourth embodiment of the present application also provides a storage medium containing computer executable instructions, which are used to execute the monocular three-dimensional target detection method based on Lyapunov optimization theory and multi-task learning when executed by a computer processor.

[0037] Of course, the storage medium containing computer executable instructions provided by the embodiments of the present application is not limited to the method operations as described above, and can also execute the related operations in the monocular three-dimensional target detection method based on Lyapunov optimization theory and multi-task learning provided by any embodiment of the present application.

[0038] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary general hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH, a hard disk or an optical disc, etc., and includes a number of instructions to make an electronic device (which can be a mobile phone, a personal computer, a server or a network device, etc.) execute the methods described in various embodiments of the present application.

[0039] It is worth noting that in the embodiments of the device of the monocular three-dimensional target detection method based on Lyapunov optimization theory and multi-task learning, each unit and module included is only divided according to functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation, and do not limit the protection scope of the present application.

[0040] According to the disclosure and teaching of the above description, those skilled in the art can also make changes and modifications to the above embodiments. Therefore, the present application is not limited to the specific embodiments disclosed and described above, and some modifications and changes of the present application should also fall within the protection scope of the claims of the present application. In addition, although some specific terms are used in the present specification, these terms are only for convenience of explanation and do not constitute any limitation on the present application.

Claims

1. A monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning, characterized by: Includes the following steps: S1: Input continuous frames of monocular images; S2: Construct a multi-task monocular 3D target detection model; S3: Based on Lyapunov optimization theory, dynamically adjust the loss weights of each sub-task to construct a multi-task learning loss function; S4: Train the model according to the multi-task learning loss function to obtain the trained monocular 3D target detection model; S5: Use the trained model to perform feature extraction and multi-task inference on the input continuous frame monocular images.

2. The monocular three-dimensional target detection method according to claim 1, characterized in that: The S3 step, which dynamically adjusts the loss weights of each subtask based on Lyapunov optimization theory, specifically includes: Construct a weighted error queue for each task. ; Define Lyapunov functions ; By minimizing Lyapunov drift By using the expected value of the weighted sum of task losses, we can derive the weight update rule.

3. The monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning according to claim 2, characterized in that: The weight update rule is as follows: in, Let be the weight of task t. The learning rate of the neural network. Let be the loss function for task t.

4. The monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning according to claim 1, characterized in that: The multi-task learning loss function in step S3 is: in, This represents the gradient similarity between tasks, used to measure the correlation between tasks.

5. The monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning according to claim 4, characterized in that: Gradient similarity between tasks: And optimize the task parameter sharing strategy based on the gradient similarity.

6. The monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning according to claim 1, characterized in that: The multi-task monocular 3D object detection model in step S2 includes a parallel depth estimation head, a pseudo-3D detection head, and a monocular 3D detection head. The multi-task inference in step S5 is as follows: the depth estimation head performs a depth estimation sub-task and outputs a dense depth map; the pseudo-3D detection head performs a pseudo-3D object detection sub-task and outputs a 2D bounding box and key point positions; the monocular 3D detection head performs a monocular 3D object detection sub-task and outputs a 3D bounding box.

7. The monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning according to claim 1, characterized in that: The multi-task monocular 3D target detection model in step S2 includes a hybrid feature extraction network of CNN and Transformer. The architecture of the hybrid feature extraction network of CNN and Transformer is as follows: the input continuous frame monocular image is first processed by the YOLOv7 neural network to output multi-scale features. The multi-scale features are then fused by the fusion module and input into the Transformer encoder and decoder for fine-grained feature extraction.

8. A monocular 3D target detection system based on Lyapunov optimization theory and multi-task learning, characterized in that: include: The image input module is used to input continuous frames of monocular images; The model building module is used to build multi-task monocular 3D object detection models. The dynamic weight optimization module, based on Lyapunov optimization theory, dynamically adjusts the loss weights of each subtask to construct a multi-task learning loss function; The model training module is used to train the model according to the multi-task learning loss function to obtain the trained monocular 3D object detection model. The 3D detection output module is used to call the trained model to perform feature extraction and multi-task inference on the input continuous frame monocular images.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the monocular 3D target detection method based on Lyapunov optimization theory and multi-task learning as described in any one of claims 1 to 7.

Images