A monocular four-dimensional object detection method is implemented through dynamic object modeling.

Abstract

This invention discloses a monocular four-dimensional object detection method based on dynamic object modeling. The method includes: for a target image, using the current frame and previous frames as input, extracting semantic features and predicting the position and velocity of potential obstacles; extracting bird's-eye view features of the current frame; estimating the object's displacement, as well as the displacements of previous and subsequent frames, and extracting bird's-eye view features from the previous frames to obtain temporal bird's-eye view features; fusing observation information from multiple frames to obtain temporally coupled object position and velocity as predicted object information; using the predicted object information, iteratively updating the detected object position and velocity to output the position, size, category, and corresponding velocity information of obstacles surrounding the current frame as detection results; and jointly performing object detection and tracking based on the detection results. This invention can more accurately predict object positions and simultaneously achieve temporal tracking of dynamic obstacles.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for unidirectional four-dimensional object detection through dynamic object modeling. Background Technology

[0002] 3D object detection aims to locate surrounding obstacles and output their 3D position, size, orientation, and category information. Currently, 3D object detection typically plays a front-end perception role in autonomous driving systems. It locates surrounding obstacles and uses tracking algorithms to correlate objects across different timeframes, which are then fed into the back-end prediction and planning modules for predicting the future positions of obstacles and making decisions for the autonomous vehicle.

[0003] Due to their low cost, monocular camera-based 3D object detection is currently one of the mainstream choices for autonomous driving. Although monocular detectors have denser and more semantically rich scene information (such as pixels), they cannot directly predict the 3D position information of objects.

[0004] In a monocular system, single-view 3D modeling presents an ambiguous problem: a set of imaging results can come from different object positions and with different object information. Therefore, the effectiveness of monocular 3D modeling is relatively limited. In contrast, with multi-view observations, multi-view modeling can be used to deduce the 3D information of objects. In autonomous driving environments, although a single frame observation provides only one perspective, combining observations from multiple frames can create a multi-view result, thereby obtaining more accurate 3D information about objects.

[0005] In existing technologies, monocular object detection schemes are mainly divided into 3D object detection based on Perspective-view (PV) and object detection based on Bird-eye-view (BEV). For 3D object detection based on Perspective-view, the detector first uses a convolutional network to model object features, and then predicts the object's 2D position, depth, and related size information at the 2D level. This PV-based 3D object detection can be well integrated with 2D detectors, and the network structure is relatively simple. However, since feature modeling is performed at the 2D level, it cannot effectively combine temporal features for object detection. For bird-eye-view based object detection, the detector first extracts 2D features and transforms them into 3D space, and then directly performs 3D object detection in 3D space. Based on the feature representation in 3D space, some recent studies have also used consecutive frames to fuse temporal information to alleviate the ambiguity of monocular depth prediction and obtain more accurate object localization results. Although bird-eye-view based object detection can output prediction results more directly, the detection distance of the detector is predefined during training, which prevents it from being modified according to the actual use case. In addition, although existing solutions integrate features from previous and next frames, their temporal features are mainly limited to dynamic objects, and generally cannot handle the more important dynamic objects in autonomous driving scenarios well.

[0006] In summary, the existing technology has the following main drawbacks:

[0007] 1) PV-based detectors cannot effectively model temporal features because they only model features at a two-dimensional level, making it difficult to directly combine features from consecutive frames at the geometric level. Therefore, current PV-based methods offer limited improvement to detectors through temporal modeling.

[0008] 2) Currently, BEV-based detectors require the size and range of the BEV feature map to be predefined during the training phase. Therefore, the detection distance is also defined during training and cannot be changed due to scene usage.

[0009] 3) Current temporal modeling methods based on BEV typically involve simply stitching together feature maps from consecutive frames, ignoring the positional offset of dynamic objects between frames. Therefore, the temporal features obtained by this modeling method cannot accurately reflect the positional information of dynamic objects.

[0010] 4) Monocular perception schemes typically employ a separate architecture for detection and tracking. Furthermore, object tracking schemes usually require additional feature modeling to correlate objects across consecutive frames. Therefore, there is currently a lack of a perception scheme that can effectively combine object detection and tracking. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for four-dimensional object detection based on monocular observation through dynamic object modeling. This method includes the following steps:

[0012] Step S1: For the target image, using the images of the current frame and the previous frame as input, use a deep learning model to extract the semantic features of the image and predict the potential obstacle position and speed information.

[0013] Step S2: Based on the potential obstacle position information and velocity information, model the bird's-eye view features of the object around the potential area with a fixed grid size to extract the bird's-eye view features of the current frame;

[0014] Step S3: Utilize the potential obstacle position information and velocity information to estimate the object's displacement, as well as the displacement of the preceding and following frames, and then extract the bird's-eye view features of the preceding frame to obtain the temporal bird's-eye view features.

[0015] Step S4: Based on the temporal bird's-eye view features, the observation information of multiple frames is fused to obtain temporally coupled object position information and velocity information, which are used as the predicted object information;

[0016] Step S5: Use the predicted object information to replace the input of step S1, and update the position and velocity of the detected object through iterative loops, and then output the position, size, category and corresponding velocity information of the obstacles around the current frame as the detection result;

[0017] Step S6: Perform object detection and tracking based on the detection results.

[0018] Compared with existing technologies, the advantages of this invention are that it addresses the problem that existing monocular 3D detection algorithms cannot handle dynamic obstacles in temporal modeling. It proposes a loop-based dynamic object modeling method and designs a 3D object detection and tracking algorithm based on a monocular camera. Furthermore, when modeling temporal features, this invention considers both the object's position and velocity, enabling more accurate prediction of object positions and simultaneous temporal tracking of dynamic obstacles.

[0019] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0021] Figure 1This is a flowchart of a method for four-dimensional object detection based on monocular object modeling according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a target detection and tracking process according to an embodiment of the present invention;

[0023] In the attached diagram, Preceding frames; Current frame; Conv; Video sequence; Local volume; Fusion; Refined Boxes & Object Motion; Candidate Boxes & Object Motion; Recurrent refinement; and Joint detection and tracking are also included. Detailed Implementation

[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0025] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0026] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0027] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0029] This invention proposes a monocular object detection and tracking algorithm based on dynamic object modeling. By simultaneously modeling the 3D position and velocity information of the detected object during temporal feature construction, more accurate object position and velocity information can be obtained, thereby achieving joint 3D object detection and tracking. Furthermore, to alleviate the ambiguity caused by simultaneously modeling the object's position and velocity under a monocular camera, this invention also designs a loop-based temporal modeling method. Further, a BEV feature modeling method at the object level is designed, which ensures that the detector's detection distance is not affected by the configuration during training.

[0030] Combination Figure 1 and Figure 2 As shown, the provided method for monocular four-dimensional object detection based on dynamic object modeling includes the following steps:

[0031] Step S110: Extract the two-dimensional features of the input image and predict the position and velocity information of potential obstacles.

[0032] For example, the input to a monocular perception-based scheme is images of the current frame and past frames. First, a deep learning model is used to extract semantic features from the images, and then a PV detector is used to predict the location and velocity of potential obstacles.

[0033] Deep learning models can take many forms, such as convolutional neural networks or recurrent neural networks. PV detectors can also take many forms, such as FCOS3D or FCOS3D++. In one embodiment, to improve processing efficiency, the relatively simple FCOS3D detector is used (see https: / / arxiv.org / abs / 2104.10956).

[0034] Step S120: Extract the BEV features of the current frame.

[0035] For example, based on the location of potential obstacles, the BEV features of an object are modeled around the potential region with a fixed grid size. This approach, which first predicts the potential location of the object using a PV detector and then performs temporal modeling, allows the detector to be unrestricted by the detection distance during the training phase, thereby improving the flexibility of object detection.

[0036] Step S130: Estimate the object displacement using the potential obstacle positions and velocities, as well as the displacements of previous and subsequent frames, to obtain temporal BEV features.

[0037] After extracting features from the current frame, temporal modeling is performed using information extracted from past frames. Specifically, the potential obstacle positions, estimated object displacements, and the vehicle's displacements in previous and subsequent frames are used to geometrically project the current potential position to obtain the object's potential positions in past frames, and this information is then used to extract the object's past frame information. Compared to existing methods that only consider potential obstacle positions when extracting past frame information, this invention additionally considers the object's temporal displacement, thus obtaining more accurate temporal results.

[0038] It should be noted that in this paper, in order to describe the temporal relationship between frames, they are divided into types such as current frame, previous frame, and next frame. The previous frame is also called the past frame or the preceding frame, and the next frame is also called the following frame.

[0039] Step S140: Use temporal BEV features to fuse features from multiple frames.

[0040] The BEV features of the current frame and past frames obtained through steps S120 and S130 are further fused to obtain temporally tightly coupled object position and velocity information. In one embodiment, the uncertainty of prediction for each frame is considered during the fusion process, and the final result is weighted and averaged based on the confidence level of each frame.

[0041] Step S150: Perform loop modeling for the object's position and velocity.

[0042] Steps S120, S130, and S140 above describe the process of multi-frame temporal modeling. Since more accurate position information can lead to more accurate velocity information, and more accurate velocity information can also lead to more accurate position information, in one embodiment, a cyclic iterative strategy is adopted, that is, the result predicted in step S140 replaces the input in step S110, and the position and velocity results of object detection are continuously improved by iterating multiple times.

[0043] Step S160: Output the detection results of the surrounding obstacles in the current frame and the corresponding speed information.

[0044] By updating the object detection position and velocity results multiple times, the detector finally outputs the position, size, category, and corresponding velocity information of the obstacles around the current frame.

[0045] Step S170: Perform time-series tracking based on the detected object position and velocity.

[0046] After obtaining velocity information, the tracking module can easily project the prediction of the current frame onto past frames and correlate the detection results of the previous and next frames through geometric distance to achieve joint object detection and tracking.

[0047] To further verify the effectiveness of this invention, experiments were conducted on the nuScenes autonomous driving dataset. Experimental results show that the proposed scheme achieves more accurate detection and tracking results. Specifically, on the nuScenes mAP metric, it improves upon state-of-the-art methods by nearly 2%. Furthermore, this invention also yields more accurate velocity results, thereby improving tracking accuracy. Moreover, compared to other tracking schemes that associate objects between consecutive frames based on semantic features, this invention is simpler and more efficient.

[0048] In summary, compared with the prior art, the present invention has the following advantages:

[0049] 1) It can simultaneously model the position and velocity of dynamic objects and obtain accurate temporal features, thereby accurately locating objects and predicting their velocities using these features. Furthermore, when modeling the temporal features of objects, it also considers the frame offset (velocity) of dynamic objects, thus being compatible with both static and dynamic objects and obtaining more accurate object localization results in one step.

[0050] 2) An object-level BEV feature modeling scheme is provided. Since this scheme only models features near the location of potential obstacles, it does not require maintaining a fixed-size feature map. This modeling scheme allows the detection range of objects to be unrestricted by the training settings, making it simpler and more efficient. Theoretically, any object appearing in the image can be detected.

[0051] 3) By simultaneously modeling the position and velocity of objects at the feature level, we can obtain temporally tightly coupled results. Therefore, we can easily combine the predictions of consecutive frames using geometric distance to obtain a unified object detection and tracking framework.

[0052] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0053] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0054] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0055] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0056] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0057] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0058] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0059] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0060] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for monocular four-dimensional object detection based on dynamic object modeling, comprising the following steps: Step S1: For the target image, using the images of the current frame and the previous frame as input, use a deep learning model to extract the semantic features of the image and predict the potential obstacle position and speed information. Step S2: Based on the potential obstacle position information and velocity information, model the bird's-eye view features of the object around the potential area with a fixed grid size to extract the bird's-eye view features of the current frame; Step S3: Utilize the potential obstacle position information and velocity information to estimate the object's displacement, as well as the displacement of the preceding and following frames, and then extract the bird's-eye view features of the preceding frame to obtain the temporal bird's-eye view features. Step S4: Based on the temporal bird's-eye view features, the observation information of multiple frames is fused to obtain temporally coupled object position information and velocity information, which are used as the predicted object information; Step S5: Use the predicted object information to replace the input of step S1, and update the position and velocity of the detected object through iterative loops, and then output the position, size, category and corresponding velocity information of the obstacles around the current frame as the detection result; Step S6: Perform object detection and tracking based on the detection results.

2. The method according to claim 1, characterized in that, In step S1, the potential obstacle location information and velocity information are predicted using a perspective-based object detector.

3. The method according to claim 1, characterized in that, Step S3 includes: Using the potential obstacle position and velocity information, the object displacement, as well as the displacement in the preceding and following frames, is estimated. Geometrically project the current potential position to obtain the potential position of the object in the previous frame; Based on the potential location of the object in the preceding frame, the bird's-eye view features of the object in the preceding frame are then extracted.

4. The method according to claim 1, characterized in that, In step S4, during the process of fusing observation information from multiple frames, the uncertainty of prediction for each frame is considered, and a weighted average is calculated based on the confidence level of each frame.

5. The method according to claim 1, characterized in that, In step S6, the prediction of the current frame is projected onto the previous frame, and the detection results of the previous and subsequent frames are correlated by geometric distance to jointly detect and track objects.

6. The method according to claim 2, characterized in that, The object detector is the FCOS3D detector.

7. The method according to claim 1, characterized in that, The deep learning model is a convolutional neural network.

8. A computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

9. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images