Monocular 3D environment perception method and device, electronic equipment and storage medium
By combining a monocular camera with a pre-set image detection model and vehicle motion estimation, the problem of unstable 3D perception in autonomous driving by monocular cameras is solved, achieving efficient and low-cost 3D environmental perception and simplifying the fusion process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI DEEPWAY TECHNOLOGY CO LTD
- Filing Date
- 2022-10-13
- Publication Date
- 2026-07-10
AI Technical Summary
In the existing technology, the 3D perception method of monocular camera used for autonomous driving has problems such as unstable perception results and complex fusion process. In particular, the use of depth camera or binocular camera is costly, and the IMU fusion method is difficult to maintain.
A monocular camera is used in conjunction with a pre-set image detection model. The model is trained through a backbone network and multiple branch task networks to obtain the feature information of the target object. Combined with the vehicle motion estimation information, 3D environmental perception is achieved, and the Kalman filter algorithm is used to correct tracking false detections.
It enables the acquisition of rich 3D environmental perception information through a monocular camera, reduces equipment costs, simplifies the fusion process, and improves the stability and maintainability of perception.
Smart Images
Figure CN115620277B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of autonomous driving and environmental perception technology, and in particular to a monocular 3D environmental perception method, device, electronic device, and storage medium. Background Technology
[0002] Perceiving and understanding rapidly changing traffic scenarios is crucial for modern autonomous driving systems. These systems need to not only detect target objects in the image domain but also provide their three-dimensional (3D) position information in the world coordinate system to prevent collisions with other traffic objects.
[0003] In related technologies, 3D perception of traffic scenes mainly relies on depth cameras or binocular cameras, while monocular cameras mainly perceive the two-dimensional (2D) position information of target objects in the image domain, or integrate IMU inertial navigation to achieve 3D perception of traffic scenes.
[0004] Using depth cameras or stereo cameras is expensive and the perception results are unstable. However, using a fusion method that combines 2D positional information from a monocular camera with an IMU to achieve 3D perception results in complex and difficult-to-maintain fusion program code. Summary of the Invention
[0005] This application provides a monocular 3D environment perception method, device, electronic device, and storage medium to achieve richer environment perception through monocular vision without the need for complex fusion.
[0006] The embodiments of this application adopt the following technical solutions:
[0007] In a first aspect, embodiments of this application provide a monocular 3D environment perception method, wherein the method includes:
[0008] Acquire an image of the target object to be detected;
[0009] The image to be detected is input into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks.
[0010] Based on the feature information of the target object and the vehicle motion estimation information, the 3D environmental perception information of the target object is obtained.
[0011] In some embodiments, the preset image detection model includes:
[0012] A backbone network is used to extract image features;
[0013] A multi-branch task network is used to connect pre-defined feature maps with multiple convolutional detection heads.
[0014] The preset feature map is a feature map with semantic features extracted from two adjacent image frames after preset processing.
[0015] In some embodiments, the feature information of the target object includes: a rotation matrix, a translation vector, a currently drivable area, a 2D edge detection box of the target object, and a 3D edge detection box of the target object. The method further includes:
[0016] The moving speed of the target object is estimated based on the rotation matrix and the translation vector.
[0017] Based on the current drivable area, the current drivable area under the current traffic scenario is obtained after semantic segmentation.
[0018] Based on the 2D edge detection bounding box of the target object, the 2D information of the target object is obtained, and based on the 3D edge detection bounding box of the target object, the 3D information of the target object is obtained.
[0019] In some embodiments, the preset image detection model further includes:
[0020] Based on two images acquired at adjacent time points, the backbone network of a convolutional neural network and path aggregation PAN are used to extract image features from the two images at adjacent time points.
[0021] Based on the image features in two images at two adjacent time points, a first feature map and a second feature map are obtained. The first feature map is used as the feature map of the current frame, and the second feature map is used as the feature map of the previous frame.
[0022] The second feature map is projected onto the first feature map for feature alignment to obtain the feature-aligned feature map.
[0023] The feature map after feature alignment is fused with the first feature map to obtain the preset feature map.
[0024] In some embodiments, 3D environmental perception information of the target object is obtained based on the target object's feature information and vehicle motion estimation information, including:
[0025] Based on the 3D information of the target object in the feature information of the target object and the motion estimation information of the vehicle, the length, width and height (L, W, H) of the 3D edge detection box, the position coordinate information (X, Y, Z) of the target object in the 3D edge detection box, the heading angle information of the target object, the absolute lateral velocity Vy and the absolute longitudinal velocity Vx of the target object are obtained in the 3D environmental perception information of the target object.
[0026] In some embodiments, obtaining the 3D environmental perception information of the target object based on the feature information of the target object and the vehicle motion estimation information includes:
[0027] The target object is tracked based on the Kalman filter algorithm, and false detections during the tracking process are corrected by using the current drivable area.
[0028] In some embodiments, the vehicle motion estimation information is determined based on the vehicle positioning information corresponding to adjacent frames obtained by the vehicle's positioning module, and has the same timestamp as the feature information of the target object.
[0029] Secondly, embodiments of this application also provide a monocular 3D environment perception device, wherein the device includes:
[0030] The acquisition module is used to acquire the image to be detected containing the target object;
[0031] The feature module is used to input the image to be detected into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks.
[0032] The perception module is used to obtain the 3D environmental perception information of the target object based on the feature information of the target object and the vehicle motion estimation information.
[0033] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the above-described method.
[0034] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the above-described method.
[0035] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:
[0036] By combining the feature information of the target object obtained through a preset image detection model with the vehicle motion estimation information, 3D environmental perception information of the target object can be obtained. Since the preset image detection model can output dimensional feature information of the target object and fuse it with the vehicle estimation information, 3D environmental perception information can be obtained. 3D environmental perception can be achieved using only a monocular camera, unlike related technologies that use depth or binocular cameras or require IMU fusion, thus obtaining richer environmental perception information (of the target object). Attached Figure Description
[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0038] Figure 1 This is a flowchart illustrating the monocular 3D environment perception method in the embodiments of this application;
[0039] Figure 2 This is a schematic diagram of the structure of the monocular 3D environment sensing device in the embodiments of this application;
[0040] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0042] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0043] This application provides a monocular 3D environment perception method, such as... Figure 1 The diagram shows a flowchart of a monocular 3D environment perception method in an embodiment of this application. The method includes at least the following steps S110 to S130:
[0044] Step S110: Obtain the image to be detected containing the target object.
[0045] The solution involves acquiring an image to be detected using a monocular camera installed on a vehicle, and the image contains a target object.
[0046] It is understood that the target objects here include, but are not limited to, vehicles, obstacles and other objects of interest, and are not specifically limited in this application.
[0047] It's important to note that the target object refers to the scenario where the vehicle perceives its current environment. This also applies to different target objects on the same vehicle.
[0048] Step S120: Input the image to be detected into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks.
[0049] The acquired image to be detected is input into a preset image detection model. A machine learning-based network model can identify preset feature information, which is used as the feature information of the target object.
[0050] The preset image detection model employs a backbone network and multiple branch task networks, and obtains the image detection model through training until convergence. The backbone network obtains image features, and the multiple branch task networks (convolution) obtain the preset feature information.
[0051] It should be noted that the feature information of the target object includes, but is not limited to, rotation matrix, translation vector, current drivable area, 2D edge detection box of the target object, and 3D edge detection box of the target object.
[0052] The parameters output by the model can enrich the feature information of the target object. For example, the 2D edge detection bounding box of the target object serves as the 2D detection result. Similarly, the 3D edge detection bounding box of the target object can serve as its 3D information. Furthermore, the currently drivable area can be obtained as the result of semantic segmentation of the image.
[0053] The rotation matrix and the translation vector can be used to estimate the (relative) moving speed of the target object.
[0054] Step S130: Based on the feature information of the target object and the vehicle motion estimation information, obtain the 3D environmental perception information of the target object.
[0055] Based on the feature information of the target object and the ego-motion estimation information of the vehicle, the 3D environmental perception information of the target object is finally obtained and used as the 9D information estimation result (L, W, H, X, Y, Z, Yaw, Vx, Vy).
[0056] It should be noted that the 9D information estimation results include: the length, width, and height (L, W, H) of the 3D edge detection box in the 3D environmental perception information of the target object, the position coordinate information (X, Y, Z) of the target object in the 3D edge detection box, the heading angle information of the target object, the absolute lateral velocity Vy and the absolute longitudinal velocity Vx of the target object, based on the 3D information of the target object in the feature information of the target object and the motion estimation information of the vehicle.
[0057] By estimating the vehicle's motion and using feature information obtained from the network model, a 3D understanding of the current traffic scene can be achieved. A monocular camera is used to establish 3D perception information primarily for objects within the traffic scene, which is then used for planning and decision-making for autonomous vehicles. The vehicle motion estimation information is determined using positioning information from the IMU, and the vehicle's motion state information is estimated using positioning information from preceding and following frames.
[0058] Furthermore, for scenarios with multiple targets, 3D environmental perception information for each target can also be obtained.
[0059] In one embodiment of this application, the preset image detection model includes: a backbone network for extracting image features; and multiple branch task networks for connecting the preset feature map with multiple convolutional detection heads, wherein the preset feature map is a feature map with semantic features extracted from two adjacent image frames after preset processing.
[0060] In specific implementation, the backbone network is used to extract image features, including but not limited to VGGNet and RESNet, which are not specifically limited in this embodiment. Multiple branch task networks are used to connect preset feature maps to multiple convolutional detection heads.
[0061] Preferably, image features are extracted using a backbone network + path aggregation PAN method.
[0062] By connecting a preset feature map with multiple convolutional detection heads (convolution), the feature information of the target object is output.
[0063] In one embodiment of this application, the feature information of the target object includes: a rotation matrix, a translation vector, a current drivable area, a 2D edge detection box of the target object, and a 3D edge detection box of the target object. The method further includes: estimating the moving speed of the target object based on the rotation matrix and the translation vector; obtaining the current drivable area obtained after semantic segmentation in the current traffic scene based on the current drivable area; obtaining the 2D information of the target object based on the 2D edge detection box of the target object; and obtaining the 3D information of the target object based on the 3D edge detection box of the target object.
[0064] In practice, the moving speed of the target object is estimated based on the rotation matrix and the translation vector, and can then be used as the subsequent target object moving speed estimate.
[0065] The current drivable area, obtained through semantic segmentation in the current traffic scenario, can be used to correct false target detections. The detection results are then corrected according to the drivable area derived from semantic segmentation.
[0066] Based on the 2D edge detection bounding box of the target object, 2D information of the target object is obtained, and based on the 3D edge detection bounding box of the target object, 3D information of the target object is obtained. Through the 2D and 3D edge detection bounding boxes, planar 2D and three-dimensional 3D information can be obtained.
[0067] In one embodiment of this application, the preset image detection model further includes: extracting image features from two adjacent images using a convolutional neural network backbone and path aggregation (PAN); obtaining a first feature map and a second feature map based on the image features from the two adjacent images, wherein the first feature map is used as the feature map of the current frame and the second feature map is used as the feature map of the previous frame; projecting the second feature map onto the first feature map for feature alignment to obtain a feature-aligned feature map; and fusing the feature-aligned feature map with the first feature map to obtain the preset feature map.
[0068] In practice, the first step is to input two images I at adjacent time points into the network model. t I t-1 ;
[0069] Then, image features between adjacent time points are extracted using a convolutional neural network backbone network + path aggregation (PAN), and the corresponding feature maps F are obtained. t F t-1 ;
[0070] Next, the feature map F t-1 Projected onto F t Below and F t Alignment is performed to obtain F′ t-1 The result after alignment; F′ t-1 and F t The feature map F is obtained by fusion, and at this time, the feature map F is a feature map with semantic information;
[0071] By connecting different detection heads (head convolution operation) to the feature map F, different information is output ((R, t), (free-space), (2Dbox), (3Dbox)).
[0072] In one embodiment of this application, obtaining the 3D environmental perception information of the target object based on the feature information of the target object and the vehicle motion estimation information includes: tracking the target object based on the Kalman filter algorithm, and correcting false detections during the tracking process through the current drivable area.
[0073] In practice, the target is tracked using the Kalman filter algorithm, and false detections during the tracking process are corrected using the current drivable area (target feature information output by the network model).
[0074] Considering the possibility of false detections of targets, false detections during the tracking process can be corrected using the current drivable area.
[0075] In one embodiment of this application, the vehicle motion estimation information is determined based on the vehicle positioning information corresponding to adjacent frames obtained by the vehicle's positioning module, and has the same timestamp as the feature information of the target object.
[0076] In practice, ego-motion determines the vehicle's positioning information based on the adjacent frames obtained by the vehicle's positioning module, and the feature information of the target object has the same timestamp (that is, the image frame acquired by the monocular camera and the IMU positioning result are estimation results for the same frame).
[0077] This application embodiment also provides a monocular 3D environment perception device 200, such as Figure 2 As shown, a schematic diagram of the structure of a monocular 3D environment perception device in an embodiment of this application is provided. The device 200 includes at least: an acquisition module 210, a feature module 220, and a perception module 230, wherein:
[0078] In one embodiment of this application, the acquisition module 210 is specifically used to: acquire an image to be detected containing the target object.
[0079] The solution involves acquiring an image to be detected using a monocular camera installed on a vehicle, and the image contains a target object.
[0080] It is understood that the target objects here include, but are not limited to, vehicles, obstacles and other objects of interest, and are not specifically limited in this application.
[0081] It's important to note that the target object refers to the scenario where the vehicle perceives its current environment. This also applies to different target objects on the same vehicle.
[0082] In one embodiment of this application, the feature module 220 is specifically used to: input the image to be detected into a preset image detection model to obtain the feature information of the target object, wherein the preset image detection model is trained using a network structure of a backbone network and multiple branch task networks.
[0083] The acquired image to be detected is input into a preset image detection model. A machine learning-based network model can identify preset feature information, which is used as the feature information of the target object.
[0084] The preset image detection model employs a backbone network and multiple branch task networks, and obtains the image detection model through training until convergence. The backbone network obtains image features, and the multiple branch task networks (convolution) obtain the preset feature information.
[0085] It should be noted that the feature information of the target object includes, but is not limited to, rotation matrix, translation vector, current drivable area, 2D edge detection box of the target object, and 3D edge detection box of the target object.
[0086] The parameters output by the model can enrich the feature information of the target object. For example, the 2D edge detection bounding box of the target object serves as the 2D detection result. Similarly, the 3D edge detection bounding box of the target object can serve as its 3D information. Furthermore, the currently drivable area can be obtained as the result of semantic segmentation of the image.
[0087] The rotation matrix and the translation vector can be used to estimate the (relative) moving speed of the target object.
[0088] In one embodiment of this application, the perception module 230 is specifically used to: obtain 3D environmental perception information of the target object based on the feature information of the target object and the vehicle motion estimation information.
[0089] Based on the feature information of the target object and the ego-motion estimation information of the vehicle, the 3D environmental perception information of the target object is finally obtained and used as the 9D information estimation result (L, W, H, X, Y, Z, Yaw, Vx, Vy).
[0090] It should be noted that the 9D information estimation result includes: based on the target object's 3D information from its feature information and the vehicle's motion estimation information, the following are obtained: the length, width, and height (L, W, H) of the 3D edge detection box in the target object's 3D environmental perception information; the target object's position coordinates (X, Y, Z) within the 3D edge detection box; the target object's heading angle; and the target object's absolute lateral velocity Vy and absolute longitudinal velocity Vx. The absolute lateral velocity Vy and absolute longitudinal velocity Vx are determined based on the target object's coordinate position in the vehicle's coordinate system and its relative velocity in the vehicle's coordinate system.
[0091] By estimating the vehicle's motion and using feature information obtained from the network model, a 3D understanding of the current traffic scene can be achieved. A monocular camera is used to establish 3D perception information primarily for objects within the traffic scene, which is then used for planning and decision-making for autonomous vehicles. The vehicle motion estimation information is determined using positioning information from the IMU, and the vehicle's motion state information is estimated using positioning information from preceding and following frames.
[0092] Furthermore, for scenarios with multiple targets, 3D environmental perception information for each target can also be obtained.
[0093] It is understood that the above-mentioned monocular 3D environment perception device can realize all the steps of the monocular 3D environment perception method provided in the foregoing embodiments. The relevant explanations of the monocular 3D environment perception method are applicable to the monocular 3D environment perception device, and will not be repeated here.
[0094] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.
[0095] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.
[0096] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.
[0097] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a monocular 3D environment perception device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:
[0098] Acquire an image of the target object to be detected;
[0099] The image to be detected is input into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks.
[0100] Based on the feature information of the target object and the vehicle motion estimation information, the 3D environmental perception information of the target object is obtained.
[0101] The above is as stated in this application. Figure 1 The method executed by the monocular 3D environment perception device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0102] The electronic device can also perform Figure 1 A method for implementing a monocular 3D environment sensing device, and how the monocular 3D environment sensing device is used in... Figure 1 The functions of the embodiments shown are not described in detail here.
[0103] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the monocular 3D environment perception device in the illustrated embodiment is specifically used to perform:
[0104] Acquire an image of the target object to be detected;
[0105] The image to be detected is input into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks.
[0106] Based on the feature information of the target object and the vehicle motion estimation information, the 3D environmental perception information of the target object is obtained.
[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0112] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0113] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0116] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.
Claims
1. A monocular 3D environment perception method, wherein, The method includes: Acquire an image of the target object to be detected; The image to be detected is input into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks. The preset image detection model includes: A backbone network is used to extract image features; A multi-branch task network is used to connect a preset feature map with multiple convolutional detection heads, wherein the preset feature map is a feature map with semantic features extracted from two adjacent image frames after preset processing; Based on two images acquired at adjacent time points, the backbone network of a convolutional neural network and path aggregation PAN are used to extract image features from the two images at adjacent time points. Based on the image features in two images at two adjacent time points, a first feature map and a second feature map are obtained. The first feature map is used as the feature map of the current frame, and the second feature map is used as the feature map of the previous frame. The second feature map is projected onto the first feature map for feature alignment to obtain the feature-aligned feature map. The feature map after feature alignment is fused with the first feature map to obtain the preset feature map; Based on the feature information of the target object and the vehicle motion estimation information, the 3D environmental perception information of the target object is obtained; Based on the feature information of the target object and the ego-motion estimation information of the vehicle, the 3D environmental perception information of the target object is finally obtained and used as the 9D information estimation result (L, W, H, X, Y, Z, Yaw, Vx, Vy). This includes the length, width, and height of the 3D edge detection box of the target object (L, W, H), the position coordinate information of the target object in the 3D edge detection box (X, Y, Z), the heading angle information of the target object (Yaw), the absolute lateral velocity (Vy), and the absolute longitudinal velocity (Vx) of the target object. When determining the ego-motion estimation information of the vehicle, the positioning information in the IMU is used, and the positioning information of the preceding and following frames is used to estimate the motion state information of the vehicle.
2. The method as described in claim 1, wherein, The feature information of the target object includes: rotation matrix, translation vector, current drivable area, 2D edge detection box of the target object, and 3D edge detection box of the target object. The method further includes: The moving speed of the target object is estimated based on the rotation matrix and the translation vector. Based on the current drivable area, the current drivable area under the current traffic scenario is obtained after semantic segmentation. Based on the 2D edge detection bounding box of the target object, the 2D information of the target object is obtained, and based on the 3D edge detection bounding box of the target object, the 3D information of the target object is obtained.
3. The method as described in claim 2, wherein, The step of obtaining the 3D environmental perception information of the target object based on its feature information and vehicle motion estimation information includes: The target object is tracked based on the Kalman filter algorithm, and false detections during the tracking process are corrected by using the current drivable area.
4. The method as described in claim 1, wherein, The vehicle motion estimation information is determined based on the vehicle positioning information corresponding to adjacent frames obtained by the vehicle's positioning module, and has the same timestamp as the feature information of the target object.
5. A monocular 3D environmental sensing device, wherein, The device includes: The acquisition module is used to acquire the image to be detected containing the target object; The feature module is used to input the image to be detected into a preset image detection model to obtain the feature information of the target object. The preset image detection model is trained using a network structure of a backbone network and multiple branch task networks. The preset image detection model includes: A backbone network is used to extract image features; A multi-branch task network is used to connect a preset feature map with multiple convolutional detection heads, wherein the preset feature map is a feature map with semantic features extracted from two adjacent image frames after preset processing; Based on two images acquired at adjacent time points, the backbone network of a convolutional neural network and path aggregation PAN are used to extract image features from the two images at adjacent time points. Based on the image features in two images at two adjacent time points, a first feature map and a second feature map are obtained. The first feature map is used as the feature map of the current frame, and the second feature map is used as the feature map of the previous frame. The second feature map is projected onto the first feature map for feature alignment to obtain the feature-aligned feature map. The feature map after feature alignment is fused with the first feature map to obtain the preset feature map; The perception module is used to obtain the 3D environmental perception information of the target object based on the feature information of the target object and the vehicle motion estimation information; Based on the feature information of the target object and the ego-motion estimation information of the vehicle, the 3D environmental perception information of the target object is finally obtained and used as the 9D information estimation result (L, W, H, X, Y, Z, Yaw, Vx, Vy). This includes the length, width, and height of the 3D edge detection box of the target object (L, W, H), the position coordinate information of the target object in the 3D edge detection box (X, Y, Z), the heading angle information of the target object (Yaw), the absolute lateral velocity (Vy), and the absolute longitudinal velocity (Vx) of the target object. When determining the ego-motion estimation information of the vehicle, the positioning information in the IMU is used, and the positioning information of the preceding and following frames is used to estimate the motion state information of the vehicle.
6. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 4.
7. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 4.