Three-dimensional object detection method and device, electronic device and storage medium

Abstract

The present disclosure provides a three-dimensional object detection method and device, electronic device and storage medium, which relate to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, and can be applied to scenes such as autonomous driving. The three-dimensional object detection method includes: extracting a feature map of an image to be detected; projecting the feature map into a target three-dimensional space to obtain a first spatial feature of the target three-dimensional space; sampling the feature map based on the projection point of a reference point in the target three-dimensional space in the feature map to obtain a second spatial feature of the target three-dimensional space; fusing the first spatial feature and the second spatial feature to obtain a fused spatial feature; and determining a three-dimensional object detection result of the image to be detected based on the fused spatial feature.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, and can be applied to scenarios such as autonomous driving. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Agent is a concept in the field of artificial intelligence, which refers to an entity that has autonomous behavior capabilities and can interact with the environment, such as self-driving vehicles, industrial robots, household robots (including sweeping robots, humanoid robots, etc.), commercial robots (including food delivery robots, tour guide robots, etc.), etc.

[0004] Three-dimensional (3D) object detection refers to the detection of three-dimensional information of an object (such as length, width, height, center point coordinates, category, orientation angle, etc.), which is widely used in the decision-making and control of intelligent agents.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] The present disclosure provides a three-dimensional object detection method and device, a three-dimensional object detection method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0007] According to one aspect of the present disclosure, a three-dimensional object detection method is provided, including: extracting a feature map of an image to be detected; projecting the feature map into a target three-dimensional space to obtain a first spatial feature of the target three-dimensional space; sampling the feature map based on a projection point of a reference point in the target three-dimensional space in the feature map to obtain a second spatial feature of the target three-dimensional space; fusing the first spatial feature and the second spatial feature to obtain a fused spatial feature; and determining a three-dimensional object detection result of the image to be detected based on the fused spatial feature.

[0008] According to one aspect of the present disclosure, a three-dimensional object detection device is provided, comprising: an extraction module, configured to extract a feature map of an image to be detected; a projection module, configured to project the feature map into a target three-dimensional space to obtain a first spatial feature of the target three-dimensional space; a sampling module, configured to sample the feature map based on a projection point of a reference point in the target three-dimensional space in the feature map to obtain a second spatial feature of the target three-dimensional space; a fusion module, configured to fuse the first spatial feature and the second spatial feature to obtain a fused spatial feature; and a determination module, configured to determine a three-dimensional object detection result of the image to be detected based on the fused spatial feature.

[0009] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0010] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0011] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program can implement the above method when executed by a processor.

[0012] According to one or more embodiments of the present disclosure, the accuracy of three-dimensional object detection can be improved.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2A flow chart of a three-dimensional object detection method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A schematic diagram showing a three-dimensional object detection model and a detection process according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A structural block diagram of a three-dimensional object detection device according to an embodiment of the present disclosure is shown; and

[0019] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations. "Multiple" refers to two or more.

[0023] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] 3D object detection technology has a wide range of applications. For example, in the field of autonomous driving, 3D object detection technology can help vehicles accurately identify and locate pedestrians, vehicles and other targets on the road, and provide them to the autonomous driving system for decision-making and control. In the field of industrial automation, 3D object detection technology can be used to accurately locate and control industrial robots, perform quality inspection and classification on products on the production line, etc. In the field of security monitoring, 3D object detection technology can be used to monitor information such as the flow of people and vehicles in public places (such as airports, railway stations, etc.), and provide real-time security alerts and warnings. In the field of virtual reality, 3D object detection technology can be used to achieve more realistic virtual scenes, allowing users to interact more naturally with objects in the virtual environment.

[0025] In the related art, monocular images, that is, images collected by a monocular image acquisition device with only one camera, are often used to perform three-dimensional object detection. Specifically, by performing depth estimation on the image, the two-dimensional image is projected into a three-dimensional space, and then a three-dimensional spatial feature is constructed, and the three-dimensional object detection result is obtained based on the three-dimensional spatial feature, that is, the three-dimensional attributes of the target object (such as a vehicle, a pedestrian, etc.) in the image, such as length, width, height, coordinates of the center point, category, orientation angle, etc.

[0026] In the above-mentioned related technologies, since the monocular image only includes two-dimensional image information and lacks three-dimensional depth information, it is difficult to accurately estimate the depth of the monocular image. The three-dimensional spatial features obtained by the one-way projection of the two-dimensional image to the three-dimensional space cannot fully and accurately reflect the three-dimensional position information of the object, resulting in low accuracy of the three-dimensional object detection results.

[0027] In view of the above problems, an embodiment of the present disclosure provides a three-dimensional object detection method based on bidirectional feature projection. According to an embodiment of the present disclosure, in the three-dimensional object detection process, the feature map of the image to be detected is first extracted. Subsequently, by projecting the two-dimensional feature map into the target three-dimensional space, the first spatial feature of the target three-dimensional space is obtained, so that the first spatial feature can simulate the position mapping relationship from the two-dimensional image to the three-dimensional space. At the same time, by sampling from the three-dimensional space to the two-dimensional image, the second spatial feature of the target three-dimensional space is obtained, so that the second spatial feature can simulate the position mapping relationship from the three-dimensional space to the two-dimensional image. The first spatial feature is fused with the second spatial feature, and the obtained fused spatial feature can simultaneously express the position mapping relationship from the two-dimensional image to the three-dimensional space and from the three-dimensional space to the two-dimensional image, thereby realizing the precise mining of the object position relationship and being able to fully and accurately reflect the three-dimensional position information of the object. Performing three-dimensional object detection based on the fused spatial feature can improve the accuracy of three-dimensional object detection.

[0028] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0030] In an embodiment of the present disclosure, the motor vehicle 110 may include an electronic device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0031] The server 120 may run one or more services or software applications that enable the execution of the methods of the disclosed embodiments. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0032] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0033] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0034] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0035] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, Wi-Fi), and / or any combination of these and other networks.

[0036] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The databases 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 150 may be of different types. In some embodiments, the databases 150 may be relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0037] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0038] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0039] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth using the IEEE 802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0040] The motor vehicle 110 may further include an inertial navigation module. The inertial navigation module and the satellite positioning module may be combined into a combined positioning system to achieve initial positioning of the motor vehicle 110.

[0041] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle, and accordingly, the motor vehicle 110 is an automatic driving vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of an electronic device on the motor vehicle side (client) according to the present disclosure.

[0042] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and devices described in the present disclosure. Specifically, the motor vehicle 110 or the server 120 can be configured to perform the three-dimensional object detection method of the embodiment of the present disclosure.

[0043] According to some embodiments, the motor vehicle 110 may perform the 3D object detection method of the embodiment of the present disclosure during its operation, and perform 3D object detection on the environment image collected by the visual camera to obtain a 3D object detection result. The detection result may be used for control and decision-making of the motor vehicle 110.

[0044] According to some embodiments, during the operation of the motor vehicle 110, the environment image captured by the visual camera can be sent to the server 120. The server 120 obtains the 3D object detection result by executing the 3D object detection method of the embodiment of the present disclosure, and returns the detection result to the motor vehicle 110.

[0045] According to some embodiments, the 3D object detection method of the embodiment of the present disclosure may be implemented by an end-to-end 3D object detection model. The model may take an environmental image captured by a visual camera of the motor vehicle 110 as input and output a 3D object detection result of the environmental image. The 3D object detection model may be deployed on the vehicle side, such as the motor vehicle 110; or may be deployed on the service side, such as the server 120, or on other servers other than the server 120 ( Figure 1 (not shown).

[0046] Figure 2 FIG. 2 is a flowchart of a three-dimensional object detection method 200 according to an embodiment of the present disclosure. As described above, the execution subject of the method 200 may be a client device (eg, Figure 1 ), or a server (e.g., a client device 101-106 shown in FIG. Figure 1 , server 120 shown in FIG.

[0047] like Figure 2 As shown, the method 200 includes steps S210 - S250 .

[0048] In step S210, a feature map of the image to be detected is extracted.

[0049] In step S220, the feature map is projected into the target three-dimensional space to obtain a first spatial feature of the target three-dimensional space.

[0050] In step S230, the feature map is sampled based on the projection points of the reference points in the target three-dimensional space in the feature map to obtain the second spatial features of the target three-dimensional space.

[0051] In step S240, the first spatial feature and the second spatial feature are fused to obtain a fused spatial feature.

[0052] In step S250, a three-dimensional object detection result of the image to be detected is determined based on the fused spatial features.

[0053] According to an embodiment of the present disclosure, a three-dimensional object detection method based on bidirectional feature projection is provided. By projecting a two-dimensional feature map into a target three-dimensional space, a first spatial feature of the target three-dimensional space is obtained, so that the first spatial feature can simulate the position mapping relationship from a two-dimensional image to a three-dimensional space. By sampling from a three-dimensional space to a two-dimensional image, a second spatial feature of the target three-dimensional space is obtained, so that the second spatial feature can simulate the position mapping relationship from a three-dimensional space to a two-dimensional image. The first spatial feature is fused with the second spatial feature, and the obtained fused spatial feature can simultaneously express the position mapping relationship from a two-dimensional image to a three-dimensional space and from a three-dimensional space to a two-dimensional image, thereby achieving accurate mining of the object position relationship and being able to fully and accurately reflect the three-dimensional position information of the object. Performing three-dimensional object detection based on fused spatial features can improve the accuracy of three-dimensional object detection.

[0054] The following is a detailed description of each step of method 200.

[0055] In step S210, a feature map of the image to be detected is extracted.

[0056] The image to be detected can be an image captured by any image acquisition device. For example, the image acquisition device can be a visual camera deployed on an intelligent body such as a motor vehicle or an industrial robot, or a visual camera deployed on the side of a road. The image to be detected only includes two-dimensional pixel coordinates and color information, and does not include color information in three-dimensional space.

[0057] According to some embodiments, there may be multiple images to be detected. The images to be detected may include environmental images captured by multiple image acquisition devices. The multiple image acquisition devices are respectively arranged at different positions of the intelligent body. Thus, the surrounding environment of the intelligent body can be fully perceived by using multiple image acquisition devices with different viewing angles, thereby improving the accuracy of three-dimensional object detection.

[0058] For example, six image acquisition devices can be set in front of the motor vehicle, in front of the left, in front of the right, in the rear, in the left, and in the rear of the right, and six environmental images acquired by the six image acquisition devices are used as images to be detected. By comprehensively analyzing the six environmental images, it is possible to fully perceive the surrounding environment of the motor vehicle and improve the accuracy of three-dimensional object detection.

[0059] According to some embodiments, in step S210, a trained feature extraction network may be used to extract a feature map of the image to be detected. Specifically, the image to be detected is input into the trained feature extraction network, and a feature map output by the feature extraction network may be obtained. In the case where there are multiple images to be detected, each image to be detected may be input into the feature extraction network to extract the feature map of the image to be detected.

[0060] The feature extraction network may be implemented as a neural network, such as a convolutional neural network, etc. According to some embodiments, the feature extraction network may be, for example, a ResNet (residual network), a FPN (Feature Pyramid Network), etc.

[0061] It can be understood that the image to be detected has a certain pixel size, for example, it can be W*H, where W and H are the number of horizontal pixels and the number of vertical pixels of the image to be detected, respectively. The image to be detected usually includes multiple color channels, for example, including three channels of RGB. For a three-channel image to be detected, its size can be expressed as W*H*3. Through feature extraction, a feature map of the image to be detected is obtained. The size of the feature map can be expressed as w*h*c, where w, h, and c represent the number of horizontal pixels, the number of vertical pixels, and the number of channels of the feature map, respectively. Since the feature map is used to express the multi-dimensional semantic information of the image to be detected, its pixel size is usually smaller than the pixel size of the image to be detected, for example, it can be w=W / 8, h=H / 8; its channel number c is usually larger than the channel number of the image to be detected, for example, c=128. It can be understood that in other embodiments, the pixel size of the feature map can also be larger than the pixel size of the image to be detected, for example, w=2W, h=2H. The number of channels c of the feature map can also be smaller than the number of channels of the image to be detected, for example, c=1 or 2.

[0062] As described above, the feature map of the image to be detected may include at least one channel. Accordingly, the feature map includes first feature vectors of each of the plurality of pixels. The first feature vector of a pixel with coordinates (u, v) (0≤u≤w-1, 0≤v≤h-1) is composed of channel values ​​of each channel of the pixel. In the case where the feature map includes c channels, the first feature vector of each pixel in the feature map is a c-dimensional vector.

[0063] Based on the feature map extracted in step S210, steps S220 and S230 are executed. The present disclosure does not limit the execution order of steps S220 and S230. For example, the two can be executed successively or in parallel.

[0064] In step S220, the feature map is projected into the target three-dimensional space to obtain a first spatial feature of the target three-dimensional space.

[0065] According to some embodiments, the target three-dimensional space may be a Bird's Eye View (BEV) space. The Bird's Eye View refers to a bird's-eye view.

[0066] According to some embodiments, the image to be detected is collected by an image acquisition device on the intelligent body, and the target three-dimensional space is determined according to the region of interest of the intelligent body. The region of interest is the environmental area that the intelligent body needs to focus on in decision-making. The region of interest of the intelligent body can be preset. For example, for an autonomous driving vehicle in motion, its region of interest can be a space of 100m*100m*4m formed by the center of the rear axle of the vehicle as the origin, 50m in front of the vehicle (positive direction of the y-axis), 50m on the left and right (positive direction of the x-axis), 1m below, and 3m above (positive direction of the z-axis). Accordingly, the space can be set as the target three-dimensional space.

[0067] According to the above embodiment, the target three-dimensional space is determined based on the region of interest of the intelligent agent, so that the three-dimensional object detection results in the target three-dimensional space are consistent with the actual application scenario of the intelligent agent, thereby achieving accurate decision-making and control of the intelligent agent.

[0068] According to some embodiments, step S220 includes steps S221 - S223 .

[0069] In step S221, based on the parameters of the image acquisition device that acquires the image to be detected, the spatial point corresponding to each pixel in the feature map is determined.

[0070] In step S222, based on the first eigenvector of the pixel, a second eigenvector of the spatial point corresponding to the pixel is determined.

[0071] In step S223, the first spatial feature is determined based on the second feature vectors of the spatial points corresponding to each of the plurality of pixels.

[0072] According to the above embodiment, the first spatial feature is obtained by projecting the two-dimensional pixel to the three-dimensional space point, so that the first spatial feature can simulate the position mapping relationship from the two-dimensional image to the three-dimensional space.

[0073] The parameters of the image acquisition device include intrinsic parameters and extrinsic parameters. The parameters of the image acquisition device can be obtained, for example, by calibrating the image acquisition device.

[0074] The intrinsic parameters of the image acquisition device include the focal length f of the image acquisition device, the scaling factor α of the pixel plane relative to the imaging plane on the horizontal axis, and the translation amount c of the origin. x , the scaling factor β on the ordinate axis and the translation c of the origin y The intrinsic parameters of the image acquisition device can be expressed as an intrinsic parameter matrix. The intrinsic parameter matrix is ​​used to describe the coordinate transformation relationship from the camera coordinate system to the pixel coordinate system.

[0075] The extrinsic parameters of an image acquisition device are the parameters of the camera in the world coordinate system, such as the camera's position, rotation angle, etc. The extrinsic parameters of an image acquisition device can be expressed as an extrinsic parameter matrix. The extrinsic parameter matrix is ​​used to describe the coordinate transformation relationship from the world coordinate system to the camera coordinate system.

[0076] For an autonomous vehicle, the world coordinate system may be, for example, the vehicle coordinate system, which may take the center of the rear axle of the vehicle as the origin, the right direction of the vehicle as the positive x-axis, the forward direction of the vehicle as the normal y-axis, and the vertically upward direction through the roof of the vehicle as the positive z-axis.

[0077] Using the parameters of the image acquisition device, the two-dimensional pixel coordinates can be converted into three-dimensional space coordinates in the world coordinate system.

[0078] According to some embodiments, step S221 may include steps S2211 and S2212.

[0079] In step S2211, a preset depth value set is obtained, where the depth value set includes a plurality of depth values.

[0080] In step S2212, for any depth value among the multiple depth values, based on the parameters of the image acquisition device and the depth value, the two-dimensional pixel coordinates of the pixel are transformed to determine the three-dimensional coordinates of the space point corresponding to the pixel. It can be understood that since the coordinate transformation is performed for each depth value in the depth value set, each depth value can calculate a corresponding space point, so each pixel in the feature map corresponds to multiple space points.

[0081] According to the above embodiment, a preset depth value set is used to complete the transformation of two-dimensional pixel coordinates to three-dimensional space coordinates. There is no need to perform depth estimation on the feature map, thus avoiding the adverse effects of inaccurate depth estimation information on three-dimensional object detection and ensuring the accuracy of three-dimensional object detection.

[0082] According to some embodiments, in step S2211, the depth values ​​included in the depth value set are obtained by sampling within the field of view of the image acquisition device. The depth value set is determined based on the field of view of the image acquisition device, so that the three-dimensional object detection situation is consistent with the actual perception situation of the image acquisition device, thereby improving the accuracy of three-dimensional object detection.

[0083] The field of view indicates the depth range that the image acquisition device can perceive. For example, if the field of view of an image acquisition device is 4 to 100 meters, it means that the image acquisition device can perceive objects within a range of 4 to 100 meters in front of it. Multiple depth values ​​can be obtained by uniformly sampling at certain intervals within the field of view of the image acquisition device. For example, by sampling in the range of [4,100] at intervals of 2 meters, 49 depth values ​​such as 4, 6, ..., 100 can be obtained.

[0084] According to some embodiments, in step S2212, for any depth value d among the multiple depth values, the two-dimensional pixel coordinates of the pixel may be transformed based on the intrinsic parameter matrix and the extrinsic parameter matrix of the image acquisition device to obtain the three-dimensional coordinates of the space point corresponding to the pixel. The three-dimensional coordinates of the space point may be, for example, three-dimensional coordinates in a world coordinate system.

[0085] Specifically, the two-dimensional pixel coordinates (u, v) of any pixel in the pixel coordinates of the feature map can be converted into three-dimensional coordinates (x, y, z) in the camera coordinate system of the corresponding image acquisition device by the following formula (1):

[0086]

[0087] In formula (1), K1 is the intrinsic parameter matrix of the image acquisition device corresponding to the feature map, d is the depth value, and f x is the product of the focal length f of the image acquisition device and the scaling factor α of the pixel plane relative to the imaging plane on the horizontal axis, c x is the translation of the pixel plane relative to the origin of the imaging plane on the horizontal axis, f y is the product of the focal length f of the image acquisition device and the scaling factor β of the pixel plane relative to the imaging plane on the ordinate axis, c y is the translation of the pixel plane relative to the origin of the imaging plane on the ordinate axis.

[0088] Furthermore, the three-dimensional coordinates (x, y, z) in the camera coordinate system can be further converted into the three-dimensional coordinates (x w ,y w , z w ):

[0089]

[0090] In formula (2), K2 is the external parameter matrix of the image acquisition device (camera).

[0091] The above equations (1) and (2) can be combined into:

[0092]

[0093] In formula (3), is the two-dimensional coordinate (u) of pixel j in the feature map of the image to be detected acquired by image acquisition device i j , v j ), K i1 , K i2 are the intrinsic parameter matrix and extrinsic parameter matrix of image acquisition device i, respectively. is the three-dimensional coordinate (x) of the spatial point corresponding to pixel j in the feature map of the image to be detected collected by image acquisition device i in the world coordinate system. w ,y w , z w ).

[0094] It is understandable that when multiple image acquisition devices are deployed at different positions of the intelligent body, the fields of view of different image acquisition devices may overlap in part. Therefore, a spatial point in the three-dimensional world space may be located in the images acquired by multiple image acquisition devices.

[0095] According to some other embodiments, in step S221, the depth value of each pixel can be estimated using a trained depth estimation network, and then the two-dimensional pixel coordinates of each pixel can be transformed using the parameters of the image acquisition device and the depth value to determine the three-dimensional coordinates of the space point corresponding to each pixel. The specific process of the coordinate transformation can refer to the above formulas (1)-(3), which will not be repeated here.

[0096] The corresponding relationship between the pixel and the spatial point is determined by step S221. In step S222, for each spatial point, the first feature vector of the pixel corresponding to the spatial point can be determined as the second feature vector of the spatial point.

[0097] After the second feature vector of each spatial point is obtained in step S222, in step S223, the first spatial feature of the target three-dimensional space can be determined based on the second feature vector of each spatial point. The first spatial feature can be, for example, a bird's-eye view feature.

[0098] According to some embodiments, the target three-dimensional space may be divided into a plurality of grids. The shape and size of the grids may be set as needed. For example, the target three-dimensional space may be divided into a plurality of grids of the same size, each grid may be a columnar space with a rectangular cross section (e.g., 0.5 m*0.5 m), and the height of the columnar space is the same as the height of the target three-dimensional space.

[0099] According to some embodiments, step S223 may include steps S2231 and S2232.

[0100] In step S2231, for any grid among the multiple grids, a third eigenvector of the grid is determined based on the second eigenvector of each spatial point in the grid.

[0101] In step S2232, the first spatial feature of the target three-dimensional space is determined based on the third feature vectors of each of the multiple grids.

[0102] According to the above embodiment, through grid division, the distribution of the third eigenvector is made more uniform and the position information expressed is more accurate, thereby improving the accuracy of the first spatial feature.

[0103] According to some embodiments, in step S2231, for each grid, the second eigenvectors of each spatial point in the grid may be aggregated to obtain the third eigenvector of the grid. For example, the weighted sum of the second eigenvectors of each spatial point in the grid may be used as the third eigenvector of the grid. The weights of the second eigenvectors of each spatial point may be the same or different.

[0104] According to the above embodiment, the third eigenvector of the grid is aggregated from the second eigenvectors of the spatial points in the grid, so that the third eigenvector can fully and accurately reflect the location information of the object, thereby improving the accuracy of the first spatial feature.

[0105] According to some embodiments, in step S2232, the third feature vectors of the plurality of grids may be concatenated to obtain the first spatial feature of the target three-dimensional space.

[0106] In step S230, the feature map is sampled based on the projection points of the reference points in the target three-dimensional space in the feature map to obtain the second spatial features of the target three-dimensional space.

[0107] According to some embodiments, step S230 may include steps S231 - S233 .

[0108] In step S231, based on the parameters of the image acquisition device that acquires the image to be detected, the reference point in the target three-dimensional space is projected into the feature map to obtain the projection point corresponding to the reference point.

[0109] In step S232, the feature map is sampled using the projection point to obtain a fourth eigenvector of the reference point.

[0110] In step S233, the second spatial feature of the target three-dimensional space is determined based on the fourth feature vector of the reference point.

[0111] According to the above embodiment, by projecting a three-dimensional space point (reference point) into a two-dimensional feature map, the feature map is sampled using the obtained projection point to obtain a second spatial feature, so that the second spatial feature can simulate the position mapping relationship from the three-dimensional space to the two-dimensional image.

[0112] There can be multiple reference points in the target three-dimensional space. By projecting each reference point, the projection point corresponding to the reference can be obtained.

[0113] According to some embodiments, random sampling may be performed within a coordinate range of the target three-dimensional space to obtain a plurality of reference points.

[0114] According to other embodiments, the target three-dimensional space can be divided into multiple grids. As described above, each grid can be a columnar space with a rectangular cross-section (e.g., 0.5m*0.5m), and the height of the columnar space is the same as the height of the target three-dimensional space. One or more spatial points can be sampled in each grid as reference points. For example, the spatial points at specific positions of each grid (e.g., the center, upper left corner, upper right corner, etc.) can be used as reference points, or reference points can be obtained by random sampling within the grid.

[0115] According to some embodiments, each grid can be further divided into multiple sub-grids along the height direction. As described above, each grid can be a cylindrical space with a rectangular cross-section. The grid is evenly divided along the height direction to obtain a sub-grid shaped like a cuboid. For example, the target three-dimensional space can be a 100m*100m*4m space around the autonomous driving vehicle. Each grid can be a cylindrical space of 0.5m*0.5m*4m. Each grid is evenly divided in units of 0.5m along the height direction, and each grid can be divided into 8 0.5m*0.5m*0.5m cubic spaces. One or more spatial points can be sampled in each sub-grid as reference points. For example, a spatial point at a specific position (such as the center, upper left corner, upper right corner, etc.) of each sub-grid can be used as a reference point, or a reference point can be obtained by random sampling in the sub-grid.

[0116] According to some embodiments, in step S231, the three-dimensional coordinates of the reference point in the world coordinate system can be converted into two-dimensional pixel coordinates in the feature map using the above formula (3), that is, the projection point corresponding to the reference point is obtained. It should be noted that the coordinates of the projection point calculated by projection are usually not integers, so the projection point does not exactly correspond to a pixel in the feature map, and the feature vector of the projection point, that is, the fifth feature vector, cannot be directly obtained from the feature map.

[0117] In order to obtain the fifth eigenvector of the projection point and further obtain the fourth eigenvector of the reference point when the coordinates of the projection point are not integers, it is necessary to sample the feature map using the projection points in step S232.

[0118] According to some embodiments, step S232 may include steps S2321 - S2323 .

[0119] In step S2321, multiple target pixels adjacent to the projection point are obtained from the feature map. For example, if the coordinates of the projection point are (u, v), the two integers floor(u) and ceil(u) closest to u and the two integers floor(v) and ceil(v) closest to v can be combined to obtain four target pixels (floor(u), floor(v)), (floor(u), ceil(v)), (ceil(u), floor(v)) and (ceil(u), ceil(v)). Where floor and ceil represent rounding down and rounding up, respectively.

[0120] In step S2322, the fifth eigenvector of the projection point is obtained by interpolating between the first eigenvectors of the plurality of target pixels. For example, when four target pixels are obtained in step S2321, the fifth eigenvector of the projection point can be obtained by bilinear interpolation.

[0121] In step S2323, the fifth eigenvector of the projection point is determined as the fourth eigenvector of the corresponding reference point.

[0122] As described above, the coordinates of the projection point obtained by the projection calculation are not necessarily integers, so the fifth eigenvector of the projection point cannot be directly obtained from the feature map. According to the above embodiment, the fifth eigenvector of the projection point is obtained by interpolating the first eigenvectors of the adjacent pixels (i.e., the target pixels) of the projection point, which can improve the accuracy of the fifth eigenvector and enable it to accurately reflect the position information of the object.

[0123] After the fourth eigenvector of the reference point is obtained in step S232, in step S233, the second spatial feature of the target three-dimensional space may be determined based on the fourth eigenvector of the reference point.

[0124] According to some embodiments, the fourth eigenvectors of the plurality of reference points may be concatenated to obtain the second spatial feature of the target three-dimensional space.

[0125] According to some embodiments, when the target three-dimensional space is divided into multiple grids and the reference point is a spatial point in any grid of the multiple grids, step S233 may further include steps S2331 and S2332.

[0126] In step S2331, for any grid among the multiple grids, a sixth eigenvector of the grid is determined based on a fourth eigenvector of a reference point in the grid.

[0127] Specifically, when the grid includes only one reference point, the fourth eigenvector of the reference point can be determined as the sixth eigenvector of the grid. When the grid includes multiple reference points, the weighted sum of the fourth eigenvectors of the multiple reference points can be determined as the sixth eigenvector of the grid. The weights of the reference points can be the same or different.

[0128] In step S2332, the second spatial feature is determined based on the sixth eigenvectors of the plurality of grids. For example, the sixth eigenvectors of the plurality of grids may be concatenated to obtain the second spatial feature of the target three-dimensional space.

[0129] According to the above embodiment, grid division makes the distribution of reference points more uniform and the position information expressed by the sixth eigenvector of the grid more accurate, thereby improving the accuracy of the second spatial feature.

[0130] According to some embodiments, when each grid is divided into multiple subgrids along the height direction and the reference point is a spatial point in any of the multiple subgrids, step S2331 may further include: for any grid among the multiple grids, obtaining the fourth eigenvector of the reference point of each of the multiple subgrids of the grid; aggregating the fourth eigenvectors of the reference points of each of the multiple subgrids to obtain the sixth eigenvector of the grid. Specifically, the weighted sum of the fourth eigenvectors of the reference points of each of the multiple subgrids may be used as the sixth eigenvector of the grid.

[0131] According to the above embodiment, the sixth eigenvector of the grid is aggregated from the fourth eigenvectors of the sub-grids within the grid, thereby enabling the sixth eigenvector to fully and accurately reflect the location information of the object.

[0132] After the first spatial feature and the second spatial feature of the target three-dimensional space are obtained through steps S220 and S230, step S240 is executed.

[0133] In step S240, the first spatial feature and the second spatial feature are fused to obtain a fused spatial feature.

[0134] According to some embodiments, a weighted sum of the first spatial feature and the second spatial feature may be used as the fused spatial feature. The weights of the first spatial feature and the second spatial feature may be the same or different.

[0135] According to some embodiments, the first spatial feature and the second spatial feature may be concatenated, and then the concatenated spatial feature may be reduced in dimension using a linear layer to obtain a fused spatial feature, wherein the size of the fused spatial feature is the same as the size of the first spatial feature and the second spatial feature.

[0136] In step S250, a three-dimensional object detection result of the image to be detected is determined based on the fused spatial features.

[0137] According to some embodiments, the fused spatial features may be input into a trained decoder to obtain a three-dimensional object detection result output by the decoder. The three-dimensional object detection result includes three-dimensional properties of the target object in the image to be detected, such as the length, width, height, coordinates of the 3D center point, orientation angle, category, etc. of the target object.

[0138] The decoder can be implemented as a convolutional layer, a fully connected layer, and a classification layer (Softmax layer), for example.

[0139] According to some embodiments, the feature extraction network and decoder described above can be combined to form an end-to-end three-dimensional object detection model, and trained using sample images labeled with three-dimensional attribute labels of the target object. During the training process of the model, the sample image is input into the feature extraction network to obtain a feature map output by the feature extraction network. The feature map is calculated through the above steps S220 and S230 to obtain the first spatial feature and the second spatial feature of the target three-dimensional space. The first spatial feature and the second spatial feature are fused through the above step S240 to obtain the fused spatial feature. The fused spatial feature is input into the decoder to obtain the predicted three-dimensional attribute of the target object output by the decoder. Based on the predicted three-dimensional attribute and the three-dimensional attribute label of the target object, the loss of the three-dimensional object detection model is calculated. Based on the loss, the parameters of the three-dimensional object detection model are adjusted, that is, the parameters of the feature extraction network and the decoder are adjusted. Until the preset termination condition is reached, the training process of the model is terminated to obtain a trained three-dimensional object detection model. Accordingly, the three-dimensional object detection method 200 of the embodiment of the present disclosure can be implemented using a trained three-dimensional object detection model.

[0140] Figure 3 A schematic diagram of a three-dimensional object detection model 300 and a detection process thereof according to an embodiment of the present disclosure is shown.

[0141] like Figure 3 As shown, the model 300 includes a feature extraction network 310 , a 2D to 3D feature projection module 320 , a 3D to 2D feature sampling module 330 , a fusion layer 340 , and a decoder 350 .

[0142] The feature extraction network 310 takes the environment images (i.e., the images to be detected in the method 200) 301 collected by the cameras of the autonomous driving vehicle at six different viewing angles as input, and outputs a feature map 302 of each environment image. The feature extraction network 310 may be, for example, ResNet, FPN, etc.

[0143] The size of the environment image 301 may be, for example, W*H*3, and the size of the feature map 302 may be, for example, (W / 8)*(H / 8)*C. In the feature map 302, each pixel corresponds to a C-dimensional feature vector (ie, the first feature vector in the method 200).

[0144] The feature projection module 320 is used to implement 2D-to-3D feature pushing (2D-to-3DPushing). The feature projection module 320 uses the intrinsic parameter matrix and extrinsic parameter matrix of each camera, and projects the feature map 302 of the environment image captured by the camera into the BEV space (i.e., the target three-dimensional space in method 200) 303 based on the preset depth value set corresponding to the camera. The BEV space can be, for example, a rectangular space of X*Y*Z=100m*100m*4m.

[0145] Specifically, the feature projection module 320 can calculate the three-dimensional coordinates in the world coordinate system corresponding to the pixel coordinates in each feature map according to the above formula (3), thereby obtaining the correspondence between the pixels in the feature map 302 and the spatial points in the world coordinate system. Then, the feature vector of each spatial point (i.e., the second feature vector in the method 200) can be set as the feature vector of the pixel corresponding to the spatial point. The dimension of the feature vector of the spatial point is the same as that of the feature vector of the pixel, both of which are C dimensions.

[0146] The BEV space can be divided into columnar BEV grids (corresponding to the grids above) with a cross-sectional size of i*j and a height of Z. i and j can be, for example, both 0.5 m. Accordingly, the BEV space is divided into 200*200 columnar BEV grids. If the size of the BEV space is expressed by the number of BEV grids, the size of the BEV space is W. BEV *H BEV =200*200.

[0147] For each BEV grid, the feature vectors of the spatial points in the BEV grid are aggregated (eg, averaged) to obtain the feature vector of the BEV grid (ie, the third feature vector in method 200). The feature vector of the BEV grid is also a C-dimensional vector.

[0148] Then, the feature vectors of each BEV grid are concatenated to obtain the first BEV feature of the BEV space (i.e., the first space feature in method 200) 304, which can be recorded as First BEV Features The size is W BEV *H BEV *C.

[0149] The feature sampling module 330 is used to implement 3D-to-2D Pulling. The feature sampling module 330 divides the BEV space 303 into i, j, and k along the X, Y, and Z directions, respectively, to obtain multiple rectangular grids. i, j, and k can be, for example, all 0.5 m. Accordingly, the BEV space is divided into 200*200*8 cube-shaped initial BEV grids (corresponding to the subgrids above). If the number of initial BEV grids is used to represent the size of the BEV space, then the size of the BEV space is W. BEV *H BEV *Z BEV =200*200*8. The center point of each initial BEV grid can be used as the reference point of the grid.

[0150] The feature sampling module 330 uses the intrinsic parameter matrix and extrinsic parameter matrix of each camera, and according to the above formula (3), projects the reference point of each initial BEV grid into the feature map 302 corresponding to the camera to obtain the pixel coordinates of the projection point. Subsequently, based on the pixel coordinates of the projection point, the feature map 302 is sampled by bilinear interpolation to obtain the feature vector of the projection point (i.e., the fifth feature vector in method 200), which is the feature vector of the corresponding reference point (i.e., the fourth feature vector in method 200), that is, the feature vector of the corresponding initial BEV grid. The feature vector of the initial BEV grid is a C-dimensional vector. The above process can be expressed as the following formula (4):

[0151]

[0152] in, is the eigenvector of the initial BEV grid, is the pixel coordinate of the projection point, F is the feature map, and GridSampling() is the sampling function.

[0153] After obtaining the feature vector of each initial BEV grid, average pooling is performed on each initial BEV grid along the height direction (Z direction), that is, the feature vectors of multiple (8) initial BEV grids with the same X*Y coordinates are averaged to obtain the C-dimensional feature vector (corresponding to the sixth feature vector in method 200) of the columnar BEV grid (corresponding to the grid in method 200). BEV *H BEV *Z BEV The initial BEV grid is reduced to W BEV *H BEV A BEV grid.

[0154] Then, the feature vectors of each BEV grid are concatenated to obtain the second BEV feature of the BEV space (i.e., the second space feature in method 200) 305, which can be recorded as Second BEV Characteristics The size is W BEV *H BEV *C.

[0155] The fusion layer 340 fuses the first BEV feature 304 and the second BEV feature 305 to obtain a fused BEV feature E BEV The fusion layer 340 may, for example, fuse the two BEV features according to the following formula (5):

[0156]

[0157] Among them, cat() is a vector concatenation function, and Linear() is a linear layer, which is used to reduce the feature dimension, that is, to reduce the concatenated 2C-dimensional vector to C dimension.

[0158] Decoder 350 is used to fuse BEV features E BEV The 3D object detection results of each environment image are output as input. The 3D object detection results include the 3D attributes of the target object (such as vehicles, pedestrians, etc.), such as the length, width, height, coordinates of the 3D center point, orientation angle, category, etc. of the target object.

[0159] The decoder can be implemented as a convolutional layer, a fully connected layer, and a classification layer (Softmax layer), for example.

[0160] According to an embodiment of the present disclosure, a three-dimensional object detection device is also provided. Figure 4 FIG. 4 shows a structural block diagram of a three-dimensional object detection device 400 according to an embodiment of the present disclosure. Figure 4 As shown, the apparatus 400 includes an extraction module 410 , a projection module 420 , a sampling module 430 , a fusion module 440 and a determination module 450 .

[0161] The extraction module 410 is configured to extract a feature map of the image to be detected.

[0162] The projection module 420 is configured to project the feature map into a target three-dimensional space to obtain a first spatial feature of the target three-dimensional space.

[0163] The sampling module 430 is configured to sample the feature map based on the projection point of the reference point in the target three-dimensional space in the feature map to obtain the second spatial feature of the target three-dimensional space.

[0164] The fusion module 440 is configured to fuse the first spatial feature and the second spatial feature to obtain a fused spatial feature.

[0165] The determination module 450 is configured to determine a three-dimensional object detection result of the image to be detected based on the fused spatial features.

[0166] According to an embodiment of the present disclosure, three-dimensional object detection is achieved based on bidirectional feature projection. By projecting a two-dimensional feature map into the target three-dimensional space, a first spatial feature of the target three-dimensional space is obtained, so that the first spatial feature can simulate the position mapping relationship from the two-dimensional image to the three-dimensional space. By sampling from the three-dimensional space to the two-dimensional image, a second spatial feature of the target three-dimensional space is obtained, so that the second spatial feature can simulate the position mapping relationship from the three-dimensional space to the two-dimensional image. The first spatial feature is fused with the second spatial feature, and the obtained fused spatial feature can simultaneously express the position mapping relationship from the two-dimensional image to the three-dimensional space and from the three-dimensional space to the two-dimensional image, thereby achieving accurate mining of the object position relationship and being able to fully and accurately reflect the three-dimensional position information of the object. Performing three-dimensional object detection based on fused spatial features can improve the accuracy of three-dimensional object detection.

[0167] According to some embodiments, the feature map includes first feature vectors of each of a plurality of pixels, and the projection module includes: a first determination unit, configured to determine the spatial point corresponding to each pixel in the feature map based on parameters of an image acquisition device that acquires the image to be detected; a second determination unit, configured to determine the second feature vector of the spatial point corresponding to the pixel based on the first feature vector of the pixel; and a third determination unit, configured to determine the first spatial feature based on the second feature vector of the spatial point corresponding to each of the plurality of pixels.

[0168] According to some embodiments, the first determination unit includes: a first acquisition subunit, configured to acquire a preset depth value set, the depth value set including multiple depth values; and a transformation subunit, configured to perform a coordinate transformation on the two-dimensional pixel coordinates of the pixel for any depth value among the multiple depth values ​​based on parameters of the image acquisition device and the depth value, so as to determine the three-dimensional coordinates of the spatial point corresponding to the pixel.

[0169] According to some embodiments, the multiple depth values ​​are obtained by sampling within the field of view of the image acquisition device.

[0170] According to some embodiments, the target three-dimensional space is divided into multiple grids, and the third determination unit includes: a first determination subunit, configured to determine, for any grid among the multiple grids, the third eigenvector of the grid based on the second eigenvectors of each spatial point within the grid; and a second determination subunit, configured to determine the first spatial feature based on the third eigenvectors of each of the multiple grids.

[0171] According to some embodiments, the first determining subunit is further configured to: aggregate the second eigenvectors of the spatial points in the grid to obtain a third eigenvector of the grid.

[0172] According to some embodiments, the feature map includes a first feature vector of each of a plurality of pixels, and the sampling module includes: a projection unit, configured to project the reference point into the feature map based on parameters of an image acquisition device that acquires the image to be detected, so as to obtain a projection point corresponding to the reference point; a sampling unit, configured to sample the feature map using the projection point, so as to obtain a fourth feature vector of the reference point; and a fourth determination unit, configured to determine the second spatial feature based on the fourth feature vector of the reference point.

[0173] According to some embodiments, the sampling unit includes: a second acquisition subunit, configured to acquire a plurality of target pixels adjacent to the projection point from the feature map; an interpolation subunit, configured to obtain a fifth feature vector of the projection point by interpolating between the respective first feature vectors of the plurality of target pixels; and a third determination subunit, configured to determine the fifth feature vector of the projection point as the fourth feature vector of the reference point.

[0174] According to some embodiments, the target three-dimensional space is divided into multiple grids, the reference point is a spatial point within any one of the multiple grids, and the fourth determination unit includes: a fourth determination subunit, configured to determine, for any one of the multiple grids, a sixth eigenvector of the grid based on a fourth eigenvector of the reference point within the grid; and a fifth determination subunit, configured to determine the second spatial feature based on the sixth eigenvectors of each of the multiple grids.

[0175] According to some embodiments, the grid is divided into multiple sub-grids along the height direction, the reference point is a spatial point within any sub-grid of the multiple sub-grids, and the fourth determination subunit is further configured to: for any grid of the multiple grids, obtain the fourth eigenvector of the reference points of the multiple sub-grids of the grid; aggregate the fourth eigenvectors of the reference points of the multiple sub-grids to obtain the sixth eigenvector of the grid.

[0176] According to some embodiments, the fusion module is further configured to: input the fused spatial features into a trained decoder to obtain the three-dimensional object detection result output by the decoder.

[0177] According to some embodiments, the image to be detected is acquired by an image acquisition device disposed on an intelligent body, and the target three-dimensional space is determined according to a region of interest of the intelligent body.

[0178] According to some embodiments, the image to be detected includes an environment image acquired by a plurality of image acquisition devices, and the plurality of image acquisition devices are arranged at different positions of the intelligent body.

[0179] It should be understood that Figure 4 The various modules and units of the apparatus 400 shown in FIG. 4 can be compared with those of the reference Figure 2 The steps in the method 200 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0180] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module.

[0181] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 4 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of modules 410-450 can be implemented together in a system on chip (System on Chip, SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.

[0182] According to one aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned three-dimensional object detection method 200.

[0183] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned three-dimensional object detection method 200.

[0184] According to one aspect of the present disclosure, a computer program product is also provided, including a computer program, which implements the above-mentioned three-dimensional object detection method 200 when executed by a processor.

[0185] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0186] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0187] Multiple components in the device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the device 500. The input unit 506 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0188] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the three-dimensional object detection method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 200 in any other appropriate manner (e.g., by means of firmware).

[0189] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0190] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0191] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0193] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0194] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0195] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0196] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples. Various elements in the embodiments or examples may be omitted or may be replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A three-dimensional object detection method, comprising: Extracting a feature map of the image to be detected, wherein the feature map includes first feature vectors of each of a plurality of pixels; Projecting the feature map into a target three-dimensional space to obtain a first spatial feature of the target three-dimensional space, wherein the first spatial feature expresses a position mapping relationship from a two-dimensional image to a three-dimensional space; Based on the projection point of the reference point in the target three-dimensional space in the feature map, sampling the feature map to obtain a second spatial feature of the target three-dimensional space, where the second spatial feature expresses a position mapping relationship from the three-dimensional space to the two-dimensional image, and sampling the feature map based on the projection point of the reference point in the target three-dimensional space in the feature map to obtain the second spatial feature of the target three-dimensional space includes: Based on the parameters of the image acquisition device that acquires the image to be detected, the reference point is projected onto the feature map to convert the three-dimensional coordinates of the reference point into two-dimensional pixel coordinates in the feature map to obtain a projection point corresponding to the reference point; Sampling the feature map using the projection point to obtain a fourth feature vector of the reference point; and determining the second spatial feature based on a fourth eigenvector of the reference point; fusing the first spatial feature and the second spatial feature to obtain a fused spatial feature, wherein the fused spatial feature simultaneously expresses a position mapping relationship from a two-dimensional image to a three-dimensional space and from a three-dimensional space to a two-dimensional image; and Based on the fused spatial features, a three-dimensional object detection result of the image to be detected is determined.

2. The method according to claim 1, wherein: The feature map includes first feature vectors of each of the plurality of pixels, and projecting the feature map into the target three-dimensional space to obtain the first spatial feature of the target three-dimensional space includes: Determine the spatial point corresponding to each pixel in the feature map based on the parameters of the image acquisition device that acquires the image to be detected; Based on the first eigenvector of the pixel, determining a second eigenvector of the spatial point corresponding to the pixel; and The first spatial feature is determined based on second feature vectors of spatial points corresponding to each of the multiple pixels.

3. The method according to claim 2, wherein: The determining, based on the parameters of the image acquisition device that acquires the image to be detected, the spatial point corresponding to each pixel in the feature map comprises: Acquire a preset depth value set, where the depth value set includes a plurality of depth values; and For any depth value among the multiple depth values, based on the parameters of the image acquisition device and the depth value, coordinate transformation is performed on the two-dimensional pixel coordinates of the pixel to determine the three-dimensional coordinates of the space point corresponding to the pixel.

4. The method according to claim 3, wherein: The multiple depth values ​​are obtained by sampling within the field of view of the image acquisition device.

5. The method according to any one of claims 2 to 4, wherein: The target three-dimensional space is divided into a plurality of grids, and the determining the first spatial feature based on the second feature vectors of the spatial points corresponding to the plurality of pixels respectively comprises: For any grid among the plurality of grids, determining a third eigenvector of the grid based on the second eigenvector of each spatial point in the grid; and The first spatial feature is determined based on the third feature vectors of each of the plurality of grids.

6. The method according to claim 5, wherein: Determining the third eigenvector of the grid based on the second eigenvector of each spatial point in the grid comprises: The second eigenvectors of the spatial points in the grid are aggregated to obtain the third eigenvector of the grid.

7. The method according to claim 1, wherein: The sampling of the feature map by using the projection point to obtain a fourth feature vector of the reference point includes: Acquire a plurality of target pixels adjacent to the projection point from the feature map; Obtaining a fifth eigenvector of the projection point by interpolating between the first eigenvectors of the plurality of target pixels; and The fifth eigenvector of the projection point is determined as the fourth eigenvector of the reference point.

8. The method according to claim 1 or 7, wherein: The target three-dimensional space is divided into a plurality of grids, the reference point is a spatial point in any grid of the plurality of grids, and the determining the second spatial feature based on the fourth eigenvector of the reference point includes: For any grid among the plurality of grids, determining a sixth eigenvector of the grid based on a fourth eigenvector of a reference point within the grid; and The second spatial feature is determined based on the sixth feature vectors of each of the plurality of grids.

9. The method according to claim 8, wherein: The grid is divided into a plurality of subgrids along a height direction, the reference point is a spatial point in any subgrid of the plurality of subgrids, and determining the sixth eigenvector of the grid based on the fourth eigenvector of the reference point in the grid comprises: For any grid among the multiple grids, obtaining fourth eigenvectors of reference points of multiple subgrids of the grid; Aggregate the fourth eigenvectors of the reference points of the multiple sub-grids to obtain a sixth eigenvector of the grid.

10. The method according to any one of claims 1 to 9, wherein: Determining the three-dimensional object detection result of the image to be detected based on the fused spatial feature includes: The fused spatial features are input into a trained decoder to obtain the three-dimensional object detection result output by the decoder.

11. The method according to any one of claims 1 to 10, wherein: The image to be detected is collected by an image acquisition device arranged on the intelligent body, and the target three-dimensional space is determined according to the region of interest of the intelligent body.

12. The method according to any one of claims 1 to 11, wherein: The image to be detected includes an environment image acquired by a plurality of image acquisition devices, and the plurality of image acquisition devices are arranged at different positions of the intelligent body.

13. A three-dimensional object detection device, comprising: An extraction module is configured to extract a feature map of the image to be detected, wherein the feature map includes first feature vectors of each of a plurality of pixels; A projection module is configured to project the feature map into a target three-dimensional space to obtain a first spatial feature of the target three-dimensional space, wherein the first spatial feature expresses a position mapping relationship from a two-dimensional image to a three-dimensional space; A sampling module is configured to sample the feature map based on a projection point of a reference point in the target three-dimensional space in the feature map to obtain a second spatial feature of the target three-dimensional space, wherein the second spatial feature expresses a position mapping relationship from the three-dimensional space to the two-dimensional image, and the sampling module includes: A projection unit is configured to project the reference point onto the feature map based on parameters of an image acquisition device that acquires the image to be detected, so as to convert the three-dimensional coordinates of the reference point into two-dimensional pixel coordinates in the feature map to obtain a projection point corresponding to the reference point; a sampling unit configured to sample the feature map using the projection point to obtain a fourth feature vector of the reference point; and a fourth determining unit, configured to determine the second spatial feature based on a fourth feature vector of the reference point; a fusion module configured to fuse the first spatial feature and the second spatial feature to obtain a fused spatial feature, wherein the fused spatial feature simultaneously expresses a position mapping relationship from a two-dimensional image to a three-dimensional space and from a three-dimensional space to a two-dimensional image; and The determination module is configured to determine the three-dimensional object detection result of the image to be detected based on the fused spatial features.

14. The device according to claim 13, wherein: The feature map includes first feature vectors of each of the plurality of pixels, and the projection module includes: A first determining unit is configured to determine a spatial point corresponding to each pixel in the feature map based on parameters of an image acquisition device that acquires the image to be detected; A second determining unit is configured to determine a second eigenvector of a spatial point corresponding to the pixel based on the first eigenvector of the pixel; and The third determining unit is configured to determine the first spatial feature based on the second feature vectors of the spatial points corresponding to each of the multiple pixels.

15. The device according to claim 14, wherein: The first determining unit includes: A first acquisition subunit is configured to acquire a preset depth value set, where the depth value set includes a plurality of depth values; and The transformation subunit is configured to perform coordinate transformation on the two-dimensional pixel coordinates of the pixel for any depth value among the multiple depth values ​​based on the parameters of the image acquisition device and the depth value, so as to determine the three-dimensional coordinates of the spatial point corresponding to the pixel.

16. The device according to claim 15, wherein: The multiple depth values ​​are obtained by sampling within the field of view of the image acquisition device.

17. The device according to any one of claims 14 to 16, wherein: The target three-dimensional space is divided into a plurality of grids, and the third determining unit includes: A first determining subunit is configured to determine, for any grid among the plurality of grids, a third eigenvector of the grid based on the second eigenvectors of each spatial point in the grid; and The second determining subunit is configured to determine the first spatial feature based on the third feature vectors of each of the plurality of grids.

18. The device according to claim 17, wherein: The first determining subunit is further configured to: The second eigenvectors of the spatial points in the grid are aggregated to obtain the third eigenvector of the grid.

19. The device according to claim 13, wherein: The sampling unit comprises: A second acquisition subunit is configured to acquire a plurality of target pixels adjacent to the projection point from the feature map; an interpolation subunit configured to obtain a fifth eigenvector of the projection point by interpolating between the first eigenvectors of the plurality of target pixels; and The third determining subunit is configured to determine the fifth eigenvector of the projection point as the fourth eigenvector of the reference point.

20. The device according to claim 13 or 19, wherein: The target three-dimensional space is divided into a plurality of grids, the reference point is a spatial point in any grid of the plurality of grids, and the fourth determining unit includes: a fourth determining subunit configured to determine, for any grid among the plurality of grids, a sixth eigenvector of the grid based on a fourth eigenvector of a reference point within the grid; and The fifth determining subunit is configured to determine the second spatial feature based on the sixth feature vectors of each of the multiple grids.

21. The device according to claim 20, wherein: The grid is divided into a plurality of sub-grids along a height direction, the reference point is a spatial point in any sub-grid of the plurality of sub-grids, and the fourth determining sub-unit is further configured as follows: For any grid among the multiple grids, obtaining fourth eigenvectors of reference points of multiple subgrids of the grid; Aggregate the fourth eigenvectors of the reference points of the multiple sub-grids to obtain a sixth eigenvector of the grid.

22. The device according to any one of claims 13 to 21, wherein: The fusion module is further configured to: The fused spatial features are input into a trained decoder to obtain the three-dimensional object detection result output by the decoder.

23. The device according to any one of claims 13 to 22, wherein: The image to be detected is collected by an image acquisition device arranged on the intelligent body, and the target three-dimensional space is determined according to the region of interest of the intelligent body.

24. The device according to any one of claims 13 to 23, wherein: The image to be detected includes an environment image acquired by a plurality of image acquisition devices, and the plurality of image acquisition devices are arranged at different positions of the intelligent body.

25. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

Images