A 3D target detection method and device based on multi-view fusion

Abstract

The embodiments of the present disclosure disclose a 3D target detection method and device based on multi-view fusion. In this method, feature extraction is performed on at least one image of the multi-camera perspective captured by the multi-camera system, and based on the internal parameters and vehicle parameters of the multi-camera system, the extracted feature data containing the target object features in the multi-camera perspective space is mapped to the same bird's-eye view space, and the feature data corresponding to each of at least one image in the bird's-eye view space is obtained, and the bird's-eye view fusion feature is obtained by feature fusion. Target prediction is performed on the target object in the bird's-eye view fusion feature to obtain the three-dimensional spatial information of the target object. When performing 3D target detection based on multi-view fusion through the solution of the embodiments of the present disclosure, multi-view feature fusion is performed first and then 3D target detection is performed, and the 3D detection of scene objects in the bird's-eye view is completed end-to-end, thereby improving detection efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a 3D object detection method and device based on multi-view fusion. Background Art

[0002] With the development of science and technology, the application of autonomous driving technology in people's lives is becoming more and more extensive. The autonomous driving carrier can perform 3D detection on target objects (vehicles, pedestrians, cyclists, etc.) within a certain distance to obtain the three-dimensional spatial information of the target object. Based on the three-dimensional spatial information of the target object, the distance and speed of the target object can be measured to achieve better driving control.

[0003] At present, autonomous driving vehicles can collect multiple images with different perspectives, and then perform 3D detection on each image separately. Finally, the 3D detection results of each image are fused to generate three-dimensional spatial information of target objects in the vehicle's surrounding environment. Summary of the invention

[0004] The existing technical solution requires 3D detection of each image collected by the autonomous driving vehicle, and then the 3D detection results of each image are fused to obtain information about other vehicles in the 360-degree environment around the vehicle, resulting in low detection efficiency.

[0005] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a 3D object detection method and device based on multi-view fusion.

[0006] According to one aspect of the present disclosure, a 3D object detection method based on multi-view fusion is provided, comprising:

[0007] Acquire at least one image collected from multiple camera perspectives;

[0008] Performing feature extraction on the at least one image to obtain feature data containing target object features corresponding to each of the at least one image in a multi-camera viewing space;

[0009] Based on internal parameters of the multi-camera system and vehicle parameters, mapping feature data corresponding to each of the at least one image in the multi-camera viewing space to the same bird's-eye view viewing space to obtain feature data corresponding to each of the at least one image in the bird's-eye view viewing space;

[0010] Performing feature fusion on the feature data corresponding to each of the at least one image in the bird's-eye view space to obtain a bird's-eye view fusion feature;

[0011] Target prediction is performed on the target object in the bird's-eye view fusion feature to obtain three-dimensional spatial information of the target object.

[0012] According to another aspect of the present disclosure, a 3D object detection device based on multi-view fusion is provided, comprising:

[0013] An image receiving module, used to acquire at least one image from the perspective of multiple cameras;

[0014] A feature extraction module, used to extract features from the at least one image acquired by the image receiving module, and obtain feature data containing target object features corresponding to each of the at least one image in a multi-camera viewing space;

[0015] An image feature mapping module, used to map the feature data corresponding to each of the at least one image in the multi-camera viewing space obtained by the feature extraction module to the same bird's-eye view viewing space based on the internal parameters of the multi-camera system and the vehicle parameters, so as to obtain the feature data corresponding to each of the at least one image in the bird's-eye view viewing space;

[0016] An image fusion module, used for fusing the feature data corresponding to each of the at least one image obtained by the image mapping module in the bird's-eye view space to obtain a bird's-eye view fusion feature;

[0017] The 3D detection module is used to perform target prediction on the target object in the bird's-eye view fusion feature obtained by the image fusion module to obtain the three-dimensional spatial information of the target object.

[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the above-mentioned 3D object detection method based on multi-view fusion.

[0019] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0020] processor;

[0021] a memory for storing instructions executable by the processor;

[0022] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned 3D object detection method based on multi-view fusion.

[0023] Based on the above-mentioned embodiment of the present disclosure, a 3D target detection method and device based on multi-view fusion is provided. Feature extraction is performed on at least one image of the multi-camera perspective collected by the multi-camera system, and based on the internal parameters of the multi-camera system, the extracted feature data containing the target object features in the multi-camera perspective space is mapped to the same bird's-eye view space to obtain the feature data corresponding to each of the at least one image in the bird's-eye view space, and the feature data corresponding to each of the at least one image in the bird's-eye view space is feature fused to obtain the bird's-eye view fusion feature. Then, target prediction is performed on the target object in the bird's-eye view fusion feature to obtain the three-dimensional spatial information of the target object. When 3D target detection based on multi-view fusion is performed through the solution of the embodiment of the present disclosure, multi-view feature fusion is performed first and then 3D target detection is performed, and the 3D target detection of scene objects in the bird's-eye view is completed end-to-end, avoiding the post-processing stage in conventional multi-view 3D detection, and improving detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 It is a scene graph to which the present disclosure is applicable.

[0026] Figure 2 It is a system block diagram of the vehicle-mounted automatic driving system provided by an embodiment of the present disclosure.

[0027] Figure 3 It is a flowchart of a 3D object detection method based on multi-view fusion provided by an exemplary embodiment of the present disclosure.

[0028] Figure 4 It is a schematic block diagram of a multi-camera system capturing images according to an exemplary embodiment of the present disclosure.

[0029] Figure 5 FIG. 4 is a schematic diagram of images from multiple camera perspectives provided by an exemplary embodiment of the present disclosure.

[0030] Figure 6 It is a schematic block diagram of feature extraction provided by an exemplary embodiment of the present disclosure.

[0031] Figure 7 It is a schematic diagram of generating a bird's-eye view image from images acquired by a multi-camera system according to an exemplary embodiment of the present disclosure.

[0032] Figure 8 is a schematic block diagram of target detection provided by an exemplary embodiment of the present disclosure.

[0033] Fig. 9 It is a flowchart of determining feature data in a bird's-eye view space provided by an exemplary embodiment of the present disclosure.

[0034] Fig.10 It is a schematic block diagram of executing step S303 and step S304 provided by an exemplary embodiment of the present disclosure.

[0035] Fig.11 is a flowchart of target detection provided by an exemplary embodiment of the present disclosure.

[0036] Fig.12 It is a schematic diagram of the output result of the prediction network provided by an exemplary embodiment of the present disclosure.

[0037] Fig.13 is another flowchart of target detection provided by an exemplary embodiment of the present disclosure.

[0038] Fig.14 is a schematic diagram of a Gaussian kernel provided by an exemplary embodiment of the present disclosure.

[0039] Fig.15 is a schematic diagram of a heat map provided by an exemplary embodiment of the present disclosure.

[0040] Fig.16 is another flowchart of target detection provided by an exemplary embodiment of the present disclosure.

[0041] Fig.17 It is a structural diagram of a 3D object detection device based on multi-view fusion provided by an exemplary embodiment of the present disclosure.

[0042] Fig.18 is another structural diagram of a 3D object detection device based on multi-view fusion provided by an exemplary embodiment of the present disclosure.

[0043] Fig.19 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0045] Application Overview

[0046] To ensure safety during autonomous driving, the autonomous driving carrier can perform real-time detection of target objects (e.g., vehicles, pedestrians, cyclists, etc.) within a certain distance around the carrier to obtain the three-dimensional spatial information of the 3D target object (e.g., location, size, orientation angle, category, and other attributes). The distance and speed of the target object are measured based on the three-dimensional spatial information of the target object to achieve better driving control. Among them, the autonomous driving carrier can be a vehicle, an aircraft, etc.

[0047] The autonomous driving vehicle can use a multi-camera system to collect multiple images with different perspectives, and then perform 3D target detection on each image separately, such as filtering and deduplication of target objects on multiple images collected by cameras with different perspectives. Finally, the 3D detection results of each image are fused to generate the three-dimensional spatial information of the target object in the vehicle's surrounding environment. It can be seen that the existing technical solutions require 3D detection of each image collected by the autonomous driving vehicle separately, and then the 3D detection results of each image are fused, resulting in low detection efficiency.

[0048] In view of this, the embodiments of the present disclosure provide a 3D target detection method and device based on multi-view fusion. When performing 3D target detection through the scheme of the present disclosure, the autonomous driving carrier can extract features from at least one image of the multi-camera perspective captured by the multi-camera system, and obtain feature data containing target object features in the multi-camera perspective space. And based on the internal parameters of the multi-camera system and the vehicle parameters, the feature data in the multi-camera perspective space is mapped to the same bird's-eye view space to obtain the feature data corresponding to at least one image in the bird's-eye view space. Then, the feature data corresponding to at least one image in the bird's-eye view space are feature fused to obtain a bird's-eye view fusion feature; target prediction is performed on the target object in the bird's-eye view fusion feature to obtain the three-dimensional spatial information of the target object in the surrounding environment of the carrier.

[0049] When the scheme of the embodiment of the present disclosure performs 3D target detection based on multi-perspective fusion, the feature data of at least one image under the perspectives of multiple cameras are simultaneously mapped to the same bird's-eye view space, which enables more reasonable and better fusion. At the same time, the three-dimensional spatial information of each target object in the surroundings of the vehicle environment is directly detected in the bird's-eye view space through the fused bird's-eye view fusion features. Therefore, when the scheme of the embodiment of the present disclosure performs 3D target detection based on multi-perspective fusion, multi-perspective feature fusion is performed first and then 3D target detection is performed, completing the 3D target detection of scene objects under the bird's-eye view end-to-end, avoiding the post-processing stage of conventional multi-perspective 3D target detection, and improving detection efficiency.

[0050] Exemplary Systems

[0051] The embodiments of the present disclosure can be applied to application scenarios that require 3D target detection, such as autonomous driving application scenarios.

[0052] For example, in an application scenario of autonomous driving, a multi-camera system is configured on an autonomous driving carrier (hereinafter referred to as the "carrier"), and images from different perspectives are captured by the multi-camera system. Then, through the solution of the embodiment of the present disclosure, 3D target detection based on multi-perspective fusion is performed to obtain three-dimensional spatial information of target objects in the surrounding environment of the carrier.

[0053] Figure 1 It is a scene graph to which the present disclosure is applicable.

[0054] like Figure 1 As shown, the embodiment of the present disclosure is applied in the application scenario of assisted driving or automatic driving, and the vehicle-mounted automatic driving system 200 and the multi-camera system 300 are configured on the vehicle-mounted automatic driving system 100 of the assisted driving or automatic driving, and the vehicle-mounted automatic driving system 200 and the multi-camera system 300 are electrically connected. The multi-camera system 300 is used to collect images of the surrounding environment of the vehicle, and the vehicle-mounted automatic driving system 200 is used to obtain the images collected by the multi-camera system 300, and perform 3D target detection based on multi-view fusion to obtain the three-dimensional spatial information of the target object in the surrounding environment of the vehicle.

[0055] Figure 2 It is a system block diagram of the vehicle-mounted automatic driving system provided by an embodiment of the present disclosure.

[0056] like Figure 2 As shown, the vehicle-mounted automatic driving system 200 includes an image receiving module 201, a feature extraction module 202, an image feature mapping module 203, an image fusion module 204 and a 3D detection module 205. The image receiving module 201 is used to obtain at least one image collected by the multi-camera system 300; the feature extraction module 202 is used to extract features from at least one image obtained by the image receiving module 201 to obtain feature data; the image feature mapping module 203 is used to map the feature data of at least one image from the multi-camera viewing space to the same bird's-eye view viewing space; the image fusion module 204 is used to perform feature fusion on the feature data corresponding to each of at least one image in the bird's-eye view viewing space to obtain a bird's-eye view fusion feature; the 3D detection module 205 is used to perform target prediction on the target object in the bird's-eye view fusion feature obtained by the image fusion module 204 to obtain the three-dimensional space information of the target object in the surrounding environment of the carrier.

[0057] The multi-camera system 300 includes multiple cameras with different viewing angles, each camera is used to collect environmental images of a viewing angle, and the multiple cameras cover a 360-degree environmental range around the carrier. Each camera defines its own camera viewing angle coordinate system, and forms its own camera viewing angle space through its own camera viewing angle coordinate system. The environmental image collected by each camera is an image in the corresponding camera viewing angle space.

[0058] Exemplary Methods

[0059] Figure 3 It is a flowchart of a 3D object detection method based on multi-view fusion provided by an exemplary embodiment of the present disclosure.

[0060] This embodiment can be applied to a vehicle-mounted automatic driving system 200, such as Figure 3 As shown, the following steps are included:

[0061] Step S301, obtaining at least one image captured from the perspective of multiple cameras.

[0062] The at least one image may be captured by at least one camera of the multi-camera system. Exemplarily, the at least one image may be an image captured in real time by the multi-camera system, or may be an image captured in advance by the multi-camera system.

[0063] Figure 4 It is a schematic block diagram of a multi-camera system capturing images according to an exemplary embodiment of the present disclosure.

[0064] like Figure 4 As shown, in one embodiment, the multi-camera system can collect multiple images of different perspectives in real time, such as images 1, 2...N, and send the collected images to the vehicle-mounted automatic driving system in real time. In this way, the images obtained by the vehicle-mounted automatic driving system can represent the real situation of the environment around the carrier at the current moment.

[0065] Figure 5 FIG. 4 is a schematic diagram of images from multiple camera perspectives provided by an exemplary embodiment of the present disclosure.

[0066] like Figure 5 As shown in (1)-(6), in one embodiment, the multi-camera system may include 6 cameras. The 6 cameras are respectively arranged at the front end, the left front end, the right front end, the rear end, the left rear end and the right rear end of the carrier. In this way, at any time, the multi-camera system can capture images of 6 different perspectives, such as the front view image (I front )、Left front view image(I frontleft )、Right front view image(I frontright )、Rear view image(I rear )、Left rear view image(I rearleft ) and right rear view image (Irearright ).

[0067] Each image includes but is not limited to target objects of various categories such as roads, traffic lights, road signs, vehicles (small cars, buses, trucks, etc.), pedestrians, cyclists, etc. As the categories and positions of the target objects in the surrounding environment of the carrier are different, the categories and positions of the target objects contained in each image are also different.

[0068] Step S302: extract features from at least one image to obtain feature data containing target object features corresponding to each of the at least one image in a multi-camera viewing space.

[0069] In one embodiment, the vehicle-mounted autonomous driving system can extract feature data in the corresponding camera viewing space from each image. The feature data can include target object features for describing the target object in the image, and the target object features include but are not limited to image texture information, edge contour information, semantic information, etc.

[0070] The image texture information is used to characterize the image texture of the target object, the edge contour information is used to characterize the edge contour of the target object, and the semantic information is used to characterize the category of the target object. The categories of the target object include but are not limited to: roads, traffic lights, road signs, vehicles (small cars, buses, trucks, etc.), pedestrians, cyclists, etc.

[0071] Figure 6 It is a schematic block diagram of feature extraction provided by an exemplary embodiment of the present disclosure.

[0072] like Figure 6 As shown, the vehicle-mounted automatic driving system can use a neural network to extract features from at least one image (image 1-N) to obtain feature data 1-N corresponding to each image in the multi-camera viewing space.

[0073] For example, the vehicle's automatic driving system can detect the front view image (I front ) to extract features, and the front view image (I front ) Feature data f in the front camera viewing space front ; For the left front view image (I frontleft ) to extract features, we can get the left front view image (I frontleft ) Feature data f in the left front camera viewing space frontleft ; For the right front view image (I frontright ) to extract features, we can get the right front view image (I frontright ) Feature data f in the right front camera viewing space frontright ; For the rear view image (I rear ) to extract features, and the rear view image (Irear ) Feature data f in the backend camera viewing space rear ; For the left rear view image (I rearleft ) to extract features, we can get the left rear view image (I rearleft ) Feature data f in the left rear camera viewing space rearleft ; For the right rear view image (I rearright ) to extract features, we can get the right rear view image (I rearright ) Feature data f in the right rear camera viewing space rearright .

[0074] Step S303, based on the internal parameters of the multi-camera system and the vehicle parameters, mapping the feature data corresponding to at least one image in the multi-camera viewing space to the same bird's-eye view viewing space, to obtain the feature data corresponding to at least one image in the bird's-eye view viewing space.

[0075] Among them, the internal parameters of the multi-camera system include the intrinsic parameters and extrinsic parameters of each camera. The intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.; the extrinsic parameters are parameters in the world coordinate system, such as the camera's position, rotation direction, etc. The vehicle parameters refer to the conversion matrix from the vehicle coordinate system (VCS) to the bird's-eye view coordinate system (BEV). The vehicle coordinate system is the coordinate system where the vehicle is located.

[0076] For example, the vehicle's autonomous driving system converts the forward-looking image (I front ) Feature data f in the front camera viewing space front Mapped to the same bird's-eye view space, we get the front view image (I front ) Feature data F in bird's-eye view space front ; The left front view image (I frontleft ) Feature data f in the left front camera viewing space frontleft Mapped to the same bird's-eye view space, we get the left front view image (I frontleft ) Feature data F in bird's-eye view space frontleft ; The right front view image (I frontright ) Feature data f in the right front camera viewing space frontright Mapped to the same bird's-eye view space, we get the right front view image (I frontright ) Feature data F in bird's-eye view space frontright ; The rear view image (I rear ) Feature data f in the backend camera viewing space rear Mapped to the same bird's-eye view space, the rear view image (I rear) Feature data F in bird's-eye view space rear ; The left rear view image (I rearleft ) Feature data f in the left rear camera viewing space rearleft Mapped to the same bird's-eye view space, the left rear view image (I rearleft ) Feature data F in bird's-eye view space rearleft ; The right rear view image (I rearright ) Feature data f in the right rear camera viewing space rearright Mapped to the same bird's-eye view space, the right rear view image (I rearright ) Feature data F in bird's-eye view space rearright .

[0077] Step S304: performing feature fusion on feature data corresponding to at least one image in the bird's-eye view space to obtain a bird's-eye view fusion feature.

[0078] Among them, the bird's-eye view fusion feature is used to characterize the feature data of the target objects around the carrier in the bird's-eye view space. The feature data of the target objects in the bird's-eye view space may include but is not limited to the shape, size, category, orientation angle, relative position and other attributes of the target objects.

[0079] In one embodiment, the vehicle-mounted automatic driving system may perform additive feature fusion on the feature data corresponding to at least one image in the bird's-eye view space to obtain a bird's-eye view fusion feature. Specifically, it can be expressed as the following formula:

[0080] F′=Add(∑F (front、frontleft、frontright、rear、rearleft、rearright) )

[0081] Among them, F′ represents the bird's-eye view fusion feature, and Add represents the addition feature fusion calculation of the feature data corresponding to at least one image in the bird's-eye view space.

[0082] It should be noted that the implementation of step S304 is not limited thereto. For example, feature fusion may be performed on feature data corresponding to images of different camera perspectives in the bird's-eye view space by multiplication, superposition, etc.

[0083] Figure 7 It is a schematic diagram of generating a bird's-eye view image from images acquired by a multi-camera system according to an exemplary embodiment of the present disclosure.

[0084] like Figure 7As shown, illustratively, the size of the bird's-eye view image can be the same as the size of at least one image captured by the multi-camera system. The bird's-eye view image can reflect the three-dimensional spatial information of the target object, and the three-dimensional spatial information includes at least one attribute information of the target object, which includes but is not limited to 3D position information (i.e., coordinate information of the X-axis, Y-axis, and Z-axis), size information (i.e., length, width, and height information), and orientation angle information.

[0085] Among them, the coordinate information of the X-axis, Y-axis, and Z-axis refers to the coordinate position (x, y, z) of the target object in the bird's-eye view space. The origin of the coordinate system of the bird's-eye view space is located at any position such as the chassis of the carrier or the center of the carrier. The X-axis direction is from front to back, the Y-axis direction is from left to right, and the Z-axis direction is vertical up and down. The heading angle refers to the angle formed by the front direction or the direction of travel of the target object in the bird's-eye view space. For example, when the target object is a moving pedestrian, the heading angle refers to the angle formed by the direction of travel of the pedestrian in the bird's-eye view space. When the target object is a stationary vehicle, the heading angle refers to the angle formed by the front direction of the vehicle in the bird's-eye view space.

[0086] It should be noted that, since at least one image captured by the multi-camera system may include target objects of different categories, the bird's-eye view image may include bird's-eye view fusion features of target objects of different categories.

[0087] Step S305 , performing target prediction on the target object in the bird's-eye view fusion feature to obtain three-dimensional spatial information of the target object.

[0088] The three-dimensional space information may include at least one of the following attributes: position, size, and orientation angle of the target object in the bird's-eye view coordinate system. Position refers to the coordinate position (x, y, z) of the target object relative to the carrier in the bird's-eye view space, size refers to the length, width, and height (Height, Width, Length) of the target object in the bird's-eye view space, and orientation angle refers to the orientation angle (rotation yaw) of the target object in the bird's-eye view space.

[0089] Figure 8 is a schematic block diagram of target detection provided by an exemplary embodiment of the present disclosure.

[0090] like Figure 8 As shown, in one embodiment, the on-board autonomous driving system can use one or more prediction networks to perform 3D target prediction on the target objects in the bird's-eye view fusion features to obtain the three-dimensional spatial information of each target object in the surrounding environment of the carrier.

[0091] If the on-board autonomous driving system uses multiple prediction networks to perform 3D target prediction, each prediction network can output one or more attributes of the target object, and different prediction networks output different attributes.

[0092] When the solution of the embodiment of the present disclosure performs 3D target detection based on multi-perspective fusion, multi-perspective feature fusion can be performed first and then 3D target detection can be performed, so as to complete end-to-end 3D target detection of scene objects from a bird's-eye view, avoid the post-processing stage of conventional multi-perspective 3D target detection, and improve detection efficiency.

[0093] Fig. 9 It is a flowchart of determining feature data in a bird's-eye view space provided by an exemplary embodiment of the present disclosure.

[0094] like Fig. 9 As shown in the above Figure 3 Based on the illustrated embodiment, step S303 may include the following steps:

[0095] Step S3031: Determine a conversion matrix from a camera coordinate system of the multi-camera system to a bird's-eye view coordinate system based on internal parameters of the multi-camera system and vehicle parameters.

[0096] Among them, the internal parameters of the multi-camera system include the intrinsic parameters and extrinsic parameters of each camera. The extrinsic parameters refer to the transformation matrix from the camera coordinate system of the multi-camera to the vehicle coordinate system. The vehicle parameters refer to the transformation matrix from the vehicle coordinate system (VCS) to the bird's-eye view coordinate system (BEV). The vehicle coordinate system is the coordinate system where the carrier is located.

[0097] In a specific implementation, step S3031 includes:

[0098] Obtain the camera intrinsic parameters and camera extrinsic parameters of the multi-camera system respectively, and obtain the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system;

[0099] Based on the camera extrinsic parameters, camera intrinsic parameters and the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system of the multi-camera, a transformation matrix from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system is determined.

[0100] In one embodiment, the vehicle-mounted autonomous driving system may determine the transformation matrix H from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system by the following formula:

[0101] HT vcs→bev @T camera→vcs @K -1 ;

[0102] Where @ represents matrix multiplication; Tcamera→vcs Represents the transformation matrix from the camera coordinate system to the vehicle coordinate system, T camera→vcs Characterize the camera external parameters; T vcs→bev Represents the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system; K represents the camera intrinsic parameters.

[0103] It should be noted that the camera extrinsic parameters, that is, the conversion matrix from the camera coordinate system to the vehicle coordinate system, can be obtained through the calibration of the multi-camera system. Once the calibration is completed, it usually does not change. The conversion matrix from the vehicle coordinate system to the bird's-eye view coordinate system can be calculated based on the artificially set bird's-eye view range (for example, a range of 100 meters in front, back, left, and right) and the resolution of the bird's-eye view image (for example, 512×512).

[0104] In this way, each camera in the multi-camera system can determine the corresponding transformation matrix. For example, the on-board autonomous driving system determines the transformation matrix H from the camera coordinate system of the front-end camera to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system, and the camera intrinsic parameters of the front-end camera. front→bev Based on the transformation matrix from the camera coordinate system of the left front camera to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system, and the camera intrinsic parameters of the left front camera, determine the transformation matrix H from the camera coordinate system of the left front camera to the bird's-eye view coordinate system frontleft→bev Based on the transformation matrix from the camera coordinate system of the right front camera to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system, and the camera intrinsic parameters of the right front camera, determine the transformation matrix H from the camera coordinate system of the right front camera to the bird's-eye view coordinate system frontright→bev Based on the transformation matrix from the camera coordinate system of the back-end camera to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system, and the camera intrinsic parameters of the back-end camera, determine the transformation matrix H from the camera coordinate system of the back-end camera to the bird's-eye view coordinate system rear→bev Based on the transformation matrix from the camera coordinate system of the left rear camera to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system, and the camera intrinsic parameters of the left rear camera, determine the transformation matrix H from the camera coordinate system of the left rear camera to the bird's-eye view coordinate system rearleft→bev Based on the transformation matrix from the camera coordinate system of the right rear camera to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system, and the camera intrinsic parameters of the right rear camera, determine the transformation matrix H from the camera coordinate system of the right rear camera to the bird's-eye view coordinate system rearright→bev .

[0105] In this implementation, since each camera has a transformation matrix from its own camera perspective coordinate system to the bird's-eye view coordinate system, the prediction network used in the embodiment of the present application when performing 3D target detection is suitable for a multi-camera system, and there is no need to train the prediction network from scratch, thereby improving detection efficiency.

[0106] Step S3032, based on the transformation matrix from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system, the feature data corresponding to at least one image in the multi-camera perspective space is transformed from the multi-camera perspective space to the bird's-eye view space to obtain the feature data corresponding to at least one image in the bird's-eye view space.

[0107] In one embodiment, the vehicle-mounted automatic driving system may obtain the feature data corresponding to at least one image in the bird's-eye view space by matrix multiplication of the transformation matrix of each camera and the feature data in the camera's view space. Specifically, it can be expressed as the following formula:

[0108] F=H@f.

[0109] Where F represents the feature data F corresponding to at least one image in the bird's-eye view space front 、F frontleft 、F frontright 、F rear 、F rearleft and F rearright ; H represents the transformation matrix H corresponding to each camera in the multi-camera system front→bev , H frontleft→bev , H frontright→bev , H rear→bev , H rearleft→bev and H rearright→bev ; f represents the feature data of at least one image in the multi-camera viewing space front 、f frontleft 、f frontright 、f rear 、f rearleft and f rearright .

[0110] It can be seen that the embodiment of the present disclosure calculates the respective transformation matrices (homography) for different cameras in the multi-camera system, and then maps the respective feature data to the bird's-eye view space based on the respective transformation matrices of each camera, thereby obtaining the feature data corresponding to each image in the bird's-eye view space. This embodiment is not only applicable to multi-camera systems of different models, but also can perform more reasonable feature fusion.

[0111] It should be noted that step S302 and step S3031 may be executed synchronously or asynchronously, depending on the actual application situation.

[0112] Fig.10 It is a schematic block diagram of executing step S303 and step S304 provided by an exemplary embodiment of the present disclosure.

[0113] like Fig.10 As shown, after all steps S302 and S3031 are executed, the feature space conversion described in step S3032 is performed based on the conversion matrix from the camera coordinate system of each camera to the bird's-eye view coordinate system obtained in step S3031 and the feature data of the corresponding camera view space obtained in step S302 to obtain the feature data of the bird's-eye view space. Finally, step S304 is executed to perform feature fusion on the feature data of the multi-camera view in the bird's-eye view space to obtain the bird's-eye view fusion feature.

[0114] Fig.11 is a flowchart of target detection provided by an exemplary embodiment of the present disclosure.

[0115] like Fig.11 As shown in the above Figure 3 Based on the illustrated embodiment, step S305 may include the following steps:

[0116] Step S3051: Use the prediction network to obtain a heat map corresponding to a first preset coordinate value of the target object in the bird's-eye view coordinate system from the bird's-eye view fusion features, and obtain other attribute maps for determining the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system.

[0117] The prediction network may be a neural network for predicting the target object. Since the target object needs to predict the three-dimensional spatial information of different attributes, the prediction network may also be multiple. Different prediction networks are used to predict the three-dimensional spatial information of different attributes.

[0118] For example, when the attribute to be predicted is the first preset coordinate value of the target object, the prediction network corresponding to the first preset coordinate value can be used to process the bird's-eye view fusion feature in the bird's-eye view image to obtain a heat map, so as to determine the first preset coordinate value of the target object in the bird's-eye view coordinate system using the heat map. The size of the heat map can be the same as the size of the bird's-eye view image.

[0119] For another example, when the attributes to be predicted are the second preset coordinate value, size and orientation angle of the target object, the prediction network corresponding to the second preset coordinate value, size and orientation angle can be used to process the bird's-eye view fusion features in the bird's-eye view image to obtain other attribute maps, so as to use the other attribute maps to determine the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system.

[0120] Among them, the first preset coordinate value is the (x, y) position in the bird's-eye view coordinate system, the second preset coordinate value is the z position in the bird's-eye view coordinate system, the dimensions are length, width and height, and the orientation angle is the orientation angle.

[0121] Step S3052: determine the first preset coordinate value of the target object in the bird's-eye view coordinate system according to the peak information in the heat map, and determine the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system from other attribute maps according to the first preset coordinate value of the target object in the bird's-eye view coordinate system.

[0122] The peak information refers to the center value of the Gaussian kernel, that is, the center point of the target object.

[0123] After predicting the first preset coordinate value of the target object in the bird's-eye view space, since other attribute maps can use the attribute output results of the heat map to output their respective attribute information, the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system can be predicted from other attribute maps based on the first preset coordinate value of the target object in the bird's-eye view coordinate system.

[0124] Step S3053: Determine the three-dimensional space information of the target object according to the first preset coordinate value, the second preset coordinate value, the size and the orientation angle of the target object in the bird's-eye view coordinate system.

[0125] In one embodiment, the vehicle-mounted automatic driving system can determine the first preset coordinate value and the second preset coordinate value as the (x, y, z) position of the target object in the bird's-eye view space, determine the size as the length, width, and height of the target object in the bird's-eye view space, and determine the orientation angle as the orientation angle of the target object in the bird's-eye view space. Finally, the three-dimensional spatial information of the target object in the surrounding environment of the carrier is determined based on the (x, y, z) position, length, width, height, and orientation angle.

[0126] Fig.12 FIG. 1 is a schematic diagram of the output result of the prediction network provided by an exemplary embodiment of the present disclosure. Fig.12 In the figure, the center A of the smallest circle is the carrier position, and the box position B around the center is the target object around the carrier.

[0127] In addition, the on-board autonomous driving system can also display the three-dimensional spatial projection of the target object onto the image from the multi-camera perspective captured by the multi-camera system, so that the user can intuitively understand the three-dimensional spatial information of the target object from the on-board display screen.

[0128] It can be seen that the disclosed embodiment can process the bird's-eye view image according to the prediction network to obtain a heat map and other attribute maps. Inputting the bird's-eye view fusion features obtained by feature fusion into the heat map and other attribute maps can directly predict the three-dimensional spatial information of the target object, thereby improving the efficiency of 3D target detection.

[0129] Fig.13 is another flowchart of target detection provided by an exemplary embodiment of the present disclosure.

[0130] like Fig.13 As shown in the above Fig.11 Based on the illustrated embodiment, step S305 may further include the following steps:

[0131] Step S3054: During the training phase of the prediction network, a first loss function is constructed between the heat map output by the prediction network and the true value heat map, and a second loss function is constructed between other attribute maps predicted by the prediction network and other true value attribute maps.

[0132] In one embodiment, the vehicle-mounted autonomous driving system can construct a Gaussian kernel for each target object based on the position of each target object in the bird's-eye view fusion feature.

[0133] Fig.14 Schematic diagram of a Gaussian kernel provided by an exemplary embodiment of the present disclosure. Fig.14 As shown in the figure, when constructing a Gaussian kernel, a Gaussian kernel of size N×N can be generated with the position (i, j) of the target object as the center. The value of the center of the Gaussian kernel is 1, and the values ​​around it decay downward to 0. The color changes from white to black, indicating that the value decays from 1 to 0.

[0134] Fig.15 is a schematic diagram of a thermal map provided by an exemplary embodiment of the present disclosure. Fig.15 As shown in , the Gaussian kernel of each target object can be placed on the heat map to obtain the true value heat map. Fig.15 In the figure, each white area represents a Gaussian kernel, that is, a target object, such as target objects 1-6.

[0135] It should be noted that the generation method of other true value attribute maps can refer to the generation method of the true value heat map, which will not be repeated here.

[0136] After determining the true value heat map, a first loss function can be constructed based on the true value heat map and the heat map of the prediction network output. The first loss function can measure the gap distribution between the output prediction value of the prediction network and the true value, and is used to supervise the training process of the prediction network.

[0137] In one embodiment, the first loss function L clsSpecifically, it can be constructed by the following formula:

[0138]

[0139] Among them, y′ i,j represents the first preset coordinate value of the position (i, j) in the true value heat map, 1 represents the peak value in the heat map, y i,j It represents the first preset coordinate value of the position (i, j) in the heat map predicted by the prediction network. α and β are adjustable hyperparameters. The ranges of α and β are both between 0 and 1. N represents the number and number of target objects in the bird's-eye view fusion feature. h and w represent the size of the bird's-eye view fusion feature.

[0140] In one embodiment, the second loss function L reg Specifically, it can be constructed by the following formula:

[0141]

[0142] Among them, B′ is the true value of the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system, B is the predicted value of the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system predicted by the prediction network, and N represents the number of target objects in the bird's-eye view fusion feature.

[0143] Step S3055: determine the total loss function of the prediction network in the training phase according to the first loss function and the second loss function to supervise the training process of the prediction network.

[0144] In one embodiment, the total loss function of the prediction network during the training phase may be determined by the following steps:

[0145] Obtaining a weight value of a first loss function and a weight value of a second loss function;

[0146] Based on the first loss function, the weight value of the first loss function, the second loss function, and the weight value of the second loss function, a total loss function of the prediction network in the training phase is determined.

[0147] Here, when using the prediction network to predict the three-dimensional spatial information of the target object, different attributes have different importance in the training process, which makes the corresponding loss function have different importance. Therefore, according to the importance of each attribute in the training process, different weight values ​​are configured for the loss functions corresponding to different attributes.

[0148] Among them, the total loss function L of the prediction network in the training stage 3d It can be determined by the following formula:

[0149] L 3d =λ1L cls +λ2Lreg ;

[0150] Among them, L cls is the first loss function, L reg is the second loss function, λ1 is the weight value of the first loss function, and λ2 is the weight value of the second loss function. Both λ1 and λ2 are between 0 and 1, λ1>λ2, and λ1+λ2=1.

[0151] It can be seen that when the prediction network is trained, the embodiment of the present disclosure constructs a total loss function to supervise the total training process to ensure that the output of various attributes of the prediction network is more accurate, thereby ensuring higher efficiency of 3D target detection.

[0152] Fig.16 is another flowchart of target detection provided by an exemplary embodiment of the present disclosure.

[0153] like Fig.16 As shown in the above Figure 3 Based on the illustrated embodiment, step S305 may further include the following steps:

[0154] Step S3056: Use a neural network to extract the bird's-eye view fusion features to obtain bird's-eye view fusion feature data containing target object features.

[0155] In one embodiment, the vehicle-mounted automatic driving system may use a neural network to perform convolution and other calculations on the bird's-eye view fusion features to achieve feature extraction and obtain bird's-eye view fusion feature data. The bird's-eye view fusion feature data includes target object features of different dimensions for characterizing the target object, that is, scene information of the target object from different dimensions in the bird's-eye view space.

[0156] The neural network may be a pre-trained neural network for feature extraction. Optionally, the neural network for feature extraction is not limited to a specific network result, such as resnet, densenet, mobilenet, etc.

[0157] Step S3057: Use the prediction network to perform target prediction on the target object in the bird's-eye view fusion feature data containing the target object features to obtain the three-dimensional spatial information of the target object.

[0158] It can be seen that, before the prediction network is used to train the bird's-eye view fusion feature, the embodiment of the present disclosure extracts the bird's-eye view fusion feature to obtain the bird's-eye view fusion feature data. The prediction network is then used to predict the bird's-eye view fusion feature data containing the target object feature, so that the prediction result is more accurate, that is, the three-dimensional spatial information of the determined target object is more accurate.

[0159] In the above Figure 3Based on the illustrated embodiment, step S302 may include the following steps:

[0160] A deep neural network is used to perform convolution calculations on the images corresponding to each perspective, and feature data of multiple different resolutions containing target object features corresponding to the images corresponding to each perspective in the multi-camera perspective space are obtained.

[0161] Here, the deep neural network can be a pre-trained neural network for feature extraction. Optionally, the neural network used for feature extraction is not limited to a specific network result, such as resnet, densenet, mobilenet, etc. By using the deep neural network to perform convolution, pooling and other calculations on the image of the target perspective, feature data of multiple different resolutions (scales) corresponding to the image of the target perspective can be obtained.

[0162] For example, the size of image A of a certain perspective is H×W×3, where H is the height of image A, W is the width of image A, and 3 indicates that there are 3 channels. For example, if it is an RGB image, 3 indicates RGB (R red, G green, B blue) 3 channels; if it is a YUV image, 3 indicates YUV (Y brightness signal, U blue component signal, V red component signal) 3 channels. Image A is input into a deep neural network, and after convolution and other calculations through the deep neural network, a feature matrix of dimensions H1×W1×N is output, where H1 and W1 are the height and width of the feature (usually smaller than H and W, N is the number of channels, and N is greater than 3). Through the fitting training of the neural network on the input data, feature data of multiple different resolutions containing the features of the target object of the input image can be obtained, such as low-level image textures corresponding to different resolutions, edge contour information, and high-level semantic information. After obtaining the feature data of the image of each perspective, the subsequent spatial transformation, multi-perspective feature fusion and target prediction steps can be performed to obtain the three-dimensional spatial information of the target object.

[0163] It can be seen that the embodiment of the present disclosure performs convolution, pooling and other calculations on the images corresponding to each perspective through a deep neural network to obtain feature data of multiple different resolutions for each perspective image. The feature data of different resolutions can better reflect the image features collected by the corresponding perspective camera and improve the efficiency of subsequent 3D object detection.

[0164] Exemplary Devices

[0165] Fig.17: is a structural diagram of a 3D object detection device based on multi-view fusion provided by an exemplary embodiment of the present disclosure. The 3D object detection device based on multi-view fusion can be set in an electronic device such as a terminal device, a server, or a carrier of assisted driving or automatic driving. For example, it can be set in an on-board automatic driving system to execute the 3D object detection method based on multi-view fusion of any of the above embodiments of the present disclosure. Fig.17 As shown, the 3D object detection device based on multi-view fusion of this embodiment includes: an image receiving module 201, a feature extraction module 202, an image feature mapping module 203, an image fusion module 204 and a 3D detection module 205.

[0166] The image receiving module 201 is used to obtain at least one image captured from the perspective of multiple cameras.

[0167] The feature extraction module 202 is used to extract features from the at least one image acquired by the image receiving module to obtain feature data containing target object features corresponding to each of the at least one image in the multi-camera viewing space.

[0168] The image feature mapping module 203 is used to map the feature data corresponding to each of the at least one image obtained by the feature extraction module in the multi-camera viewing space to the same bird's-eye view space based on the internal parameters of the multi-camera system and the vehicle parameters, so as to obtain the feature data corresponding to each of the at least one image in the bird's-eye view space.

[0169] The image fusion module 204 is used to perform feature fusion on the feature data corresponding to each of the at least one image obtained by the image mapping module in the bird's-eye view space to obtain a bird's-eye view fusion feature.

[0170] The 3D detection module 205 is used to perform target prediction on the target object in the bird's-eye view fusion feature obtained by the image fusion module to obtain three-dimensional spatial information of the target object.

[0171] It can be seen that when the device of the embodiment of the present disclosure performs 3D target detection based on multi-perspective fusion, the feature data of at least one image under the perspectives of multiple cameras are simultaneously mapped to the same bird's-eye view space through middle fusion, which can perform more reasonable and better fusion. At the same time, the three-dimensional spatial information of each target object in the surroundings of the vehicle environment is directly detected in the bird's-eye view space through the fused bird's-eye view fusion features. Therefore, when the device of the embodiment of the present disclosure performs 3D target detection based on multi-perspective fusion, the 3D detection of scene objects under the bird's-eye view is completed end-to-end, avoiding the post-processing stage of conventional multi-perspective 3D target detection, thereby improving detection efficiency.

[0172] Fig.18 is another structural diagram of a 3D object detection device based on multi-view fusion provided by an exemplary embodiment of the present disclosure.

[0173] Further, such as Fig.18 As shown in the structural diagram, the image feature mapping module 203 includes:

[0174] A conversion matrix determining unit 2031 is used to determine a conversion matrix from a camera coordinate system of multiple cameras of the multi-camera system to a bird's-eye view coordinate system based on internal parameters of the multi-camera system and vehicle parameters;

[0175] The spatial conversion unit 2032 is used to convert the feature data corresponding to each of the at least one image in the multi-camera perspective space from the multi-camera perspective space to the bird's-eye view perspective space based on the conversion matrix from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system determined by the conversion matrix determination unit 2031, so as to obtain the feature data corresponding to each of the at least one image in the bird's-eye view perspective space.

[0176] In a feasible implementation manner, the conversion matrix determining unit 2031 includes:

[0177] A conversion matrix acquisition subunit, used to respectively acquire the camera intrinsic parameters and camera extrinsic parameters of the multi-camera system, and to acquire the conversion matrix from the vehicle coordinate system to the bird's-eye view coordinate system;

[0178] The transformation matrix determination subunit is used to determine the transformation matrix from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system based on the camera extrinsic parameters, camera intrinsic parameters and the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system of the multi-camera acquired by the transformation matrix acquisition subunit.

[0179] Furthermore, the 3D detection module 205 includes:

[0180] The detection network acquisition unit 2051 is used to acquire, from the bird's-eye view fusion feature, a heat map corresponding to a first preset coordinate value of the target object in the bird's-eye view coordinate system using a prediction network, and to acquire other attribute maps for determining a second preset coordinate value, size, and orientation angle of the target object in the bird's-eye view coordinate system;

[0181] The information detection unit 2052 is used to determine the first preset coordinate value of the target object in the bird's-eye view coordinate system according to the peak information in the heat map obtained by the detection network acquisition unit 2051, and determine the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system from the other attribute map according to the first preset coordinate value of the target object in the bird's-eye view coordinate system;

[0182] The information determination unit 2053 is used to determine the three-dimensional space information of the target object according to the first preset coordinate value, the second preset coordinate value, the size and the orientation angle of the target object detected by the information detection unit 2052 in the bird's-eye view coordinate system.

[0183] In a feasible implementation manner, the 3D detection module 205 further includes:

[0184] The loss function construction unit 2054 is used to construct a first loss function between a heat map predicted by the prediction network and a true value heat map, and to construct a second loss function between other attribute maps predicted by the prediction network and other true value attribute maps during the training phase of the prediction network.

[0185] The total loss function determining unit 2055 is used to determine the total loss function of the prediction network in the training stage according to the first loss function and the second loss function constructed by the loss function constructing unit 2054, so as to supervise the training process of the prediction network.

[0186] In a feasible implementation manner, the total loss function determining unit 2055 includes:

[0187] A weight value acquisition subunit, used to acquire a weight value of the first loss function and a weight value of the second loss function;

[0188] The total loss function determination subunit is used to determine the total loss function of the prediction network in the training stage based on the first loss function and the second loss function constructed by the loss function construction unit 2054, and the weight value of the first loss function and the weight value of the second loss function obtained by the weight value acquisition subunit.

[0189] In a feasible implementation manner, the 3D detection module 205 further includes:

[0190] A fusion feature extraction unit 2056 is used to extract the bird's-eye view fusion feature by using a neural network to obtain bird's-eye view fusion feature data containing target object features;

[0191] The target prediction unit 2057 is used to use a prediction network to perform target prediction on the target object in the bird's-eye view fusion feature data containing target object features obtained by the feature extraction unit 2056 to obtain three-dimensional spatial information of the target object.

[0192] Furthermore, the feature extraction module 202 includes:

[0193] The feature extraction unit 2021 is used to perform convolution calculations on the images corresponding to each perspective using a deep neural network to obtain feature data of multiple different resolutions containing target object features corresponding to the images corresponding to each perspective in the multi-camera perspective space.

[0194] Exemplary Electronic Devices

[0195] Below, reference Fig.19 An electronic device according to an embodiment of the present disclosure is described.

[0196] Fig.19 is a structural block diagram of an electronic device provided by an exemplary embodiment of the present disclosure.

[0197] like Fig.19 As shown, the electronic device 11 includes one or more processors 111 and a memory 112 .

[0198] The processor 111 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

[0199] The memory 112 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 111 may run the program instructions to implement the 3D object detection method based on multi-view fusion and / or other desired functions of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0200] In one example, the electronic device 11 may further include: an input device 113 and an output device 114 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0201] In addition, the input device 113 may also include, for example, a keyboard, a mouse, etc.

[0202] The output device 114 can output various information to the outside, including determined distance information, direction information, etc. The output device 114 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0203] Of course, to simplify, Fig.19 Only some of the components related to the present disclosure in the electronic device 11 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 11 may also include any other appropriate components.

[0204] Exemplary computer program products and computer-readable storage media

[0205] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the 3D target detection method based on multi-perspective fusion according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0206] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0207] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the 3D target detection method based on multi-perspective fusion according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0208] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0209] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0210] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0211] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0212] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0213] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A 3D object detection method based on multi-view fusion, comprising: Acquire at least one image collected from multiple camera perspectives; Performing feature extraction on the at least one image to obtain feature data containing target object features corresponding to each of the at least one image in a multi-camera viewing space; Based on internal parameters of the multi-camera system and vehicle parameters, mapping feature data corresponding to each of the at least one image in the multi-camera viewing space to the same bird's-eye view viewing space to obtain feature data corresponding to each of the at least one image in the bird's-eye view viewing space; Performing feature fusion on the feature data corresponding to each of the at least one image in the bird's-eye view space to obtain a bird's-eye view fusion feature; Performing target prediction on the target object in the bird's-eye view fusion feature to obtain three-dimensional spatial information of the target object; Among them, target prediction is performed on the target object in the bird's-eye view fusion feature to obtain the three-dimensional spatial information of the target object, including: using a prediction network to obtain a heat map corresponding to a first preset coordinate value of the target object in a bird's-eye view coordinate system from the bird's-eye view fusion feature, and obtaining other attribute maps for determining the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate system; determining the first preset coordinate value of the target object in the bird's-eye view coordinate system according to the peak information in the heat map, and determining the second preset coordinate value, size and orientation angle of the target object in the bird's-eye view coordinate from the other attribute maps according to the first preset coordinate value; determining the three-dimensional spatial information of the target object according to the first preset coordinate value, the second preset coordinate value, the size and the orientation angle.

2. The method according to claim 1, wherein: The mapping of the feature data corresponding to each of the at least one image in the multi-camera viewing space to the same bird's-eye view viewing space based on the internal parameters of the multi-camera system and the vehicle parameters to obtain the feature data corresponding to each of the at least one image in the bird's-eye view viewing space includes: Determine a conversion matrix from a camera coordinate system of multiple cameras of the multi-camera system to a bird's-eye view coordinate system based on internal parameters of the multi-camera system and vehicle parameters; Based on the transformation matrix from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system, the feature data corresponding to each of the at least one image in the multi-camera perspective space is transformed from the multi-camera perspective space to the bird's-eye view space to obtain the feature data corresponding to each of the at least one image in the bird's-eye view space.

3. The method according to claim 2, wherein: The step of determining a conversion matrix from a camera coordinate system of a multi-camera of the multi-camera system to a bird's-eye view coordinate system based on internal parameters of the multi-camera system and vehicle parameters includes: Respectively obtain the camera intrinsic parameters and camera extrinsic parameters of the multi-camera in the multi-camera system, and obtain the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system; Based on the camera extrinsic parameters, camera intrinsic parameters and the transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system of the multi-camera, a transformation matrix from the camera coordinate system of the multi-camera to the bird's-eye view coordinate system is determined.

4. The method according to claim 1, wherein: Also includes: In the training phase of the prediction network, a first loss function is constructed between the heat map predicted by the prediction network and the true value heat map, and a second loss function is constructed between other attribute maps predicted by the prediction network and other true value attribute maps; A total loss function of the prediction network in a training phase is determined according to the first loss function and the second loss function to supervise the training process of the prediction network.

5. The method according to claim 4, wherein: The determining the total loss function of the prediction network in the training phase according to the first loss function and the second loss function includes: Obtaining a weight value of a first loss function and a weight value of a second loss function; Based on the first loss function, the weight value of the first loss function, the second loss function and the weight value of the second loss function, a total loss function of the prediction network in the training phase is determined.

6. The method according to claim 1, wherein: The step of performing target prediction on the target object in the bird's-eye view fusion feature to obtain three-dimensional spatial information of the target object includes: Extracting the bird's-eye view fusion feature using a neural network to obtain bird's-eye view fusion feature data containing target object features; The prediction network is used to perform target prediction on the target object in the bird's-eye view fusion feature data containing the target object features to obtain the three-dimensional spatial information of the target object.

7. The method according to claim 1, wherein: The step of extracting features from the at least one image to obtain feature data corresponding to each of the at least one image in the multi-camera viewing space and containing features of the target object includes: A deep neural network is used to perform convolution calculations on the images corresponding to each perspective, and feature data of multiple different resolutions containing target object features corresponding to the images corresponding to each perspective in the multi-camera perspective space are obtained.

8. A 3D object detection device based on multi-view fusion, comprising: An image receiving module, used to acquire at least one image from the perspective of multiple cameras; A feature extraction module, used to extract features from the at least one image acquired by the image receiving module, and obtain feature data containing target object features corresponding to each of the at least one image in a multi-camera viewing space; An image feature mapping module, used to map the feature data corresponding to each of the at least one image in the multi-camera viewing space obtained by the feature extraction module to the same bird's-eye view viewing space based on the internal parameters of the multi-camera system and the vehicle parameters, so as to obtain the feature data corresponding to each of the at least one image in the bird's-eye view viewing space; An image fusion module, used for fusing the feature data corresponding to each of the at least one image obtained by the image mapping module in the bird's-eye view space to obtain a bird's-eye view fusion feature; A 3D detection module, used to perform target prediction on the target object in the bird's-eye view fusion feature obtained by the image fusion module to obtain three-dimensional spatial information of the target object; The 3D detection module is specifically used to obtain, from the bird's-eye view fusion feature, a heat map corresponding to a first preset coordinate value of the target object in the bird's-eye view coordinate system using a prediction network, and to obtain other attribute maps for determining a second preset coordinate value, size, and orientation angle of the target object in the bird's-eye view coordinate system; Determine the first preset coordinate value of the target object in the bird's-eye view coordinate system based on the peak information in the heat map, and determine the second preset coordinate value, the size and the orientation angle of the target object in the bird's-eye view coordinate system from the other attribute maps based on the first preset coordinate value; determine the three-dimensional spatial information of the target object based on the first preset coordinate value, the second preset coordinate value, the size and the orientation angle.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the 3D object detection method based on multi-view fusion as described in any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the 3D target detection method based on multi-view fusion described in any one of claims 1-7 above.

Images