Target detection method and device

Abstract

The embodiment of the invention discloses a target detection method and device. A specific embodiment of the method comprises the following steps: extracting two-dimensional image features of a fisheye image shot by a fisheye camera; calculating a three-dimensional position implicit code of the fisheye image based on the parameters of the fisheye camera; and performing target detection based on the two-dimensional image features and the three-dimensional position implicit coding of the fisheye image to obtain a target detection result. According to the embodiment, the distortion parameters of the fisheye camera are introduced into the projection process of the two-dimensional features to the three-dimensional space, so that three-dimensional target detection of camera distortion perception is realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a target detection method and device. BACKGROUND

[0002] With the continuous development of automatic driving perception technology, using vehicle-side multi-camera input images to cooperate with perception algorithms to perform three-dimensional target detection in the space of an automatic driving vehicle, and then inputting the three-dimensional target detection result to a downstream decision control module to perform automatic driving, has gradually become a mainstream solution for low-cost automatic driving. The current mainstream technology first performs a de-distortion operation on multi-camera images; then extracts two-dimensional features of the de-distortion images using a neural network; then projects the two-dimensional features into a three-dimensional space by using a converter and internal and external parameters of each camera to obtain three-dimensional features; and finally obtains a three-dimensional target detection result by performing inference on the three-dimensional features through a neural network, thereby supporting the performance of downstream planning and control tasks of automatic driving.

[0003] A fisheye camera has the advantage of a large field of view, and can better compensate for the field-of-view blind area of a traditional camera, and is often used in an auxiliary driving system to assist parking. However, existing three-dimensional target detection technology is not suitable for directly using fisheye images with a wider field of view. On the one hand, performing a de-distortion operation on fisheye images will cause a substantial loss of the effective FOV (Field of view) of the fisheye camera, and the advantage of a wide viewing angle of the fisheye camera is lost; on the other hand, the imaging principle of the fisheye camera does not satisfy rigid body linear transformation, so that two-dimensional features cannot be projected into a three-dimensional space through linear geometric transformation of internal and external parameters of the camera. SUMMARY

[0004] Embodiments of the present disclosure provide a target detection method and device.

[0005] In a first aspect, embodiments of the present disclosure provide a target detection method, which includes: extracting two-dimensional image features of a fisheye image captured by a fisheye camera; calculating three-dimensional position implicit encodings of the fisheye image based on parameters of the fisheye camera; and performing target detection based on the two-dimensional image features and the three-dimensional position implicit encodings of the fisheye image to obtain a target detection result.

[0006] In a second aspect, embodiments of the present disclosure provide a target detection device, which includes: a first extraction unit configured to extract two-dimensional image features of a fisheye image captured by a fisheye camera; a first calculation unit configured to calculate three-dimensional position implicit encodings of the fisheye image based on parameters of the fisheye camera; and a detection unit configured to perform target detection based on the two-dimensional image features and the three-dimensional position implicit encodings of the fisheye image to obtain a target detection result.

[0007] In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a memory device storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any implementation of the first aspect.

[0008] In a fourth aspect, embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any implementation of the first aspect.

[0009] The method provided by the above-mentioned embodiments of the present disclosure performs three-dimensional target detection based on input images of fisheye cameras in multiple different orientations of an autonomous vehicle. By introducing fisheye camera distortion parameters into the projection process of two-dimensional features to three-dimensional space, three-dimensional target detection with camera distortion perception is achieved. This method does not need to perform distortion correction on fisheye images, so it can complete three-dimensional target detection while retaining the full FOV of the fisheye camera. BRIEF DESCRIPTION OF DRAWINGS

[0010] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments thereof as taken in conjunction with the accompanying drawings:

[0011] Figure 1 is an exemplary system architecture diagram to which one embodiment of the present disclosure can be applied;

[0012] Figure 2 is a flowchart of one embodiment of a target detection method according to the present disclosure;

[0013] Figure 3 is a flowchart of yet another embodiment of a target detection method according to the present disclosure;

[0014] Figure 4 is a block diagram of one application scenario of a target detection method according to an embodiment of the present disclosure;

[0015] Figure 5 is a structural schematic diagram of one embodiment of a target detection device according to the present disclosure;

[0016] Figure 6 is a structural schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] The present disclosure will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of convenience, only the parts related to the application are shown in the drawings.

[0018] It should be noted that the collection, updating, analysis, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with relevant laws and regulations, are used for legitimate and reasonable purposes, and are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by national regulatory authorities. Necessary measures should be taken to selectively block the use or access to personal information data to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Furthermore, once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 An exemplary architecture 100 is shown, to which embodiments of the target detection method or target detection apparatus of this disclosure may be applied.

[0021] like Figure 1 As shown, system architecture 100 may include a pinhole camera 101, a fisheye camera 102, a network 103, and a server 104. Network 103 serves as the medium for providing a communication link between the pinhole camera 101, the fisheye camera 102, and the server 104. Network 103 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0022] The pinhole camera 101 and fisheye camera 102 can be installed on an autonomous vehicle and interact with the server 104 via the network 103 to receive or send messages, etc. The pinhole camera 101 and fisheye camera 102 can capture images of the area around the autonomous vehicle and send them to the server 104 for processing.

[0023] Server 104 can be an onboard server for autonomous vehicles or a cloud server. Server 104 can analyze and process received images and generate processing results (such as object detection results) for subsequent control of autonomous vehicles.

[0024] It should be noted that the target detection method provided in the embodiments of this disclosure is generally executed by server 104, and correspondingly, the target detection device is generally set in server 104.

[0025] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 104 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0026] It should be understood that Figure 1 The number of pinhole cameras, fisheye cameras, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of pinhole cameras, fisheye cameras, networks, and servers can be included.

[0027] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a target detection method according to the present disclosure. The target detection method includes the following steps:

[0028] Step 201: Extract the two-dimensional image features of the fisheye image captured by the fisheye camera.

[0029] In this embodiment, the execution body of the target detection method (e.g.) Figure 1 The server 104 shown can extract two-dimensional image features from fisheye images captured by a fisheye camera.

[0030] Autonomous vehicles can be equipped with multiple fisheye cameras facing different directions. A fisheye camera is a camera with a fisheye lens, which is a lens with an extremely short focal length and a very wide field of view. Fisheye cameras can be used to capture fisheye images with a wider field of view. Fisheye images are two-dimensional images, and feature extraction can be performed on fisheye images to obtain their two-dimensional image features.

[0031] In some embodiments, the fisheye image is first preprocessed to obtain a preprocessed fisheye image; then the preprocessed fisheye image is input into a neural network to obtain the two-dimensional image features of the fisheye image.

[0032] The first preprocessing step may include, but is not limited to, at least one of the following: image format conversion, image scaling, and image normalization. It should be noted that the first preprocessing step does not include distortion correction, thus preserving the full field of view (FOV) of the fisheye camera. Furthermore, considering the relatively short effective sensing distance of the fisheye camera, the fisheye image can be scaled to a smaller ratio to reduce algorithm complexity.

[0033] Neural networks, such as deformable convolutional neural networks (CNNs), can be used to extract two-dimensional features from images. The distribution of the convolutional kernels in a deformable CNN is learned and variable. Because fisheye images have a large field of view, they suffer from significant distortion. Deformable CNNs can be used to dynamically find features in fisheye images.

[0034] Step 202: Calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera.

[0035] In this embodiment, the aforementioned execution entity can calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera.

[0036] In some embodiments, the parameters of the fisheye camera are first preprocessed to obtain second preprocessed parameters. Then, based on the second preprocessed parameters, the preprocessed fisheye image is projected from the image coordinate system to the camera coordinate system, and then projected into three-dimensional space to obtain the implicit three-dimensional position encoding of the fisheye image. The second preprocessed parameters may include at least one of the following: second preprocessing intrinsic parameters and second preprocessing extrinsic parameters. Based on the focal length, principal point offset, and distortion parameters in the second preprocessing intrinsic parameters, the preprocessed fisheye image can be projected from the image coordinate system to the camera coordinate system, and then projected into three-dimensional space through the second preprocessing extrinsic parameters to obtain the implicit three-dimensional position encoding of the fisheye image. The calculation process is as follows:

[0037] The first step is to construct a four-dimensional fisheye tensor F_Coords with dimensions (H,W,D,3) based on the preprocessed fisheye image and its two-dimensional image features.

[0038] Where H represents the height of the two-dimensional image feature of the fisheye image. W represents the width of the two-dimensional image feature of the fisheye image. D represents the number of depth bins of the two-dimensional image feature of the fisheye image, calculated using the camera's maximum visible range and depth resolution. For example, if the camera's maximum visible range is 100 meters and the depth resolution is 1 meter, the number of depth bins is equal to the quotient of the camera's maximum visible range divided by the depth resolution, i.e., 100. 3 represents three coordinates, namely F_Coords. x F_Coords y F_Coords z This corresponds to the width, height, and depth of the preprocessed fisheye image. The four-dimensional fisheye tensor F_Coords can represent the three-dimensional coordinates of the two-dimensional image features of the fisheye image in the image coordinate system.

[0039] The second step is to repeat the four-dimensional fisheye tensor F_Coords according to the number of fisheye images to obtain a five-dimensional fisheye tensor F_Coords with dimensions (N,H,W,D,3).

[0040] Where N is the number of fisheye images.

[0041] The third step involves using the second preprocessing internal parameter K. FThe focal lengths fx, fy, principal point offsets cx, cy, and distortion parameters ds(ds0, ds1, ds2, ds3) are used to project the five-dimensional fisheye tensor F_Coords onto the camera coordinate system, resulting in the three-dimensional fisheye coordinates F_Cs(F_Cs) in the camera coordinate system. x ,F_Cs y ,F_Cs z ).

[0042] The calculation process is as follows:

[0043] x_norm=(F_Coords x -cx) / fx;

[0044] y_norm=(F_Coords y -cy) / fy;

[0045] r2 = x_norm 2 +y_norm 2 ;

[0046] r4 = r2 2 ;

[0047] r6 = r2 × r4;

[0048] r8 = r4 2 ;

[0049] cr=1.0+ds0×r2+ds1×r4+ds2×r6+ds3×r8;

[0050] F_Cs x = x_norm × cr;

[0051] F_Cs y =y_norm×cr;

[0052] F_Cs z =F_Coords z .

[0053] Among them, x_norm, y_norm, r2, r4, r6, r8, and cr are intermediate variables.

[0054] Fourth step, use the second preprocessing extrinsic parameter RT F Projecting the fisheye 3D coordinate point F_Cs in the camera coordinate system onto the radar coordinate system yields the fisheye 3D coordinate point L in the radar coordinate system with dimensions (N,H,W,D,3). F .

[0055] Among them, L F =RT F ×F_Cs.

[0056] The fifth step is to use a convolutional neural network to process the three-dimensional coordinates of the fisheye point L in the radar coordinate system. F The processing yields the three-dimensional implicit encoding of the fisheye image with dimensions (N,H,W,C,3).

[0057] Where C represents the channel dimension of the encoding.

[0058] Step 203: Target detection is performed based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image to obtain the target detection result.

[0059] In this embodiment, the aforementioned execution entity can perform target detection based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image to obtain the target detection result.

[0060] In some embodiments, a three-dimensional target query vector set is first generated; then, target detection is performed based on the three-dimensional target query vector set, the two-dimensional image features of the fisheye image, and the implicit encoding of the three-dimensional position, to obtain the target detection result. Specifically, a corresponding number of sparse three-dimensional target query vector sets can be generated according to a pre-set maximum number of targets to be identified. Each vector in the three-dimensional target query vector set corresponds to a high-dimensional vector representation of the three-dimensional position of a potential target, and these vectors are all randomly initialized.

[0061] In some embodiments, the 3D target query vector set, the 2D image features of the fisheye image, and the 3D position implicit encoding are first input into a multilayer transformer decoder to obtain a 3D target feature vector set. Then, the 3D target feature vector set is input into a target classification network and a target regression network to obtain the target detection result. Here, the 3D target query vector set is the query input. The 2D image features of the fisheye image and the 3D position implicit encoding are the query inputs. The 3D target feature vector corresponding to each query vector is calculated through multiple cross-attention mechanisms between the query input and the query input. The target detection result is a 3D target detection result, including the category, center point position, length, width, height, and yaw angle of each target, etc.

[0062] The method provided in the above embodiments of this disclosure performs 3D target detection based on input images from multiple fisheye cameras facing different directions on an autonomous vehicle. By incorporating fisheye camera distortion parameters into the projection process of 2D features into 3D space, 3D target detection with camera distortion awareness is achieved. This method does not require distortion correction of the fisheye images, thus enabling 3D target detection while preserving the entire field of view (FOV) of the fisheye camera.

[0063] Further reference Figure 3This illustrates a flow 300 of another embodiment of the target detection method according to the present disclosure. The target detection method includes the following steps:

[0064] Step 301: Extract the two-dimensional image features of the fisheye image captured by the fisheye camera, and extract the two-dimensional image features of the pinhole image captured by the pinhole camera.

[0065] In this embodiment, the execution body of the target detection method (e.g.) Figure 1 The server 104 shown can extract two-dimensional image features from fisheye images captured by a fisheye camera, and extract two-dimensional image features from pinhole images captured by a pinhole camera.

[0066] Autonomous vehicles can be equipped with multiple fisheye cameras and pinhole cameras facing different directions. A fisheye camera is a camera with a fisheye lens, a type of lens with an extremely short focal length and a very wide field of view. Fisheye cameras can capture fisheye images with a wider field of view. Fisheye images are two-dimensional images; feature extraction from fisheye images yields their two-dimensional image features. A pinhole camera is a camera that follows the principle of pinhole imaging. When a plate with a small hole is placed between a screen and an object, an inverted image of the object is formed on the screen; this phenomenon is called pinhole imaging. Pinhole images are two-dimensional images; feature extraction from pinhole images yields their two-dimensional image features.

[0067] In some embodiments, the fisheye image is first preprocessed to obtain a preprocessed fisheye image; then the preprocessed fisheye image is input into a deformable convolutional neural network to obtain the two-dimensional image features of the fisheye image. Similarly, the pinhole image is first preprocessed to obtain a preprocessed pinhole image; then the preprocessed pinhole image is input into a convolutional neural network to obtain the two-dimensional image features of the pinhole image.

[0068] The first preprocessing step may include, but is not limited to, at least one of the following: image format conversion, image scaling, and image normalization. The third preprocessing step may include, but is not limited to, at least one of the following: image format conversion, image scaling, image normalization, and distortion correction. It should be noted that the first preprocessing step does not include distortion correction, thus preserving the entire field of view (FOV) of the fisheye camera. The third preprocessing step includes distortion correction, thereby correcting distortion in the pinhole image. Furthermore, considering the relatively short effective sensing distance of the fisheye camera, to reduce algorithm complexity, the fisheye image can be scaled down to a size smaller than the pinhole image.

[0069] In a convolutional neural network (CNN), the convolutional kernels are pre-defined and fixed. In a deformable CNN, the distribution of the convolutional kernels is learned and is variable. Because fisheye images have a large field of view, they suffer from significant distortion. Deformable CNNs can be used to dynamically find features in fisheye images.

[0070] Step 302: Calculate the three-dimensional position implicit code of the fisheye image based on the parameters of the fisheye camera, and calculate the three-dimensional position implicit code of the pinhole image based on the parameters of the pinhole camera.

[0071] In this embodiment, the aforementioned execution entity can calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera, and calculate the three-dimensional position implicit encoding of the pinhole image based on the parameters of the pinhole camera.

[0072] In some embodiments, the parameters of the fisheye camera are first preprocessed to obtain second preprocessed parameters. Then, based on the second preprocessed parameters, the preprocessed fisheye image is projected from the image coordinate system to the camera coordinate system, and then projected into three-dimensional space to obtain the implicit three-dimensional position encoding of the fisheye image. The second preprocessed parameters may include at least one of the following: second preprocessing intrinsic parameters and second preprocessing extrinsic parameters. Based on the focal length, principal point offset, and distortion parameters in the second preprocessing intrinsic parameters, the preprocessed fisheye image can be projected from the image coordinate system to the camera coordinate system, and then projected into three-dimensional space through the second preprocessing extrinsic parameters to obtain the implicit three-dimensional position encoding of the fisheye image. The calculation process is as follows:

[0073] The first step is to construct a four-dimensional fisheye tensor F_Coords with dimensions (H,W,D,3) based on the preprocessed fisheye image and its two-dimensional image features.

[0074] Where H represents the height of the two-dimensional image feature of the fisheye image. W represents the width of the two-dimensional image feature of the fisheye image. D represents the number of depth bins of the two-dimensional image feature of the fisheye image, calculated using the camera's maximum visible range and depth resolution. For example, if the camera's maximum visible range is 100 meters and the depth resolution is 1 meter, the number of depth bins is equal to the quotient of the camera's maximum visible range divided by the depth resolution, i.e., 100. 3 represents three coordinates, namely F_Coords. x F_Coords y F_Coords z This corresponds to the width, height, and depth of the preprocessed fisheye image. The four-dimensional fisheye tensor F_Coords can represent the three-dimensional coordinates of the two-dimensional image features of the fisheye image in the image coordinate system.

[0075] The second step is to repeat the four-dimensional fisheye tensor F_Coords according to the number of fisheye images to obtain a five-dimensional fisheye tensor F_Coords with dimensions (N,H,W,D,3).

[0076] Where N is the number of fisheye images.

[0077] The third step involves using the second preprocessing internal parameter K. FThe focal lengths fx, fy, principal point offsets cx, cy, and distortion parameters ds(ds0, ds1, ds2, ds3) are used to project the five-dimensional fisheye tensor F_Coords onto the camera coordinate system, resulting in the three-dimensional fisheye coordinates F_Cs(F_Cs) in the camera coordinate system. x ,F_Cs y ,F_Cs z ).

[0078] The calculation process is as follows:

[0079] x_norm=(F_Coords x -cx) / fx;

[0080] y_norm=(F_Coords y -cy) / fy;

[0081] r2 = x_norm 2 +y_norm 2 ;

[0082] r4 = r2 2 ;

[0083] r6 = r2 × r4;

[0084] r8 = r4 2 ;

[0085] cr=1.0+ds0×r2+ds1×r4+ds2×r6+ds3×r8;

[0086] F_Cs x = x_norm × cr;

[0087] F_Cs y =y_norm×cr;

[0088] F_Cs z =F_Coords z .

[0089] Among them, x_norm, y_norm, r2, r4, r6, r8, and cr are intermediate variables.

[0090] Fourth step, use the second preprocessing extrinsic parameter RT F Projecting the fisheye 3D coordinate point F_Cs in the camera coordinate system onto the radar coordinate system yields the fisheye 3D coordinate point L in the radar coordinate system with dimensions (N,H,W,D,3). F .

[0091] Among them, L F =RT F ×F_Cs.

[0092] The fifth step is to use a convolutional neural network to process the three-dimensional coordinates of the fisheye point L in the radar coordinate system. F The processing yields the three-dimensional implicit encoding of the fisheye image with dimensions (N,H,W,C,3).

[0093] Where C represents the channel dimension of the encoding.

[0094] In some embodiments, the parameters of the pinhole camera are first preprocessed to obtain fourth preprocessing parameters. Then, based on the fourth preprocessing parameters, the preprocessed pinhole image is projected into three-dimensional space to obtain the implicit three-dimensional position encoding of the pinhole image. The fourth preprocessing parameters include at least one of the following: fourth preprocessing intrinsic parameters and fourth preprocessing extrinsic parameters. Based on the focal length and principal point offset in the fourth preprocessing intrinsic parameters, and the fourth preprocessing extrinsic parameters, the coordinate projection matrix of the preprocessed pinhole image onto the vehicle can be calculated, and the preprocessed pinhole image can be projected into three-dimensional space using the coordinate projection matrix to obtain the implicit three-dimensional position encoding of the pinhole image.

[0095] The calculation process is as follows:

[0096] The first step is to construct a four-dimensional pinhole tensor P_Coords with dimensions (H,W,D,3) based on the pinhole image and its two-dimensional image features.

[0097] Where H represents the height of the two-dimensional image feature of the pinhole image. W represents the width of the two-dimensional image feature of the pinhole image. D represents the number of depth bins of the two-dimensional image feature of the pinhole image, calculated using the camera's maximum visible range and depth resolution. For example, if the camera's maximum visible range is 100 meters and the depth resolution is 1 meter, the number of depth bins is equal to the quotient of the camera's maximum visible range divided by the depth resolution, i.e., 100. 3 represents three coordinates, namely P_Coords x P_Coords y P_Coords z The width, height, and depth of the preprocessed pinhole image are represented by these values. The four-dimensional pinhole tensor P_Coords can characterize the three-dimensional coordinates of the two-dimensional image features of the pinhole image in the image coordinate system.

[0098] The second step is to repeat the four-dimensional hole tensor P_Coords according to the number of hole images to obtain a five-dimensional hole tensor P_Coords with dimensions (N,H,W,D,3).

[0099] Where N is the number of pinhole images.

[0100] The third step involves using the fourth preprocessing internal parameter K. P and the fourth preprocessing extrinsic parameter RT PProjecting the five-dimensional tensor P_Coords of the pinhole onto the radar coordinate system yields the three-dimensional coordinates L of the pinhole in the radar coordinate system with dimensions (N,H,W,D,3). P .

[0101] Among them, L P =RT P ×K P ×P_Coords.

[0102] The fourth step is to use a convolutional neural network to analyze the three-dimensional coordinates of the pinhole point L in the radar coordinate system. P The processing yields the three-dimensional implicit encoding of the pinhole image with dimensions (N,H,W,C,3).

[0103] Where C represents the channel dimension of the encoding.

[0104] Step 303: Based on the two-dimensional image features and three-dimensional position implicit coding of the fisheye image, and the two-dimensional image features and three-dimensional position implicit coding of the pinhole image, target detection is performed to obtain the target detection result.

[0105] In this embodiment, the aforementioned execution entity can perform target detection based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, as well as the two-dimensional image features and three-dimensional position implicit encoding of the pinhole image, to obtain the target detection result.

[0106] In some embodiments, a three-dimensional target query vector group is first generated; then, the three-dimensional target query vector group, the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, and the two-dimensional image features and three-dimensional position implicit encoding of the pinhole image are input into the multilayer converter decoder to obtain a three-dimensional target feature vector group; finally, the three-dimensional target feature vector group is input into the target classification network and the target regression network to obtain the target detection result.

[0107] Based on a pre-defined maximum number of targets to be identified, a corresponding number of sparse 3D target query vector sets can be generated. Each vector in the 3D target query vector set corresponds to a high-dimensional vector representation of the 3D position of a potential target, and these vectors are all randomly initialized. The 3D target query vector sets are the query inputs. The 2D image features and implicit 3D position encoding of the fisheye image are the query inputs. The 3D target feature vectors corresponding to each query vector are calculated through multiple cross-attention mechanisms between the query inputs and the query inputs. The target detection result is a 3D target detection result, including the category, center point position, length, width, height, and yaw angle of each target, etc.

[0108] The method provided in the above embodiments of this disclosure performs 3D target detection based on input images from multiple pinhole cameras and fisheye cameras facing different directions in an autonomous vehicle. By incorporating fisheye camera distortion parameters into the projection process of 2D features into 3D space, 3D target detection with camera distortion perception is achieved. This method does not require distortion correction of the fisheye images, thus enabling 3D target detection while preserving the entire field of view (FOV) of the fisheye camera. Furthermore, to address the shortcomings of conventional convolutional neural networks directly extracting features from fisheye images with significant distortion, separate neural networks are used for feature extraction of pinhole and fisheye images from the autonomous vehicle. Specifically, conventional convolutional neural networks are used directly for feature extraction of pinhole camera images, while deformable convolutional neural networks are used for feature extraction of fisheye camera images. This fully leverages the large FOV advantage of fisheye cameras, significantly reducing blind spots in the autonomous driving perception system and improving the safety and intelligence of autonomous driving.

[0109] For ease of understanding, Figure 4 A block diagram illustrating an application scenario of a target detection method according to an embodiment of the present disclosure is shown.

[0110] The first step involves installing multiple pinhole cameras facing different directions on the autonomous vehicle to capture pinhole image 1, pinhole image 2, ..., pinhole image N. Similarly, multiple fisheye cameras facing different directions are installed on the autonomous vehicle to capture fisheye image 1, fisheye image 2, ..., fisheye image N.

[0111] The second step involves preprocessing the pinhole images 1, 2, ..., N, and then inputting them into a convolutional neural network for feature extraction to obtain the two-dimensional features of the pinhole images. Similarly, the fisheye images 1, 2, ..., N are preprocessed and then input into a deformable convolutional neural network for feature extraction to obtain the two-dimensional features of the fisheye images. During preprocessing, distortion correction is only performed on the pinhole images.

[0112] The third step involves preprocessing the intrinsic and extrinsic parameters of the pinhole camera to encode the two-dimensional features of the pinhole image, resulting in an implicit three-dimensional position code for the pinhole. Similarly, the intrinsic and extrinsic parameters and distortion parameters of the fisheye camera are preprocessed to encode the two-dimensional features of the fisheye image, resulting in an implicit three-dimensional position code for the fisheye.

[0113] The fourth step is to generate a sparse three-dimensional target query vector group based on the pre-set maximum number of targets to be identified.

[0114] The fifth step involves using the implicit encoding of the three-dimensional position of the small hole and the implicit encoding of the three-dimensional position of the fisheye as the input to the query, and inputting them together with the sparse three-dimensional target query vector group into the converter decoder to obtain the three-dimensional target feature vector group.

[0115] The sixth step is to input the three-dimensional target feature vector group into the target classification network and the target regression network, and output the three-dimensional target detection results.

[0116] The implementation of the methods shown in the above figures, this disclosure provides an embodiment of a target detection device, which is similar to... Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 5 As shown, the target detection device 500 provided in this embodiment includes a first extraction unit 501, a first calculation unit 502, and a detection unit 503. The first extraction unit 501 is configured to extract two-dimensional image features from a fisheye image captured by a fisheye camera; the first calculation unit 502 is configured to calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera; and the detection unit 503 is configured to perform target detection based on the two-dimensional image features and the three-dimensional position implicit encoding of the fisheye image to obtain a target detection result.

[0118] In this embodiment, the specific processing of the first extraction unit 501, the first calculation unit 502, and the detection unit 503 in the target detection device 500, and the resulting technical effects, can be found in reference to [reference needed]. Figure 2 The relevant descriptions of steps 201, 202 and 203 in the corresponding embodiments will not be repeated here.

[0119] In some optional implementations of this embodiment, the first extraction unit 501 is further configured to: perform a first preprocessing on the fisheye image to obtain a preprocessed fisheye image, wherein the first preprocessing includes at least one of the following: image format conversion, image scaling and image normalization; and input the preprocessed fisheye image into a neural network to obtain the two-dimensional image features of the fisheye image.

[0120] In some optional implementations of this embodiment, the neural network is a deformable convolutional neural network.

[0121] In some optional implementations of this embodiment, the first calculation unit 502 includes: a second preprocessing subunit configured to perform second preprocessing on the parameters of the fisheye camera to obtain second preprocessing parameters; and a first projection subunit configured to project the preprocessed fisheye image from the image coordinate system to the camera coordinate system based on the second preprocessing parameters, and then project it to three-dimensional space to obtain the three-dimensional position implicit encoding of the fisheye image.

[0122] In some optional implementations of this embodiment, the second preprocessing parameter includes at least one of the following: a second preprocessing intrinsic parameter and a second preprocessing extrinsic parameter.

[0123] In some optional implementations of this embodiment, the first projection subunit includes: a first projection module, configured to project the preprocessed fisheye image from the image coordinate system to the camera coordinate system based on the focal length, principal point offset and distortion parameters in the second preprocessing intrinsic parameters, and then project it to three-dimensional space through the second preprocessing extrinsic parameters to obtain the three-dimensional position implicit encoding of the fisheye image. In some optional implementations of this embodiment, the first projection module is further configured to: construct a four-dimensional fisheye tensor based on the preprocessed fisheye image and its two-dimensional image features, wherein the four-dimensional fisheye tensor represents the three-dimensional coordinate points of the two-dimensional image features of the fisheye image in the image coordinate system; repeat the four-dimensional fisheye tensor according to the number of fisheye images to obtain a five-dimensional fisheye tensor; project the five-dimensional fisheye tensor onto the camera coordinate system using the focal length, principal point offset, and distortion parameters in the second preprocessing intrinsic parameters to obtain the three-dimensional fisheye coordinate points in the camera coordinate system; project the three-dimensional fisheye coordinate points in the camera coordinate system onto the radar coordinate system using the second preprocessing extrinsic parameters to obtain the three-dimensional fisheye coordinate points in the radar coordinate system; and process the three-dimensional fisheye coordinate points in the radar coordinate system using a convolutional neural network to obtain the implicit three-dimensional position encoding of the fisheye image.

[0124] In some optional implementations of this embodiment, the detection unit 503 includes: a generation subunit configured to generate a three-dimensional target query vector set, wherein one vector in the three-dimensional target query vector set corresponds to one target; and a detection subunit configured to perform target detection based on the three-dimensional target query vector set, the two-dimensional image features of the fisheye image, and the three-dimensional position implicit encoding, to obtain a target detection result. In some optional implementations of this embodiment, the detection subunit is further configured to: input the three-dimensional target query vector set, the two-dimensional image features of the fisheye image, and the three-dimensional position implicit encoding to a multilayer converter decoder to obtain a three-dimensional target feature vector set; and input the three-dimensional target feature vector set to a target classification network and a target regression network to obtain a target detection result. In some optional implementations of this embodiment, the target detection device 500 further includes: a second extraction unit configured to extract two-dimensional image features of a pinhole image captured by a pinhole camera; a second calculation unit configured to calculate a three-dimensional position implicit code of the pinhole image based on the parameters of the pinhole camera; and a detection unit 503 further configured to perform target detection based on the two-dimensional image features and three-dimensional position implicit code of the fisheye image, and the two-dimensional image features and three-dimensional position implicit code of the pinhole image, to obtain a target detection result.

[0125] In some optional implementations of this embodiment, the detection unit 503 is further configured to input the three-dimensional target query vector group, the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, and the two-dimensional image features and three-dimensional position implicit encoding of the pinhole image into the multilayer converter decoder to obtain the three-dimensional target feature vector group; and input the three-dimensional target feature vector group into the target classification network and the target regression network to obtain the target detection result.

[0126] In some optional implementations of this embodiment, the second extraction unit is further configured to: perform a third preprocessing on the pinhole image to obtain a preprocessed pinhole image, wherein the third preprocessing includes at least one of the following: image format conversion, image scaling, image normalization, and distortion correction; and input the preprocessed pinhole image into a convolutional neural network to obtain the two-dimensional image features of the pinhole image.

[0127] In some optional implementations of this embodiment, the second calculation unit includes: a second preprocessing subunit configured to perform a fourth preprocessing on the parameters of the pinhole camera to obtain second preprocessing parameters, wherein the fourth preprocessing parameters include fourth preprocessing intrinsic parameters and fourth preprocessing extrinsic parameters; and a second projection subunit configured to calculate the coordinate projection matrix of the preprocessed pinhole image onto the vehicle based on the focal length and principal point offset in the fourth preprocessing intrinsic parameters and the fourth preprocessing extrinsic parameters, and to project the preprocessed pinhole image onto a three-dimensional space using the coordinate projection matrix to obtain the three-dimensional position implicit encoding of the pinhole image.

[0128] In some optional implementations of this embodiment, the second projection subunit is further configured to: construct a four-dimensional pinhole tensor based on the preprocessed pinhole image and the two-dimensional image features of the pinhole image, wherein the four-dimensional pinhole tensor represents the three-dimensional coordinate points of the two-dimensional image features of the pinhole image in the image coordinate system; repeat the four-dimensional pinhole tensor according to the number of pinhole images to obtain a five-dimensional pinhole tensor; project the five-dimensional pinhole tensor onto the radar coordinate system through the fourth preprocessing intrinsic parameter and the fourth preprocessing extrinsic parameter to obtain the three-dimensional coordinate points of the pinhole in the radar coordinate system; and process the three-dimensional coordinate points of the pinhole in the radar coordinate system through a convolutional neural network to obtain the implicit encoding of the three-dimensional position of the pinhole image.

[0129] The apparatus provided in the above embodiments of this disclosure performs 3D target detection based on input images from multiple fisheye cameras facing different directions on an autonomous vehicle. By incorporating fisheye camera distortion parameters into the projection process of 2D features into 3D space, 3D target detection with camera distortion awareness is achieved. This apparatus does not require distortion correction processing of the fisheye images, thus enabling 3D target detection while preserving the entire field of view (FOV) of the fisheye camera.

[0130] The following is for reference. Figure 6It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 1 The diagram shows the structure of the server (600) in this disclosure. The terminal devices in the embodiments of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The server shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0131] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0132] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0133] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by a processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0134] It should be noted that the computer-readable medium described in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0135] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: extract two-dimensional image features of a fisheye image captured by a fisheye camera; calculate a three-dimensional position implicit code of the fisheye image based on the parameters of the fisheye camera; and perform target detection based on the two-dimensional image features and the three-dimensional position implicit code of the fisheye image to obtain a target detection result.

[0136] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The units described in the embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first extraction unit, a first calculation unit, a generation unit, and a detection unit. The names of these units do not necessarily limit the specific unit; for example, the first extraction unit may also be described as "a unit for extracting two-dimensional image features from a fisheye image captured by a fisheye camera."

[0139] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A target detection method, comprising: extracting two-dimensional image features of a fisheye image captured by a fisheye camera; calculating three-dimensional position implicit encodings of the fisheye image based on parameters of the fisheye camera; performing target detection based on the two-dimensional image features and the three-dimensional position implicit encodings of the fisheye image to obtain a target detection result.

2. The method of claim 1, wherein, The extracting of the two-dimensional image features of the fisheye image captured by the fisheye camera comprises: performing first preprocessing on the fisheye image to obtain a preprocessed fisheye image, wherein the first preprocessing comprises at least one of image format conversion, image scaling, and image normalization; inputting the preprocessed fisheye image into a neural network to obtain the two-dimensional image features of the fisheye image.

3. The method of claim 2, wherein, The neural network is a deformable convolutional neural network.

4. The method of claim 1, wherein, The calculating of the three-dimensional position implicit encodings of the fisheye image based on the parameters of the fisheye camera comprises: performing second preprocessing on the parameters of the fisheye camera to obtain second preprocessed parameters; projecting the preprocessed fisheye image from an image coordinate system to a camera coordinate system and then to a three-dimensional space based on the second preprocessed parameters to obtain the three-dimensional position implicit encodings of the fisheye image.

5. The method of claim 4, wherein, The second preprocessed parameters comprise at least one of second preprocessed intrinsic parameters and second preprocessed extrinsic parameters.

6. The method of claim 5, wherein, The projecting of the preprocessed fisheye image from the image coordinate system to the camera coordinate system and then to the three-dimensional space based on the second preprocessed parameters to obtain the three-dimensional position implicit encodings of the fisheye image comprises: projecting the preprocessed fisheye image from the image coordinate system to the camera coordinate system based on focal length, principal point offset, and distortion parameters in the second preprocessed intrinsic parameters, and then projecting to the three-dimensional space through the second preprocessed extrinsic parameters to obtain the three-dimensional position implicit encodings of the fisheye image.

7. The method of claim 6, wherein, The projecting of the fisheye image from the image coordinate system to the camera coordinate system based on the focal length, the principal point offset, and the distortion parameters in the second preprocessed intrinsic parameters and then projecting to the three-dimensional space through the second preprocessed extrinsic parameters to obtain the three-dimensional position implicit encodings of the fisheye image comprises: constructing a fisheye four-dimensional tensor based on the preprocessed fisheye image and the two-dimensional image features of the fisheye image, wherein the fisheye four-dimensional tensor represents three-dimensional coordinate points of the two-dimensional image features of the fisheye image in the image coordinate system; repeating the fisheye four-dimensional tensor according to the number of fisheye images to obtain a fisheye five-dimensional tensor; projecting the fisheye five-dimensional tensor to the camera coordinate system through the focal length, the principal point offset, and the distortion parameters in the second preprocessed intrinsic parameters to obtain fisheye three-dimensional coordinate points in the camera coordinate system; projecting the fisheye three-dimensional coordinate points in the camera coordinate system to a radar coordinate system using the second preprocessed extrinsic parameters to obtain fisheye three-dimensional coordinate points in the radar coordinate system; processing the fisheye three-dimensional coordinate points in the radar coordinate system through a convolutional neural network to obtain the three-dimensional position implicit encodings of the fisheye image.

8. The method of claim 1, wherein, The performing of target detection based on the two-dimensional image features and the three-dimensional position implicit encodings of the fisheye image to obtain a target detection result comprises: generate a three-dimensional target query vector group, wherein one vector in the three-dimensional target query vector group corresponds to one target; perform target detection based on the three-dimensional target query vector group, the two-dimensional image feature of the fisheye image, and the three-dimensional position implicit encoding to obtain a target detection result.

9. The method of claim 8, wherein, The target detection based on the three-dimensional target query vector group, the two-dimensional image feature of the fisheye image, and the three-dimensional position implicit encoding to obtain a target detection result includes: inputting the three-dimensional target query vector group, the two-dimensional image feature of the fisheye image, and the three-dimensional position implicit encoding into a multi-layer converter decoder to obtain a three-dimensional target feature vector group; inputting the three-dimensional target feature vector group into a target classification network and a target regression network to obtain the target detection result.

10. The method of any one of claims 1-9, wherein, The method further includes: extracting a two-dimensional image feature of a pinhole image captured by a pinhole camera; calculating a three-dimensional position implicit encoding of the pinhole image based on parameters of the pinhole camera; and The target detection based on the two-dimensional image feature and the three-dimensional position implicit encoding of the fisheye image to obtain a target detection result includes: performing target detection based on the two-dimensional image feature and the three-dimensional position implicit encoding of the fisheye image, and the two-dimensional image feature and the three-dimensional position implicit encoding of the pinhole image to obtain the target detection result.

11. An apparatus for target detection, comprising: a first extraction unit configured to extract a two-dimensional image feature of a fisheye image captured by a fisheye camera; a first calculation unit configured to calculate a three-dimensional position implicit encoding of the fisheye image based on parameters of the fisheye camera; a detection unit configured to perform target detection based on the two-dimensional image feature and the three-dimensional position implicit encoding of the fisheye image to obtain a target detection result.

12. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-10.

13. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method of any one of claims 1-10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images