Object detection method and apparatus

Abstract

Disclosed in the embodiments of the present disclosure are an object detection method and apparatus. A particular embodiment of the method comprises: extracting a two-dimensional image feature of a fisheye image captured by a fisheye camera; on the basis of parameters of the fisheye camera, computing an implicit three-dimensional positional encoding of the fisheye image; and on the basis of the two-dimensional image feature and the implicit three-dimensional positional encoding of the fisheye image, performing object detection to obtain an object detection result. In this embodiment, by means of introducing distortion parameters of the fisheye cameras into the process of projecting two-dimensional features onto a three-dimensional space, camera distortion-aware three-dimensional object detection is realized.

Description

Target detection method and device

[0001] Cross-reference to related applications

[0002] The present application claims priority to Chinese Patent Application No. 202410850763.9 entitled "Target detection method and device" filed on June 27, 2024, the disclosure of which is hereby incorporated by reference in its entirety into the present application as part or parts thereof. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a target detection method and device. BACKGROUND

[0004] With the continuous development of automatic driving perception technology, using vehicle-side multi-camera input images in combination with perception algorithms to perform three-dimensional target detection in the space of an automatic driving vehicle, and then inputting the three-dimensional target detection results to a downstream decision control module for automatic driving, has gradually become a mainstream solution for low-cost automatic driving. The current mainstream technology first performs distortion removal on multi-camera images; then extracts two-dimensional features of the distortion-removed images using a neural network; then projects the two-dimensional features into a three-dimensional space through a converter and the internal and external parameters of each camera to obtain three-dimensional features; and finally obtains three-dimensional target detection results through a neural network inference, thereby supporting the performance of downstream planning and control tasks for automatic driving.

[0005] Fish-eye cameras have the advantage of a large field of view, which can better compensate for the blind area of traditional cameras and are often used in auxiliary driving systems to assist parking. However, existing three-dimensional target detection techniques are not suitable for directly using fish-eye images with a wider field of view. On the one hand, performing distortion removal on fish-eye images will cause a significant loss of the effective FOV (Field of view) of the fish-eye camera, losing the advantage of a wide viewing angle of the fish-eye camera. On the other hand, the imaging principle of the fish-eye camera does not satisfy rigid body linear transformation, which makes it impossible to project two-dimensional features into a three-dimensional space through linear geometric transformation of the camera internal and external parameters. SUMMARY

[0006] Embodiments of the present disclosure provide a target detection method and device.

[0007] According to a first aspect of the present disclosure, embodiments of the present disclosure provide a target detection method, which includes: extracting two-dimensional image features of a fish-eye image captured by a fish-eye camera; calculating three-dimensional position implicit encodings of the fish-eye image based on parameters of the fish-eye camera; and performing target detection based on the two-dimensional image features and the three-dimensional position implicit encodings of the fish-eye image to obtain a target detection result.

[0008] According to a second aspect of the present disclosure, embodiments of the present disclosure provide a target detection apparatus, comprising: a first extraction unit configured to extract a two-dimensional image feature of a fisheye image captured by a fisheye camera; a first calculation unit configured to calculate a three-dimensional position implicit encoding of the fisheye image based on a parameter of the fisheye camera; and a detection unit configured to perform target detection based on the two-dimensional image feature and the three-dimensional position implicit encoding of the fisheye image, to obtain a target detection result.

[0009] According to a third aspect of the present disclosure, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation manner of the first aspect.

[0010] According to a fourth aspect of the present disclosure, embodiments of the present disclosure provide a computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the method as described in any implementation manner of the first aspect.

[0011] The method provided by the above embodiments of the present disclosure performs three-dimensional target detection based on input images of fisheye cameras in multiple different orientations of an autonomous vehicle. By introducing fisheye camera distortion parameters into the projection process of two-dimensional features to three-dimensional space, three-dimensional target detection with camera distortion perception is achieved. This method does not need to perform fisheye image de-distortion processing, and thus can complete three-dimensional target detection while retaining the full FOV of the fisheye camera. BRIEF DESCRIPTION OF DRAWINGS

[0012] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0013] FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure can be applied;

[0014] FIG. 2 is a flowchart of one embodiment of a target detection method according to the present disclosure;

[0015] FIG. 3 is a flowchart of another embodiment of a target detection method according to the present disclosure;

[0016] FIG. 4 is a block diagram of one application scenario of a target detection method according to embodiments of the present disclosure;

[0017] FIG. 5 is a structural schematic diagram of one embodiment of a target detection apparatus according to the present disclosure;

[0018] FIG. 6 is a structural schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] The present disclosure will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings.

[0020] It should be noted that in the technical solutions of the present application, the collection / acquisition, updating, analysis, use, transmission, storage, etc. of user personal information are in line with the relevant legal regulations, are used for legal and reasonable purposes, are not shared, disclosed or sold outside these legal uses, and are subject to the supervision and management of the state regulatory authorities. Necessary measures should be taken to selectively prevent the use or access of personal information data to prevent illegal access to such personal information data, to ensure that personnel with access to personal information data comply with relevant legal regulations, and to ensure the security of user personal information. In addition, once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0021] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0022] FIG. 1 shows an exemplary architecture 100 to which embodiments of the target detection method or target detection apparatus of the present disclosure can be applied.

[0023] As shown in FIG. 1, the system architecture 100 can include a pinhole camera 101, a fisheye camera 102, a network 103, and a server 104. The network 103 serves as a medium to provide a communication link between the pinhole camera 101, the fisheye camera 102, and the server 104. The network 103 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0024] The pinhole camera 101 and the fisheye camera 102 can be installed on an autonomous vehicle and interact with the server 104 through the network 103 to receive or send messages, etc. The pinhole camera 101 and the fisheye camera 102 can collect images around the autonomous vehicle and send them to the server 104 for processing.

[0025] The server 104 can be an on-board server or a cloud server of the autonomous vehicle. The server 104 can analyze and process the received images, etc., and generate processing results (e.g., target detection results) for subsequent control of the autonomous vehicle.

[0026] It should be noted that the target detection method provided by the embodiments of the present disclosure is generally executed by the server 104, and accordingly, the target detection apparatus is generally arranged in the server 104.

[0027] It should be noted that the server 104 can be hardware or software. When the server 104 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 104 is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0028] It should be understood that the number of pinhole cameras, fisheye cameras, networks and servers in FIG. 1 is merely illustrative. According to the needs of implementation, there can be any number of pinhole cameras, fisheye cameras, networks and servers.

[0029] With reference to FIG. 2, a flow 200 of one embodiment of the target detection method according to the present disclosure is shown. The target detection method comprises the following steps:

[0030] In step 201, a two-dimensional image feature of a fisheye image captured by a fisheye camera is extracted.

[0031] In the present embodiment, the subject of execution of the target detection method (for example, the server 104 shown in FIG. 1) can extract the two-dimensional image feature of the fisheye image captured by the fisheye camera.

[0032] An autonomous vehicle can be equipped with multiple fisheye cameras of different orientations. A fisheye camera refers to a camera with a fisheye lens, which is a lens with extremely short focal length and extremely large field of view. A fisheye camera can be used to capture a fisheye image with a wider field of view. The fisheye image is a two-dimensional image, and the two-dimensional image feature of the fisheye image can be obtained by performing feature extraction on the fisheye image.

[0033] In some embodiments, the fisheye image is first preprocessed to obtain a preprocessed fisheye image, and then the preprocessed fisheye image is input into a neural network to obtain the two-dimensional image feature of the fisheye image.

[0034] The first preprocessing can include, but is not limited to, at least one of the following: image format conversion, image scaling, image normalization, and the like. It should be noted that the first preprocessing does not include de-distortion, thereby preserving the full FOV of the fisheye camera. In addition, considering that the effective perception distance of the fisheye camera is relatively close, in order to reduce the algorithm complexity, the fisheye image can be scaled to a smaller scale.

[0035] The neural network can be used to extract two-dimensional features of the image, such as a deformable convolutional neural network. The distribution of the convolution kernel of the deformable convolutional neural network is obtained by learning and is variable. Due to the large angle of view of the fisheye image, the distortion is large. The deformable convolutional neural network can dynamically find the features of the fisheye image.

[0036] In step 202, based on the parameters of the fisheye camera, the three-dimensional position implicit encoding of the fisheye image is calculated.

[0037] In the embodiment, the above execution subject can calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera.

[0038] In some embodiments, the parameters of the fisheye camera are first secondly preprocessed to obtain secondly preprocessed parameters; then the preprocessed fisheye image is projected from the image coordinate system to the camera coordinate system and then to the three-dimensional space based on the secondly preprocessed parameters, to obtain the three-dimensional position implicit encoding of the fisheye image. The secondly preprocessed parameters can include at least one of the secondly preprocessed intrinsic parameters and the secondly preprocessed extrinsic parameters. Based on the focal length, the principal point offset and the distortion parameters in the secondly preprocessed intrinsic parameters, the preprocessed fisheye image can be projected from the image coordinate system to the camera coordinate system, and then projected to the three-dimensional space through the secondly preprocessed extrinsic parameters, to obtain the three-dimensional position implicit encoding of the fisheye image. The calculation process is as follows:

[0039] Firstly, based on the preprocessed fisheye image and the two-dimensional image features of the fisheye image, a fisheye four-dimensional tensor F_Coords with a dimension of (H, W, D, 3) is constructed.

[0040] Wherein, H is the height of the two-dimensional image features of the fisheye image. W is the width of the two-dimensional image features of the fisheye image. D is the number of depth bins of the two-dimensional image features of the fisheye image, which is calculated by the farthest visible range of the camera and the depth resolution. For example, the farthest visible range of the camera is 100 meters, and the depth resolution is 1 meter, and the number of depth bins is equal to the quotient of the farthest visible range of the camera divided by the depth resolution, that is, 100. 3 represents 3 coordinates, respectively F_Coords x , F_Coords y , F_Coords z , corresponding to the width, height and depth of the preprocessed fisheye image. The fisheye four-dimensional tensor F_Coords can represent the three-dimensional coordinate points of the two-dimensional image features of the fisheye image in the image coordinate system.

[0041] Secondly, the fisheye four-dimensional tensor F_Coords is repeated according to the number of fisheye images to obtain a fisheye five-dimensional tensor F_Coords with a dimension of (N, H, W, D, 3).

[0042] Wherein, N is the number of fisheye images.

[0043] Third, project the fisheye five-dimensional tensor F_Coords to the camera coordinate system by the focal length fx, fy, principal point offset cx, cy and distortion parameters ds (ds0, ds1, ds2, ds3) in the second pre-processing internal parameter K F , to get the fisheye three-dimensional coordinate point F_Cs (F_Cs x , F_Cs y , F_Cs z ) in the camera coordinate system.

[0044] Wherein, the calculation process is as follows:

[0045] x_norm = (F_Coords x -cx) / fx;

[0046] y_norm = (F_Coords y -cy) / fy;

[0047] r2 = x_norm 2 +y_norm 2 ;

[0048] r4 = r2 2 ;

[0049] r6 = r2 x r4;

[0050] r8 = r4 2 ;

[0051] cr = 1.0 + ds0 x r2 + ds1 x r4 + ds2 x r6 + ds3 x r8;

[0052] F_Cs x =x_norm x cr;

[0053] F_Cs y =y_norm x cr;

[0054] F_Cs z =F_Coords z .

[0055] Wherein, x_norm, y_norm, r2, r4, r6, r8, cr are intermediate variables.

[0056] Fourth, project the fisheye three-dimensional coordinate point F_Cs in the camera coordinate system to the radar coordinate system using the second pre-processing external parameter RT F , to get the fisheye three-dimensional coordinate point L F in the radar coordinate system with dimension (N, H, W, D, 3).

[0057] wherein, L F = RT F x F Cs.

[0058] In the fifth step, the fisheye three-dimensional coordinate points L F in the radar coordinate system are processed by a convolutional neural network to obtain three-dimensional position implicit encoding of the fisheye image with a dimension of (N, H, W, C, 3).

[0059] wherein, C is the channel dimension of the encoding.

[0060] In step 203, target detection is performed based on the two-dimensional image features and the three-dimensional position implicit encoding of the fisheye image to obtain a target detection result.

[0061] In the present embodiment, the above execution subject can perform target detection based on the two-dimensional image features and the three-dimensional position implicit encoding of the fisheye image to obtain a target detection result.

[0062] In some embodiments, a three-dimensional target query vector group is first generated; then target detection is performed based on the three-dimensional target query vector group, the two-dimensional image features and the three-dimensional position implicit encoding of the fisheye image to obtain a target detection result. Wherein, according to the maximum number of targets to be identified, a corresponding number of sparse three-dimensional target query vector groups can be generated. Wherein, one vector in the three-dimensional target query vector group corresponds to a high-dimensional vector representation of the three-dimensional position of a potential target, and these vectors are all randomly initialized.

[0063] In some embodiments, the three-dimensional target query vector group, the two-dimensional image features and the three-dimensional position implicit encoding of the fisheye image are first input into a multi-layer Transformer decoder to obtain a three-dimensional target feature vector group; then the three-dimensional target feature vector group is input into a target classification network and a target regression network to obtain a target detection result. Wherein, the three-dimensional target query vector group is a query input. The two-dimensional image features and the three-dimensional position implicit encoding of the fisheye image are a queried input. The three-dimensional target feature vector corresponding to each query vector is obtained by performing cross-attention mechanism calculation between the query input and the queried input multiple times. The target detection result is a three-dimensional target detection result, including the category, center point position, length-width-height size and yaw angle of each target, etc.

[0064] The method provided by the above embodiments of the present disclosure performs three-dimensional target detection based on the input images of the fisheye cameras of the autonomous vehicle in multiple different directions. By introducing the fisheye camera distortion parameters into the projection process from two-dimensional features to three-dimensional space, three-dimensional target detection with camera distortion perception is realized. This method does not need to perform distortion removal processing on the fisheye image, so it can complete three-dimensional target detection while retaining the full FOV of the fisheye camera.

[0065] With further reference to FIG. 3, it illustrates a flow 300 of yet another embodiment of the target detection method according to the present disclosure. The target detection method comprises the following steps:

[0066] At step 301, a two-dimensional image feature of the fisheye image captured by the fisheye camera is extracted, and a two-dimensional image feature of the pinhole image captured by the pinhole camera is extracted.

[0067] In the present embodiment, the execution subject of the target detection method (e.g. the server 104 shown in FIG. 1) can extract a two-dimensional image feature of the fisheye image captured by the fisheye camera, and extract a two-dimensional image feature of the pinhole image captured by the pinhole camera.

[0068] The autonomous vehicle can be installed with fisheye cameras and pinhole cameras of different orientations. The fisheye camera refers to a camera with a fisheye lens, which is a lens with extremely short focal length and extremely large viewing angle. The fisheye camera can be used to capture fisheye images with wider field of view. The fisheye image is a two-dimensional image, and the two-dimensional image feature of the fisheye image can be obtained by feature extraction on the fisheye image. The pinhole camera is a camera that conforms to the principle of pinhole imaging. A screen and an object are shielded by a plate with a pinhole, and the inverse image of the object will be formed on the screen, which is called pinhole imaging. The pinhole image is a two-dimensional image, and the two-dimensional image feature of the pinhole image can be obtained by feature extraction on the pinhole image.

[0069] In some embodiments, the fisheye image is first preprocessed to obtain a preprocessed fisheye image, and then the preprocessed fisheye image is input into the deformable convolutional neural network to obtain the two-dimensional image feature of the fisheye image. Similarly, the pinhole image is first preprocessed to obtain a preprocessed pinhole image, and then the preprocessed pinhole image is input into the convolutional neural network to obtain the two-dimensional image feature of the pinhole image.

[0070] The first preprocessing can include, but is not limited to, at least one of the following: image format conversion, image scaling, image normalization, and the like. The third preprocessing can include, but is not limited to, at least one of the following: image format conversion, image scaling, image normalization, and de-distortion, and the like. It should be noted that the first preprocessing does not include de-distortion, so as to retain the full FOV of the fisheye camera. The third preprocessing includes de-distortion, so as to correct the distortion of the pinhole image. In addition, considering that the effective perception distance of the fisheye camera is relatively short, in order to reduce the algorithm complexity, the fisheye image can be scaled to be smaller than the pinhole image.

[0071] The convolution kernel of the convolutional neural network is preset and fixed. The distribution of the convolution kernel of the deformable convolutional neural network is obtained by learning and is variable. Since the fisheye image has a very large viewing angle, it results in a very large distortion. The deformable convolutional neural network can be used to dynamically find the features of the fisheye image.

[0072] Step 302, based on the parameters of the fisheye camera, calculate the three-dimensional position implicit encoding of the fisheye image, and based on the parameters of the pinhole camera, calculate the three-dimensional position implicit encoding of the pinhole image.

[0073] In the embodiment, the above execution subject can calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera, and calculate the three-dimensional position implicit encoding of the pinhole image based on the parameters of the pinhole camera.

[0074] In some embodiments, first, the parameters of the fisheye camera are secondly preprocessed to obtain second preprocessed parameters; then, based on the second preprocessed parameters, the preprocessed fisheye image is projected from the image coordinate system to the camera coordinate system, and then to the three-dimensional space, to obtain the three-dimensional position implicit encoding of the fisheye image. The second preprocessed parameters can include at least one of: a second preprocessed intrinsic parameter and a second preprocessed extrinsic parameter, etc. Based on the focal length, principal point offset and distortion parameters in the second preprocessed intrinsic parameter, the preprocessed fisheye image can be projected from the image coordinate system to the camera coordinate system, and then projected to the three-dimensional space through the second preprocessed extrinsic parameter, to obtain the three-dimensional position implicit encoding of the fisheye image. The calculation process is as follows:

[0075] First, based on the preprocessed fisheye image and the two-dimensional image features of the fisheye image, a fisheye four-dimensional tensor F_Coords with dimensions (H, W, D, 3) is constructed.

[0076] Wherein, H is the height of the two-dimensional image features of the fisheye image. W is the width of the two-dimensional image features of the fisheye image. D is the number of depth bins of the two-dimensional image features of the fisheye image, which is calculated by the farthest visible range of the camera and the depth resolution. For example, the farthest visible range of the camera is 100 meters, and the depth resolution is 1 meter, and the number of depth bins is equal to the quotient of the farthest visible range of the camera divided by the depth resolution, that is, 100. 3 represents 3 coordinates, respectively F_Coords x , F_Coords y , F_Coords z , corresponding to the width, height and depth of the preprocessed fisheye image. The fisheye four-dimensional tensor F_Coords can represent the three-dimensional coordinate points of the two-dimensional image features of the fisheye image in the image coordinate system.

[0077] Second, the fisheye four-dimensional tensor F_Coords is repeated according to the number of fisheye images to obtain a fisheye five-dimensional tensor F_Coords with dimensions (N, H, W, D, 3).

[0078] Wherein, N is the number of fisheye images.

[0079] Third, through the second preprocessed intrinsic parameter K Ffocal length fx, fy, principal point offset cx, cy and distortion parameters ds (ds0, ds1, ds2, ds3) in the camera coordinate system, the fisheye five-dimensional tensor F_Coords is projected into the camera coordinate system to obtain the fisheye three-dimensional coordinate point F_Cs (F_Cs x , F_Cs y , F_Cs z ) in the camera coordinate system.

[0080] Wherein, the calculation process is as follows:

[0081] x_norm = (F_Coords x -cx) / fx;

[0082] y_norm = (F_Coords y -cy) / fy;

[0083] r2 = x_norm 2 +y_norm 2 ;

[0084] r4 = r2 2 ;

[0085] r6 = r2 x r4;

[0086] r8 = r4 2 ;

[0087] cr = 1.0 + ds0 x r2 + ds1 x r4 + ds2 x r6 + ds3 x r8;

[0088] F_Cs x =x_norm x cr;

[0089] F_Cs y =y_norm x cr;

[0090] F_Cs z =F_Coords z .

[0091] Wherein, x_norm, y_norm, r2, r4, r6, r8, cr are intermediate variables.

[0092] Fourth, using the second pre-processing external parameter RT F The fisheye three-dimensional coordinate point F_Cs in the camera coordinate system is projected into the radar coordinate system to obtain the fisheye three-dimensional coordinate point L F in the radar coordinate system with dimension (N, H, W, D, 3).

[0093] Wherein, L F =RT F x F_Cs.

[0094] Fifth, the fish-eye three-dimensional coordinate points L in the radar coordinate system are processed by a convolutional neural network to obtain three-dimensional position implicit encoding of the fish-eye image with a dimension of (N, H, W, C, 3). F

[0095] Wherein, C is the channel dimension of the encoding.

[0096] In some embodiments, the fourth pre-processing is first performed on the parameters of the pinhole camera to obtain fourth pre-processing parameters; and then the pre-processed pinhole image is projected into a three-dimensional space based on the fourth pre-processing parameters to obtain three-dimensional position implicit encoding of the pinhole image. The fourth pre-processing parameters include at least one of: fourth pre-processing intrinsic parameters and fourth pre-processing extrinsic parameters. Based on the focal length and principal point offset in the fourth pre-processing intrinsic parameters and the fourth pre-processing extrinsic parameters, a coordinate projection matrix of the pre-processed pinhole image to the vehicle can be calculated, and the pre-processed pinhole image is projected into a three-dimensional space by using the coordinate projection matrix to obtain three-dimensional position implicit encoding of the pinhole image.

[0097] The calculation process is as follows:

[0098] First, based on the pinhole image and the two-dimensional image features of the pinhole image, a pinhole four-dimensional tensor P_Coords with a dimension of (H, W, D, 3) is constructed.

[0099] Wherein, H is the height of the two-dimensional image features of the pinhole image. W is the width of the two-dimensional image features of the pinhole image. D is the number of depth bins of the two-dimensional image features of the pinhole image, which is calculated by the farthest visible range of the camera and the depth resolution. For example, the farthest visible range of the camera is 100 meters, and the depth resolution is 1 meter, and the number of depth bins is equal to the quotient of the farthest visible range of the camera divided by the depth resolution, i.e. 100.3 represents 3 coordinates, P_Coords x , P_Coords y , P_Coords z , corresponding to the width, height and depth of the pre-processed pinhole image. The pinhole four-dimensional tensor P_Coords can represent the three-dimensional coordinate points of the two-dimensional image features of the pinhole image in the image coordinate system.

[0100] Second, the pinhole four-dimensional tensor P_Coords is repeated according to the number of pinhole images to obtain a pinhole five-dimensional tensor P_Coords with a dimension of (N, H, W, D, 3).

[0101] Wherein, N is the number of pinhole images.

[0102] Third, the fourth pre-processing intrinsic parameters K P and the fourth pre-processing extrinsic parameters RT P ​Projecting the five-dimensional tensor P_Coords of the pinhole onto the radar coordinate system yields the three-dimensional coordinates L of the pinhole in the radar coordinate system with dimensions (N,H,W,D,3). P .

[0103] Among them, L P =RT P ×K P ×P_Coords.

[0104] The fourth step is to use a convolutional neural network to analyze the three-dimensional coordinates of the pinhole point L in the radar coordinate system. P The processing yields the three-dimensional implicit encoding of the pinhole image with dimensions (N,H,W,C,3).

[0105] Where C represents the channel dimension of the encoding.

[0106] Step 303: Based on the two-dimensional image features and three-dimensional position implicit coding of the fisheye image, and the two-dimensional image features and three-dimensional position implicit coding of the pinhole image, target detection is performed to obtain the target detection result.

[0107] In this embodiment, the aforementioned execution entity can perform target detection based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, as well as the two-dimensional image features and three-dimensional position implicit encoding of the pinhole image, to obtain the target detection result.

[0108] In some embodiments, a three-dimensional target query vector group is first generated; then, the three-dimensional target query vector group, the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, and the two-dimensional image features and three-dimensional position implicit encoding of the pinhole image are input into the multilayer converter decoder to obtain a three-dimensional target feature vector group; finally, the three-dimensional target feature vector group is input into the target classification network and the target regression network to obtain the target detection result.

[0109] Based on a pre-defined maximum number of targets to be identified, a corresponding number of sparse 3D target query vector sets can be generated. Each vector in the 3D target query vector set corresponds to a high-dimensional vector representation of the 3D position of a potential target, and these vectors are all randomly initialized. The 3D target query vector sets are the query inputs. The 2D image features and implicit 3D position encoding of the fisheye image are the query inputs. The 3D target feature vectors corresponding to each query vector are calculated through multiple cross-attention mechanisms between the query inputs and the query inputs. The target detection result is a 3D target detection result, including the category, center point position, length, width, height, and yaw angle of each target, etc.

[0110] The method provided by the above embodiments of the present disclosure performs three-dimensional target detection based on input images of multiple small-hole cameras and fisheye cameras in different directions of an autonomous vehicle. By introducing fisheye camera distortion parameters into the projection process of two-dimensional features to three-dimensional space, three-dimensional target detection with camera distortion perception is achieved. This method does not need to perform distortion removal on fisheye images, and thus can complete three-dimensional target detection while retaining the full FOV of the fisheye camera. In addition, in order to improve the irrationality of directly extracting fisheye images with obvious distortion by a conventional convolutional neural network, independent neural networks are used for feature extraction of small-hole images and fisheye images in an autonomous vehicle. Among them, a conventional convolutional neural network is directly used for feature extraction of small-hole camera images, while a deformable convolutional neural network is used for feature extraction of fisheye camera images. The advantages of the large-angle FOV of the fisheye camera are fully utilized, the blind area of the autonomous driving perception system is greatly reduced, and the safety and intelligence of autonomous driving are improved.

[0111] For ease of understanding, FIG. 4 shows a block diagram of one application scenario of a target detection method according to an embodiment of the present disclosure.

[0112] In the first step, multiple small-hole cameras in different directions are installed on an autonomous vehicle to collect small-hole image 1, small-hole image 2, …, and small-hole image N. Similarly, multiple fisheye cameras in different directions are installed on the autonomous vehicle to collect fisheye image 1, fisheye image 2, …, and fisheye image N.

[0113] In the second step, after the small-hole image 1, small-hole image 2, …, and small-hole image N are preprocessed, they are input to a convolutional neural network for feature extraction to obtain small-hole image two-dimensional features. Similarly, after the fisheye image 1, fisheye image 2, …, and fisheye image N are preprocessed, they are input to a deformable convolutional neural network for feature extraction to obtain fisheye image two-dimensional features. In the preprocessing, only the small-hole images are subjected to distortion removal.

[0114] In the third step, after the internal and external parameters of the small-hole camera are preprocessed, they are used to encode the small-hole image two-dimensional features to obtain small-hole three-dimensional position implicit encoding. Similarly, after the internal and external parameters of the fisheye camera and the fisheye camera distortion parameters are preprocessed, they are used to encode the fisheye image two-dimensional features to obtain fisheye three-dimensional position implicit encoding.

[0115] In the fourth step, a sparse three-dimensional target query vector group is generated according to a pre-set maximum number of targets to be identified.

[0116] In the fifth step, the small-hole three-dimensional position implicit encoding and the fisheye three-dimensional position implicit encoding are input as a query, together with the sparse three-dimensional target query vector group, to a converter decoder to obtain a three-dimensional target feature vector group.

[0117] In the sixth step, the three-dimensional target feature vector group is input into the target classification network and the target regression network, and a three-dimensional target detection result is output.

[0118] To implement the method shown in the above figures, the disclosure provides an embodiment of a target detection device, which corresponds to the method embodiment shown in FIG. 2, and the device can be applied to various electronic devices.

[0119] As shown in FIG. 5, the target detection device 500 provided in this embodiment includes a first extraction unit 501, a first calculation unit 502, and a detection unit 503. The first extraction unit 501 is configured to extract a two-dimensional image feature of a fisheye image captured by a fisheye camera. The first calculation unit 502 is configured to calculate a three-dimensional position implicit encoding of the fisheye image based on parameters of the fisheye camera. The detection unit 503 is configured to perform target detection based on the two-dimensional image feature and the three-dimensional position implicit encoding of the fisheye image, and obtain a target detection result.

[0120] In this embodiment, the specific processing of the first extraction unit 501, the first calculation unit 502, and the detection unit 503 in the target detection device 500 and the technical effects brought by the specific processing can be respectively referred to the related descriptions of the steps 201, 202, and 203 in the corresponding embodiment of FIG. 2, which will not be repeated here.

[0121] In some optional implementations of this embodiment, the first extraction unit 501 is further configured to: perform first preprocessing on the fisheye image to obtain a preprocessed fisheye image, wherein the first preprocessing includes at least one of the following: image format conversion, image scaling, and image normalization; and input the preprocessed fisheye image into a neural network to obtain the two-dimensional image feature of the fisheye image.

[0122] In some optional implementations of this embodiment, the neural network is a deformable convolutional neural network.

[0123] In some optional implementations of this embodiment, the first calculation unit 502 includes: a second preprocessing subunit configured to perform second preprocessing on the parameters of the fisheye camera to obtain second preprocessed parameters; and a first projection subunit configured to project the preprocessed fisheye image from an image coordinate system to a camera coordinate system, and then to a three-dimensional space based on the second preprocessed parameters, to obtain the three-dimensional position implicit encoding of the fisheye image.

[0124] In some optional implementations of this embodiment, the second preprocessed parameters include at least one of the following: second preprocessed intrinsic parameters and second preprocessed extrinsic parameters.

[0125] In some optional implementations of the present embodiment, the first projection subunit comprises: a first projection module configured to project the pre-processed fisheye image from the image coordinate system to the camera coordinate system based on the focal length, the principal point offset and the distortion parameters in the second pre-processing intrinsic parameter, and then project to the three-dimensional space through the second pre-processing extrinsic parameter to obtain the three-dimensional position implicit encoding of the fisheye image. In some optional implementations of the present embodiment, the first projection module is further configured to: construct a fisheye four-dimensional tensor based on the pre-processed fisheye image and the two-dimensional image features of the fisheye image, wherein the fisheye four-dimensional tensor represents the three-dimensional coordinate points of the two-dimensional image features of the fisheye image in the image coordinate system; repeat the fisheye four-dimensional tensor according to the number of fisheye images to obtain a fisheye five-dimensional tensor; project the fisheye five-dimensional tensor to the camera coordinate system through the focal length, the principal point offset and the distortion parameters in the second pre-processing intrinsic parameter to obtain the fisheye three-dimensional coordinate points in the camera coordinate system; project the fisheye three-dimensional coordinate points in the camera coordinate system to the radar coordinate system using the second pre-processing extrinsic parameter to obtain the fisheye three-dimensional coordinate points in the radar coordinate system; and process the fisheye three-dimensional coordinate points in the radar coordinate system through the convolutional neural network to obtain the three-dimensional position implicit encoding of the fisheye image.

[0126] In some optional implementations of the present embodiment, the detection unit 503 comprises: a generation subunit configured to generate a three-dimensional target query vector group, wherein one vector in the three-dimensional target query vector group corresponds to one target; and a detection subunit configured to perform target detection based on the three-dimensional target query vector group, the two-dimensional image features of the fisheye image and the three-dimensional position implicit encoding to obtain a target detection result. In some optional implementations of the present embodiment, the detection subunit is further configured to: input the three-dimensional target query vector group, the two-dimensional image features of the fisheye image and the three-dimensional position implicit encoding into a multi-layer converter decoder to obtain a three-dimensional target feature vector group; and input the three-dimensional target feature vector group into a target classification network and a target regression network to obtain the target detection result. In some optional implementations of the present embodiment, the target detection apparatus 500 further comprises: a second extraction unit configured to extract two-dimensional image features of a pinhole image captured by a pinhole camera; a second calculation unit configured to calculate three-dimensional position implicit encoding of the pinhole image based on parameters of the pinhole camera; and the detection unit 503 is further configured to perform target detection based on the two-dimensional image features of the fisheye image and the three-dimensional position implicit encoding, and the two-dimensional image features of the pinhole image and the three-dimensional position implicit encoding to obtain the target detection result.

[0127] In some optional implementations of the present embodiment, the detection unit 503 is further configured to input the set of three-dimensional target query vectors, the two-dimensional image features and the three-dimensional position implicit encodings of the fisheye image, and the two-dimensional image features and the three-dimensional position implicit encodings of the pinhole image into a multi-layer converter decoder to obtain a set of three-dimensional target feature vectors; and input the set of three-dimensional target feature vectors into a target classification network and a target regression network to obtain a target detection result.

[0128] In some optional implementations of the present embodiment, the second extraction unit is further configured to perform third preprocessing on the pinhole image to obtain a preprocessed pinhole image, wherein the third preprocessing includes at least one of image format conversion, image scaling, image normalization, and de-distortion; and input the preprocessed pinhole image into a convolutional neural network to obtain the two-dimensional image features of the pinhole image.

[0129] In some optional implementations of the present embodiment, the second calculation unit includes: a second preprocessing subunit configured to perform fourth preprocessing on parameters of the pinhole camera to obtain second preprocessed parameters, wherein the second preprocessed parameters include fourth preprocessed intrinsic parameters and fourth preprocessed extrinsic parameters; and a second projection subunit configured to calculate a coordinate projection matrix of the preprocessed pinhole image to the vehicle based on a focal length and a principal point offset in the fourth preprocessed intrinsic parameters and the fourth preprocessed extrinsic parameters, and project the preprocessed pinhole image to a three-dimensional space by using the coordinate projection matrix to obtain the three-dimensional position implicit encoding of the pinhole image.

[0130] In some optional implementations of the present embodiment, the second projection subunit is further configured to: construct a pinhole four-dimensional tensor based on the two-dimensional image features of the preprocessed pinhole image and the pinhole image, wherein the pinhole four-dimensional tensor represents three-dimensional coordinate points of the two-dimensional image features of the pinhole image in an image coordinate system; repeat the pinhole four-dimensional tensor according to the number of pinhole images to obtain a pinhole five-dimensional tensor; project the pinhole five-dimensional tensor into a radar coordinate system by using the fourth preprocessed intrinsic parameters and the fourth preprocessed extrinsic parameters to obtain pinhole three-dimensional coordinate points in the radar coordinate system; and process the pinhole three-dimensional coordinate points in the radar coordinate system by using the convolutional neural network to obtain the three-dimensional position implicit encoding of the pinhole image.

[0131] The device provided by the above embodiments of the present disclosure performs three-dimensional target detection based on input images of fisheye cameras in multiple different orientations of an autonomous vehicle. By introducing fisheye camera distortion parameters into the projection process from two-dimensional features to three-dimensional space, three-dimensional target detection with camera distortion perception is achieved. The device does not need to perform de-distortion processing on fisheye images, and thus can complete three-dimensional target detection while retaining the full FOV of the fisheye camera.

[0132] Reference is now made to FIG. 6, which shows a structural diagram of an electronic device 600 (e.g., a server in FIG. 1) suitable for implementing embodiments of the present disclosure. The terminal device in embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (e.g., a car navigation terminal), and the like, as well as a stationary terminal such as a digital TV, a desktop computer, and the like. The server shown in FIG. 6 is merely an example and should not impose any limitation on the functions and use range of embodiments of the present disclosure.

[0133] As shown in FIG. 6, the electronic device 600 can include a processing device (e.g., a central processor, a graphic processor, etc.) 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0134] In general, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; a storage device 608 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 609. The communication device 609 can allow the electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 6 shows the electronic device 600 having various devices, it should be understood that all of the illustrated devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed. Each block shown in FIG. 6 can represent one device or, as necessary, multiple devices.

[0135] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0136] It should be noted that the computer readable medium in the embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0137] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: extract a two-dimensional image feature of a fisheye image captured by a fisheye camera; calculate an implicit encoding of a three-dimensional position of the fisheye image based on parameters of the fisheye camera; perform target detection based on the two-dimensional image feature and the implicit encoding of the three-dimensional position of the fisheye image to obtain a target detection result.

[0138] Computer program code for carrying out operations of embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0139] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0140] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. The described units can also be implemented in a processor, for example, can be described as: a processor includes a first extraction unit, a first calculation unit, a generation unit and a detection unit. Among them, the name of these units does not constitute a limitation to the units themselves in some cases, for example, the first extraction unit can also be described as: a unit for extracting two-dimensional image features of the fisheye image photographed by the fisheye camera.

[0141] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the inventive scope of the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the above inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) form the technical solutions.

Claims

1. A target detection method, comprising: Extract two-dimensional image features from fisheye images captured by a fisheye camera; Based on the parameters of the fisheye camera, calculate the three-dimensional position implicit encoding of the fisheye image; Target detection is performed based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image to obtain the target detection result.

2. The method according to claim 1, wherein, The extraction of two-dimensional image features from fisheye images captured by a fisheye camera includes: The fisheye image is subjected to a first preprocessing to obtain a preprocessed fisheye image, wherein the first preprocessing includes at least one of the following: image format conversion, image scaling, and image normalization; The preprocessed fisheye image is input into a neural network to obtain the two-dimensional image features of the fisheye image.

3. The method according to claim 2, wherein, The neural network is a deformable convolutional neural network.

4. The method according to claim 1, wherein, The step of calculating the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera includes: The parameters of the fisheye camera are subjected to a second preprocessing to obtain second preprocessed parameters; Based on the second preprocessing parameters, the preprocessed fisheye image is projected from the image coordinate system to the camera coordinate system, and then projected into three-dimensional space to obtain the three-dimensional position implicit encoding of the fisheye image.

5. The method according to claim 4, wherein, The second preprocessing parameter includes at least one of the following: a second preprocessing intrinsic parameter and a second preprocessing extrinsic parameter.

6. The method according to claim 5, wherein, The step of projecting the preprocessed fisheye image from the image coordinate system to the camera coordinate system, and then projecting it into three-dimensional space based on the second preprocessing parameters to obtain the three-dimensional position implicit encoding of the fisheye image includes: Based on the focal length, principal point offset, and distortion parameters in the second preprocessing intrinsic parameters, the preprocessed fisheye image is projected from the image coordinate system to the camera coordinate system, and then projected into three-dimensional space through the second preprocessing extrinsic parameters to obtain the three-dimensional position implicit encoding of the fisheye image.

7. The method according to claim 6, wherein, The process of projecting the fisheye image from the image coordinate system to the camera coordinate system based on the focal length, principal point offset, and distortion parameters in the second preprocessing intrinsic parameters, and then projecting it into three-dimensional space through the second preprocessing extrinsic parameters, to obtain the implicit three-dimensional position encoding of the fisheye image, includes: Based on the preprocessed fisheye image and the two-dimensional image features of the fisheye image, a four-dimensional fisheye tensor is constructed, wherein the four-dimensional fisheye tensor represents the three-dimensional coordinate points of the two-dimensional image features of the fisheye image in the image coordinate system. The fisheye four-dimensional tensor is repeated according to the number of fisheye images to obtain the fisheye five-dimensional tensor; By using the focal length, principal point offset, and distortion parameters in the second preprocessing intrinsic parameters, the five-dimensional tensor of the fisheye is projected onto the camera coordinate system to obtain the three-dimensional coordinate points of the fisheye in the camera coordinate system. The fisheye 3D coordinate points in the camera coordinate system are projected onto the radar coordinate system using the second preprocessing extrinsic parameters to obtain the fisheye 3D coordinate points in the radar coordinate system. The three-dimensional coordinate points of the fisheye image in the radar coordinate system are processed by a convolutional neural network to obtain the implicit three-dimensional position encoding of the fisheye image.

8. The method according to claim 1, wherein, The target detection based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, to obtain the target detection result, includes: Generate a three-dimensional target query vector group, wherein one vector in the three-dimensional target query vector group corresponds to one target; Target detection is performed based on the three-dimensional target query vector group, the two-dimensional image features of the fisheye image, and the three-dimensional position implicit encoding to obtain the target detection result.

9. The method according to claim 8, wherein, The target detection based on the three-dimensional target query vector group, the two-dimensional image features of the fisheye image, and the three-dimensional position implicit encoding yields the target detection result, including: The three-dimensional target query vector group, the two-dimensional image features of the fisheye image, and the three-dimensional position implicit encoding are input into the multilayer converter decoder to obtain the three-dimensional target feature vector group; The three-dimensional target feature vector group is input into the target classification network and the target regression network to obtain the target detection result.

10. The method according to any one of claims 1-9, wherein, The method further includes: Extract two-dimensional image features from pinhole images captured by a pinhole camera; Based on the parameters of the pinhole camera, calculate the three-dimensional position implicit encoding of the pinhole image; and The target detection based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, to obtain the target detection result, includes: Based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, and the two-dimensional image features and three-dimensional position implicit encoding of the pinhole image, target detection is performed to obtain the target detection result.

11. A target detection device, comprising: The first extraction unit is configured to extract two-dimensional image features from fisheye images captured by a fisheye camera; The first computing unit is configured to calculate the three-dimensional position implicit encoding of the fisheye image based on the parameters of the fisheye camera. The detection unit is configured to perform target detection based on the two-dimensional image features and three-dimensional position implicit encoding of the fisheye image, and obtain the target detection result.

12. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images