Bird-eye view-based target detection method and related device

Abstract

The invention discloses a bird's-eye view-based target detection method and a related device, and the method comprises the steps: obtaining original image data collected by a roadside camera, carrying out the feature extraction of a current frame image, obtaining the features of a current image, and carrying out the recognition of the current image based on a mapping table of the image features and a three-dimensional voxel space; mapping the current image feature to a three-dimensional voxel space to obtain a three-dimensional voxel feature, processing the three-dimensional voxel feature to obtain a current aerial view feature of the current frame image, and performing time sequence alignment and weighted fusion processing on the current aerial view feature and an aerial view feature of a historical frame image to obtain an aerial view fusion feature; and inputting the aerial view fusion features to a detection head for identification processing to obtain a target detection result. According to the method, the detection technology based on the roadside aerial view is adopted, and the three-dimensional position information of vehicles, pedestrians and other traffic participants can be accurately captured by performing two-dimensional to three-dimensional conversion on the acquired image features, so that the complete information of the detection target in the whole road scene is obtained, and the accuracy of the detection result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and more particularly to a target detection method based on a bird's-eye view and a related device. Background Art

[0002] Roadside cameras are cameras installed on roadside poles with fixed positions and states. They have various functions and are widely used in traffic management and security monitoring. At present, roadside cameras can provide a continuous bird's-eye view to achieve real-time monitoring of the overall road conditions.

[0003] However, roadside cameras are usually installed in fixed positions with limited viewing angles and field of view, making it difficult to capture complete information of the entire road scene from a single perspective. As a result, the detection targets (such as vehicles, pedestrians, etc.) may not be fully presented in the image due to occlusion or truncation, making the position and shape information of the detection targets in the bird's-eye view inaccurate, affecting the accuracy of the detection results. Summary of the invention

[0004] In view of this, the present invention discloses a target detection method based on a bird's-eye view and related devices, which adopt a detection technology based on a roadside bird's-eye view. By transforming the collected image features from two dimensions to three dimensions, the three-dimensional position information of vehicles, pedestrians and other traffic participants can be accurately captured, thereby obtaining complete information of the detection target in the entire road scene and improving the accuracy of the detection result.

[0005] A target detection method based on a bird's-eye view, comprising:

[0006] Obtaining raw image data collected by roadside cameras;

[0007] Extracting features of a current frame image in the original image data to obtain current image features;

[0008] Based on a pre-established mapping table between image features and three-dimensional voxel space, mapping the current image features to the three-dimensional voxel space to obtain three-dimensional voxel features;

[0009] Processing the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image;

[0010] Performing temporal alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain bird's-eye view fusion features;

[0011] The bird's-eye view fusion features are input into the detection head for recognition processing to obtain the target detection result.

[0012] Optionally, extracting features of the current frame image in the original image data to obtain current image features includes:

[0013] Extracting multi-scale features from the current frame image using a backbone network;

[0014] The multi-scale features are fused to obtain the current image features.

[0015] Optionally, mapping the current image feature to the three-dimensional voxel space based on a pre-established mapping table between the image feature and the three-dimensional voxel space to obtain the three-dimensional voxel feature includes:

[0016] In the process of mapping the current image feature to the three-dimensional pixel space, a nearest voxel sorting method is used to determine one or more voxels corresponding to each pixel corresponding to the current image feature in the three-dimensional pixel space.

[0017] Optionally, the process of establishing a mapping table between image features and three-dimensional voxel space includes:

[0018] Obtaining an intrinsic parameter matrix and an extrinsic parameter matrix of the roadside camera;

[0019] The three-dimensional voxel position of each voxel is converted into a camera coordinate system by using the external parameter matrix to obtain the position of each voxel in the camera coordinate system, wherein each voxel is: each small cubic unit obtained by dividing the continuous three-dimensional space where the area to be detected is located into a regular grid;

[0020] Projecting the position of each voxel in the camera coordinate system onto a two-dimensional image plane to obtain corresponding image pixel coordinates;

[0021] Each image pixel coordinate is associated with the corresponding three-dimensional voxel position to establish a mapping table between image features and three-dimensional voxel space.

[0022] Optionally, the processing the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image includes:

[0023] compressing the three-dimensional voxel features into bird's-eye view features;

[0024] A weighted fusion multi-scale feature method is used on the bird's-eye view feature to obtain an initial bird's-eye view feature;

[0025] The initial bird's-eye view feature is encoded using a bird's-eye view encoder to obtain the current bird's-eye view feature.

[0026] Optionally, compressing the three-dimensional voxel features into bird's-eye view features comprises:

[0027] The depth dimension of the three-dimensional voxel feature is compressed into a two-dimensional plane using a compression function to obtain the bird's-eye view feature.

[0028] A target detection device based on a bird's-eye view, comprising:

[0029] An acquisition unit, used to acquire raw image data collected by a roadside camera;

[0030] An extraction unit, used for extracting features of a current frame image in the original image data to obtain current image features;

[0031] A mapping unit, configured to map the current image feature to the three-dimensional voxel space based on a pre-established mapping table between the image feature and the three-dimensional voxel space, so as to obtain the three-dimensional voxel feature;

[0032] A feature processing unit, used for processing the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image;

[0033] A fusion unit, used for performing temporal alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain bird's-eye view fusion features;

[0034] The recognition unit is used to input the bird's-eye view fusion features into the detection head for recognition processing to obtain the target detection result.

[0035] A computer program product includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the above-mentioned target detection method based on a bird's-eye view.

[0036] An electronic device comprises at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned target detection method based on the bird's-eye view.

[0039] A computer storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the above-mentioned bird's-eye view-based target detection method.

[0040] It can be seen from the above technical solutions that the present invention discloses a target detection method based on a bird's-eye view and related devices, which obtains the original image data collected by the roadside camera, extracts features of the current frame image in the original image data to obtain the current image features, maps the current image features to the three-dimensional voxel space based on a pre-established mapping table of image features and three-dimensional voxel space to obtain three-dimensional voxel features, processes the three-dimensional voxel features to obtain the current bird's-eye view features of the current frame image, performs temporal alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain the bird's-eye view fusion features, and obtains the target detection results by inputting the bird's-eye view fusion features into the detection head for recognition processing. The present application adopts a detection technology based on a roadside bird's-eye view, and by performing a two-dimensional to three-dimensional transformation on the collected image features, the three-dimensional position information of vehicles, pedestrians and other traffic participants can be accurately captured, thereby obtaining complete information of the detection target in the entire road scene and improving the accuracy of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0042] Figure 1 A flow chart of a target detection method based on a bird's-eye view disclosed in an embodiment of the present invention;

[0043] Figure 2 A flow chart of a method for extracting features of a current frame image in original image data to obtain features of the current image disclosed in an embodiment of the present invention;

[0044] Figure 3 A flow chart of a method for establishing a mapping table between image features and three-dimensional voxel space disclosed in an embodiment of the present invention;

[0045] Figure 4 A flow chart of a method for processing three-dimensional voxel features to obtain current bird's-eye view features of a current frame image disclosed in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a target detection device based on a bird's-eye view disclosed in an embodiment of the present invention;

[0047] Figure 6 The present invention is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The embodiment of the present invention discloses a target detection method based on a bird's-eye view and a related device, which obtains the original image data collected by a roadside camera, extracts features of the current frame image in the original image data to obtain the current image features, maps the current image features to the three-dimensional voxel space based on a pre-established mapping table of image features and three-dimensional voxel space to obtain three-dimensional voxel features, processes the three-dimensional voxel features to obtain the current bird's-eye view features of the current frame image, performs time-series alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain the bird's-eye view fusion features, and obtains the target detection results by inputting the bird's-eye view fusion features into the detection head for recognition processing. The present application adopts a detection technology based on a roadside bird's-eye view, and by performing a two-dimensional to three-dimensional transformation on the collected image features, the three-dimensional position information of vehicles, pedestrians and other traffic participants can be accurately captured, thereby obtaining complete information of the detection target in the entire road scene and improving the accuracy of the detection results.

[0050] See also Figure 1 , a flow chart of a target detection method based on a bird's-eye view disclosed in an embodiment of the present invention, the method comprising:

[0051] Step S101: Acquire original image data collected by a roadside camera.

[0052] In practical applications, raw image data collected by a single road test camera or multiple road test cameras may be obtained.

[0053] Step S102: extract features of the current frame image in the original image data to obtain current image features.

[0054] The original image data contains many frame images, and the application defines the image currently being processed as the current frame image. The image before the current frame image, such as the previous frame image of the current frame image, the previous frame image of the previous frame image, etc., can be defined as a historical frame image.

[0055] This application uses the same image processing method for each frame of the original image data.

[0056] Step S103: Based on a pre-established mapping table between image features and three-dimensional voxel space, the current image features are mapped to the three-dimensional voxel space to obtain three-dimensional voxel features.

[0057] The present application pre-establishes a mapping table between image features and three-dimensional voxel space, which effectively avoids the problem of inaccuracy in detecting depth from the image prediction perspective.

[0058] The process of mapping to obtain three-dimensional voxel features can be seen in the following expression:

[0059] ;

[0060] In the formula, represents the 3D voxel features, Represents the current image features, Represents the reverse lookup table of T(x,y,z), which is a mapping table between image features and 3D voxel space. img represents the image identifier. When there are multiple roadside cameras working simultaneously in the system, each roadside camera captures images from different perspectives. img is used to distinguish different roadside cameras. For example, img=1 represents the image of the first camera, img represents the image of the second camera, and so on.

[0061] T (x, y, z) is a mapping table pre-calculated based on the intrinsic parameter matrix and extrinsic parameter matrix of the roadside camera. The mapping table records the mapping relationship between image coordinates and voxel space coordinates, that is, the mapping relationship between image features and three-dimensional voxel space.

[0062] T′(x,y) is the reverse lookup table of T(x,y,z), containing the set of all voxels that map to pixel (x,y).

[0063] Three-dimensional voxel features refer to dividing an object or scene into a series of small cubes (voxels) in three-dimensional space, and extracting the intersection or feature information of these voxels with the surface of the object or other key points. They can be used to describe the shape, structure, surface features of the object, and its spatial relationship with other objects.

[0064] Voxel is the most basic unit of 3D volume data, usually a small cube. In 3D space, an object can be viewed as composed of many voxels.

[0065] It should be noted that a voxel in three-dimensional space can only correspond to one pixel in two-dimensional space, but a pixel in two-dimensional space may correspond to multiple voxels on the same line bundle.

[0066] In the process of mapping image features to three-dimensional voxel space, a pixel (x, y) on an image may correspond to multiple voxels. Since images do not have depth information, voxels at different depths on the same line of sight are mapped to the same pixel position. This means that for a certain pixel (x, y), there may be multiple voxels , satisfying the following projection relationship:

[0067] ;

[0068] Refers to Look up the voxel corresponding to the pixel (x, y) on the image marked with img.

[0069] Indicates the voxel found. In practical applications, multiple voxels may be found.

[0070] In the process of mapping the current image feature to the three-dimensional pixel space, the present application adopts a nearest voxel sorting method to determine one or more voxels corresponding to each pixel corresponding to the current image feature in the three-dimensional pixel space.

[0071] This application uses the nearest voxel sorting method to determine the voxel corresponding to the pixel, and the expression is as follows:

[0072] ;

[0073] In the formula, represents the most determined voxel.

[0074] Step S104: Process the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image.

[0075] The process of processing the three-dimensional voxel features includes but is not limited to compression, fusion and encoding.

[0076] The current bird's-eye view features may include: global view, vertical projection, proportional relationship, etc.

[0077] Step S105 , performing temporal alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain bird's-eye view fusion features.

[0078] The current bird's-eye view feature of the current frame image is referred to as the "current frame feature". express.

[0079] In practical applications, the current frame image may include n historical frame images before it. The bird's-eye view features of the historical frame images are referred to as "historical frame features". express.

[0080] Combine the current frame features with the n historical frame features Perform timing alignment and fusion.

[0081] The timing alignment and fusion process is as follows:

[0082] The historical frame features are transformed through camera external parameters Convert to the coordinate system of the current frame t, the expression is as follows:

[0083] ;

[0084] In the formula, It represents the feature representation of the historical frame k aligned to the current frame t, and E represents the external parameter matrix of the roadside camera.

[0085] Each aligned history frame With the current frame features In each frame fusion, the attenuation factor Controls the weight of history frames.

[0086] make is the weighted coefficient of the kth frame, and the generation formula is as follows:

[0087] ;

[0088] Where k=1,2,…,n is the historical frame number, is the attenuation factor, which controls the weight attenuation of historical frame features during the fusion process.

[0089] Using individual history frames The weighted sum of is used to fuse the features, so as to obtain the fused bird's-eye view fusion features, which are expressed as follows:

[0090] ;

[0091] In the formula, represents the fusion features of the bird's-eye view, represents the weight of the current frame, , ensuring that the current frame feature has the highest weight in the fusion process, Indicates the current frame features.

[0092] Step S106: input the bird's-eye view fusion features into the detection head for recognition processing to obtain target detection results.

[0093] The detection head usually refers to a neural network module for target detection tasks. This neural network module receives the processed feature map (bird's eye view fusion feature in this application) as input and outputs the location, category and other relevant information about the target object.

[0094] The expression of the target detection result is as follows:

[0095] ;

[0096] In the formula, D represents the target detection result, F represents the bird's-eye view fusion feature, Indicates the detection head.

[0097] In summary, the present invention discloses a target detection method based on a bird's-eye view, which obtains the original image data collected by the roadside camera, extracts features of the current frame image in the original image data to obtain the current image features, maps the current image features to the three-dimensional voxel space based on a pre-established mapping table of image features and three-dimensional voxel space to obtain three-dimensional voxel features, processes the three-dimensional voxel features to obtain the current bird's-eye view features of the current frame image, performs time-series alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain the bird's-eye view fusion features, and obtains the target detection results by inputting the bird's-eye view fusion features into the detection head for recognition processing. The present application adopts a detection technology based on a roadside bird's-eye view, and by performing a two-dimensional to three-dimensional transformation on the collected image features, the three-dimensional position information of vehicles, pedestrians and other traffic participants can be accurately captured, thereby obtaining complete information of the detection target in the entire road scene and improving the accuracy of the detection results.

[0098] In addition, this application adopts a pure convolution method, which avoids the problem that the Transformer architecture in the traditional solution is unfriendly to edge device deployment. The network structure is simple, the edge device inference speed is fast, and the network is easy to expand.

[0099] In one embodiment, see Figure 2 , a flowchart of a method for extracting features of a current frame image in raw image data to obtain features of the current image is disclosed in an embodiment of the present invention, that is, step S102 may specifically include:

[0100] Step S201: extract multi-scale features from the current frame image using the backbone network.

[0101] The backbone network is the foundation of the object detection architecture. Its main purpose is to extract features from the input image. These networks are usually pre-trained deep convolutional neural networks that can capture basic to complex image features. The output of the backbone network is a set of multi-scale features that contain key visual information in the image.

[0102] Preferably, the backbone network in the present application can be ResNet-50. As a deep residual network, ResNet-50 has powerful feature extraction capabilities. Its network structure contains multiple residual blocks, each of which extracts features from the input image through a convolutional layer and passes the input directly to the output through a residual connection, thereby retaining more image information. This design enables ResNet-50 to extract richer image features, which is crucial for computer vision tasks such as image classification and target detection.

[0103] Specifically, the size of the current frame image I input to the backbone network ResNet-50 is C×H×W, where C is the number of channels, the number of RGB (Red Green Blue) image channels is 3, H is the image height, W is the image width, and the resolution is 1080×1920.

[0104] The backbone network ResNet-50 extracts multi-scale features from the current frame image It can be expressed as:

[0105] ;

[0106] The multi-scale features extracted from the current frame image in this application include 4 layers of features, namely: .

[0107] Step S202: Fusing the multi-scale features to obtain the current image features.

[0108] Preferably, this application uses the pyramid network SECONDFPN to fuse multi-scale features to obtain the current image features. The fusion formula is expressed as follows:

[0109] ;

[0110] F FPN is the current image feature.

[0111] In one embodiment, see Figure 3 , a flowchart of a method for establishing a mapping table between image features and three-dimensional voxel space disclosed in an embodiment of the present invention, the method may include:

[0112] Step S301: Obtain an intrinsic parameter matrix and an extrinsic parameter matrix of the roadside camera.

[0113] The external parameter matrix is ​​used to describe the rigid body transformation between the voxel coordinate system and the camera coordinate system, and the internal parameter matrix is ​​used to correct the image distortion caused by the lens.

[0114] The two-dimensional pixel coordinates on the image are Indicates that x represents the horizontal coordinate of the pixel in the image, and y represents the vertical coordinate of the pixel in the image.

[0115] The three-dimensional voxel coordinates are expressed in homogeneous coordinates as: , represents the coordinate of the voxel in the X direction, represents the coordinate of the voxel in the Y direction, Represents the coordinate of the voxel in the Z direction. 1 is a placeholder to make up the homogeneous coordinates. The advantage of homogeneous coordinates is that they can be used to uniformly represent rigid body transformations such as rotation and translation by matrix multiplication.

[0116] The expression of the external parameter matrix E is as follows:

[0117] ;

[0118] Among them, in the external parameter matrix E, represents the rotation matrix, Represents the translation vector.

[0119] In the formula, r 11 Indicates that the X axis of the voxel coordinate system is in the camera coordinate system X C The projection component in the direction defines how the X axis is rotated to the camera's X C direction.

[0120] r 12 Indicates that the X axis of the voxel coordinate system is in the Y axis of the camera coordinate system C The projection component in the direction, which defines how the X axis is rotated to the Y axis of the camera C direction.

[0121] r 13 Indicates that the X axis of the voxel coordinate system is in the Z axis of the camera coordinate system C The projection component in the direction, defining how the X axis is rotated to the Z axis of the camera C direction.

[0122] r 21 Indicates that the Y axis of the voxel coordinate system is in the X axis of the camera coordinate system C The projection component in the direction, which defines how the Y axis is rotated to the camera's X C direction.

[0123] r 22 Indicates that the Y axis of the voxel coordinate system is in the camera coordinate system Y C The projection component in the direction defines how the Y axis is rotated to the camera's Y C direction.

[0124] r 23 Indicates that the Y axis of the voxel coordinate system is in the Z axis of the camera coordinate system C The projection component in the direction, which defines how the Y axis is rotated to the camera's Z C direction.

[0125] r 31 Indicates that the Z axis of the voxel coordinate system is in the X axis of the camera coordinate system C The projection component in the direction, which defines how the Z axis is rotated to the camera's X C direction.

[0126] r 32 Indicates that the X axis of the voxel coordinate system is in the Y axis of the camera coordinate system CThe projection component in the direction, which defines how the Z axis is rotated to the Y axis of the camera C direction.

[0127] r 33 The X axis of the voxel coordinate system is in the Z axis of the camera coordinate system. C The projection component in the direction, which defines how the Z axis is rotated to the camera's Z C direction.

[0128] t X It represents the translation of the roadside camera along the X-axis in the voxel coordinate system, and defines the position offset of the roadside camera relative to the origin of the voxel coordinate system in the X direction.

[0129] t y It represents the translation of the roadside camera along the Y axis in the voxel coordinate system, and defines the position offset of the roadside camera in the Y direction relative to the origin of the voxel coordinate system.

[0130] t z It represents the translation of the roadside camera along the Z axis in the voxel coordinate system, and defines the position offset of the roadside camera in the Z direction relative to the origin of the voxel coordinate system.

[0131] The expression of the internal parameter matrix K is as follows:

[0132] ;

[0133] In the formula, f x is the focal length along the x direction on the image plane, f y is the focal length in the y direction on the image plane, which can be calculated from the physical focal length in pixels and the sensor pixel size. In pixels, are the coordinates of the principal point (i.e. the center of camera distortion) in the image plane.

[0134] Step S302: Use the extrinsic parameter matrix to transform the three-dimensional voxel position of each voxel into a camera coordinate system to obtain the position of each voxel in the camera coordinate system.

[0135] Each of the voxels is a small cubic unit obtained by dividing the continuous three-dimensional space where the area to be detected is located into a regular grid.

[0136] Define voxel space and discretize it: Divide the continuous three-dimensional space into regular grids, and each grid corresponds to a small cubic unit called a voxel. According to the defined size and range, the entire detection area is divided into multiple voxels to generate a three-dimensional voxel grid.

[0137] Specifically, the expression for mapping a point in the voxel coordinate system (i.e., the three-dimensional voxel position of the voxel) to the camera coordinate system using the external parameter matrix E is as follows:

[0138] ;

[0139] In the formula, is the position coordinate of each voxel in the camera coordinate system. The two coordinate systems have the same size and unit length, but different origin positions and rotation angles, which is caused by the installation position of the roadside camera.

[0140] Step S303: Project the position of each voxel in the camera coordinate system onto a two-dimensional image plane to obtain corresponding image pixel coordinates.

[0141] Specifically, the calculation method of the image pixel coordinates corresponding to each voxel is as follows:

[0142] ;

[0143] Solving for this yields:

[0144] ;

[0145] ;

[0146] ;

[0147] The most obtained are the image pixel coordinates.

[0148] Indicates depth, only when and Only when the voxel is within the image boundary, it is mapped and recorded in the mapping table T between image features and three-dimensional voxel space.

[0149] Step S304: Associating each of the image pixel coordinates with the corresponding three-dimensional voxel position to establish a mapping table between image features and three-dimensional voxel space.

[0150] The voxels in the entire voxel space are calculated and encapsulated into the mapping table T, thus obtaining:

[0151] ;

[0152] Where img is the image identifier. When there are multiple cameras working simultaneously in the system, each camera captures images from different perspectives. img is used to distinguish different cameras. img=1 represents the image of the first camera, img represents the image of the second camera, and so on.

[0153] Indicates that the coordinates in voxel space are located at A voxel of is a table that iterates over each voxel.

[0154] In one embodiment, see Figure 4 , a flowchart of a method for processing three-dimensional voxel features to obtain current bird's-eye view features of a current frame image disclosed in an embodiment of the present invention, that is, step S104 may include:

[0155] Step S401: compress the three-dimensional voxel features into bird's-eye view features.

[0156] Specifically, the 3D voxel features Projected to the bird's-eye view, using the bird's-eye view compression function The depth dimension of the 3D voxel feature Compress to a two-dimensional plane to obtain the bird's-eye view features , the expression is as follows:

[0157] .

[0158] In order to reduce the amount of calculation, compression adopts the summation method, and the expression is as follows:

[0159] ;

[0160] In the formula, Representing voxel space Bird's-eye view features.

[0161] Step S402: A weighted fusion multi-scale feature method is used to obtain initial bird's-eye view features for the bird's-eye view features.

[0162] The bird's-eye view features in this application include scale features at different levels, namely: , this application applies different levels of scale features Through weighted fusion, the initial bird's-eye view features are obtained .

[0163] Step S403: Encode the initial bird's-eye view features using a bird's-eye view encoder to obtain current bird's-eye view features.

[0164] The bird's eye view encoder in this application uses a residual structure and hybrid partial convolution to enhance feature extraction capabilities.

[0165] The encoding process can be seen in the following expression:

[0166] ;

[0167] In the formula, Indicates the current bird's-eye view feature, represents the initial bird's-eye view features, It indicates that the convolution and normalization method is adopted, which only includes a single two-dimensional convolution layer and a normalization layer.

[0168] Corresponding to the above method embodiment, the present invention also discloses a target detection device based on a bird's-eye view.

[0169] See also Figure 5 , a schematic diagram of a target detection device based on a bird's-eye view disclosed in an embodiment of the present invention, the device may include:

[0170] An acquisition unit 501 is used to acquire raw image data collected by a roadside camera;

[0171] In practical applications, raw image data collected by a single road test camera or multiple road test cameras may be obtained.

[0172] An extraction unit 502 is used to extract features of a current frame image in the original image data to obtain current image features;

[0173] The original image data contains many frame images, and the application defines the image currently being processed as the current frame image. The image before the current frame image, such as the previous frame image of the current frame image, the previous frame image of the previous frame image, etc., can be defined as a historical frame image.

[0174] This application uses the same image processing method for each frame of the original image data.

[0175] A mapping unit 503 is used to map the current image feature to the three-dimensional voxel space based on a pre-established mapping table between the image feature and the three-dimensional voxel space to obtain a three-dimensional voxel feature;

[0176] The present application pre-establishes a mapping table between image features and three-dimensional voxel space, which effectively avoids the problem of inaccuracy in detecting depth from the image prediction perspective.

[0177] The process of mapping to obtain three-dimensional voxel features can be seen in the following expression:

[0178] ;

[0179] In the formula, represents the 3D voxel features, Represents the current image features, Represents the reverse lookup table of T(x,y,z), which is a mapping table between image features and 3D voxel space. img represents the image identifier. When there are multiple roadside cameras working simultaneously in the system, each roadside camera captures images from different perspectives. img is used to distinguish different roadside cameras. For example, img=1 represents the image of the first camera, img represents the image of the second camera, and so on.

[0180] T (x, y, z) is a mapping table pre-calculated based on the intrinsic parameter matrix and extrinsic parameter matrix of the roadside camera. The mapping table records the mapping relationship between image coordinates and voxel space coordinates, that is, the mapping relationship between image features and three-dimensional voxel space.

[0181] T′(x,y) is the reverse lookup table of T(x,y,z), containing the set of all voxels that map to pixel (x,y).

[0182] Three-dimensional voxel features refer to dividing an object or scene into a series of small cubes (voxels) in three-dimensional space, and extracting the intersection or feature information of these voxels with the surface of the object or other key points. They can be used to describe the shape, structure, surface features of the object, and its spatial relationship with other objects.

[0183] Voxel is the most basic unit of 3D volume data, usually a small cube. In 3D space, an object can be viewed as composed of many voxels.

[0184] It should be noted that a voxel in three-dimensional space can only correspond to one pixel in two-dimensional space, but a pixel in two-dimensional space may correspond to multiple voxels on the same line bundle.

[0185] In the process of mapping image features to three-dimensional voxel space, a pixel (x, y) on an image may correspond to multiple voxels. Since images do not have depth information, voxels at different depths on the same line of sight are mapped to the same pixel position. This means that for a certain pixel (x, y), there may be multiple voxels , satisfying the following projection relationship:

[0186] ;

[0187] Refers to Look up the voxel corresponding to the pixel (x, y) on the image marked with img.

[0188] Indicates the voxel found. In practical applications, multiple voxels may be found.

[0189] In the process of mapping the current image feature to the three-dimensional pixel space, the present application adopts a nearest voxel sorting method to determine one or more voxels corresponding to each pixel corresponding to the current image feature in the three-dimensional pixel space.

[0190] This application uses the nearest voxel sorting method to determine the voxel corresponding to the pixel, and the expression is as follows:

[0191] ;

[0192] In the formula, represents the most determined voxel.

[0193] A feature processing unit 504 is used to process the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image;

[0194] A fusion unit 505 is used to perform time-series alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain a bird's-eye view fusion feature;

[0195] The process of processing the three-dimensional voxel features includes but is not limited to compression, fusion and encoding.

[0196] The current bird's-eye view features may include: global view, vertical projection, proportional relationship, etc.

[0197] The recognition unit 506 is used to input the bird's-eye view fusion features into the detection head for recognition processing to obtain the target detection result.

[0198] In summary, the present invention discloses a target detection device based on a bird's-eye view, which obtains the original image data collected by the roadside camera, extracts features of the current frame image in the original image data to obtain the current image features, maps the current image features to the three-dimensional voxel space based on a pre-established mapping table of image features and three-dimensional voxel space to obtain three-dimensional voxel features, processes the three-dimensional voxel features to obtain the current bird's-eye view features of the current frame image, performs time-series alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain the bird's-eye view fusion features, and obtains the target detection results by inputting the bird's-eye view fusion features into the detection head for recognition processing. The present application adopts a detection technology based on a roadside bird's-eye view, and by performing a two-dimensional to three-dimensional transformation on the collected image features, the three-dimensional position information of vehicles, pedestrians and other traffic participants can be accurately captured, thereby obtaining complete information of the detection target in the entire road scene and improving the accuracy of the detection results.

[0199] In addition, this application adopts a pure convolution method, which avoids the problem that the Transformer architecture in the traditional solution is unfriendly to edge device deployment. The network structure is simple, the edge device inference speed is fast, and the network is easy to expand.

[0200] In one embodiment, the extraction unit 502 may be specifically used to:

[0201] Extracting multi-scale features from the current frame image using a backbone network;

[0202] The multi-scale features are fused to obtain the current image features.

[0203] In one embodiment, the mapping unit 503 can be specifically used to: 3. In the process of mapping the current image feature to the three-dimensional pixel space, use the nearest voxel sorting method to determine one or more voxels corresponding to each pixel corresponding to the current image feature in the three-dimensional pixel space.

[0204] In one embodiment, the target detection device based on the bird's-eye view may further include:

[0205] The mapping table establishing unit is used to establish a mapping table between image features and three-dimensional voxel space.

[0206] The mapping table establishment unit can be specifically used for:

[0207] Obtaining an intrinsic parameter matrix and an extrinsic parameter matrix of the roadside camera;

[0208] The three-dimensional voxel position of each voxel is converted into a camera coordinate system by using the external parameter matrix to obtain the position of each voxel in the camera coordinate system, wherein each voxel is: each small cubic unit obtained by dividing the continuous three-dimensional space where the area to be detected is located into a regular grid;

[0209] Projecting the position of each voxel in the camera coordinate system onto a two-dimensional image plane to obtain corresponding image pixel coordinates;

[0210] Each image pixel coordinate is associated with the corresponding three-dimensional voxel position to establish a mapping table between image features and three-dimensional voxel space.

[0211] In one embodiment, the feature processing unit 504 may be specifically used to:

[0212] compressing the three-dimensional voxel features into bird's-eye view features;

[0213] A weighted fusion multi-scale feature method is used for the bird's-eye view feature to obtain an initial bird's-eye view feature;

[0214] The initial bird's-eye view feature is encoded using a bird's-eye view encoder to obtain the current bird's-eye view feature.

[0215] In one embodiment, the feature processing unit 504 may also be used to:

[0216] The depth dimension of the three-dimensional voxel feature is compressed into a two-dimensional plane using a compression function to obtain the bird's-eye view feature.

[0217] It should be noted that, for the specific working principles of each component in the device embodiment, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0218] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (Personal Digital Assistants), PADs (Portable Application Devices), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0219] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the ROM 602 or the program loaded from the storage device 608 to the RAM 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0220] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0221] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the bird's-eye view-based target detection methods provided in the embodiments of the present application.

[0222] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the bird's-eye view based target detection methods provided in the embodiments of the present application.

[0223] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0224] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0225] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0226] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, training equipment, or data center to another website, computer, training equipment, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD (Digital Versatile Disc, Digital Versatile Disc)), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A target detection method based on a bird's-eye view, characterized in that: include: Obtaining raw image data collected by roadside cameras; Extracting features of a current frame image in the original image data to obtain current image features; Based on a pre-established mapping table between image features and three-dimensional voxel space, mapping the current image features to the three-dimensional voxel space to obtain three-dimensional voxel features; Processing the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image; Performing temporal alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain bird's-eye view fusion features; The bird's-eye view fusion features are input into the detection head for recognition processing to obtain the target detection result.

2. The target detection method according to claim 1, characterized in that: The extracting features of the current frame image in the original image data to obtain current image features includes: Extracting multi-scale features from the current frame image using a backbone network; The multi-scale features are fused to obtain the current image features.

3. The target detection method according to claim 1, characterized in that: The mapping of the current image feature to the three-dimensional voxel space based on the pre-established mapping table between the image feature and the three-dimensional voxel space to obtain the three-dimensional voxel feature includes: In the process of mapping the current image feature to the three-dimensional pixel space, a nearest voxel sorting method is used to determine one or more voxels corresponding to each pixel corresponding to the current image feature in the three-dimensional pixel space.

4. The target detection method based on a bird's-eye view according to any one of claims 1 to 3, characterized in that: The process of establishing the mapping table between image features and three-dimensional voxel space includes: Obtaining an intrinsic parameter matrix and an extrinsic parameter matrix of the roadside camera; The three-dimensional voxel position of each voxel is converted into a camera coordinate system by using the external parameter matrix to obtain the position of each voxel in the camera coordinate system, wherein each voxel is: each small cubic unit obtained by dividing the continuous three-dimensional space where the area to be detected is located into a regular grid; Projecting the position of each voxel in the camera coordinate system onto a two-dimensional image plane to obtain corresponding image pixel coordinates; Each image pixel coordinate is associated with the corresponding three-dimensional voxel position to establish a mapping table between image features and three-dimensional voxel space.

5. The target detection method according to claim 1, characterized in that: The processing of the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image includes: compressing the three-dimensional voxel features into bird's-eye view features; A weighted fusion multi-scale feature method is used for the bird's-eye view feature to obtain an initial bird's-eye view feature; The initial bird's-eye view feature is encoded using a bird's-eye view encoder to obtain the current bird's-eye view feature.

6. The target detection method according to claim 5, characterized in that: The compressing the three-dimensional voxel features into bird's-eye view features comprises: The depth dimension of the three-dimensional voxel feature is compressed into a two-dimensional plane using a compression function to obtain the bird's-eye view feature.

7. A target detection device based on a bird's-eye view, characterized in that: include: An acquisition unit, used to acquire raw image data collected by a roadside camera; An extraction unit, used for extracting features of a current frame image in the original image data to obtain current image features; A mapping unit, configured to map the current image feature to the three-dimensional voxel space based on a pre-established mapping table between the image feature and the three-dimensional voxel space, so as to obtain the three-dimensional voxel feature; A feature processing unit, used for processing the three-dimensional voxel features to obtain current bird's-eye view features of the current frame image; A fusion unit, used for performing temporal alignment and weighted fusion processing on the current bird's-eye view features and the bird's-eye view features of the historical frame images to obtain bird's-eye view fusion features; The recognition unit is used to input the bird's-eye view fusion features into the detection head for recognition processing to obtain the target detection result.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the target detection method based on a bird's-eye view as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the target detection method based on the bird's-eye view as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the bird's-eye view-based target detection method as described in any one of claims 1 to 6.

Images