Target detection method and device based on fisheye image and vehicle

Abstract

This application provides a target detection method, device, and vehicle based on fisheye images, relating to the field of vehicle detection technology. The method includes acquiring multiple fisheye images captured by multiple fisheye cameras mounted on the body of a target vehicle; extracting image features from the multiple fisheye images to obtain multiple sets of initial fisheye image features; determining the real-time region of interest (ROI) in each fisheye image based on the initial fisheye image features; determining the target fisheye image features of the ROI from each set of initial fisheye image features; mapping the target fisheye image features to a bird's-eye view BEV space to obtain the bird's-eye view features of the ROI; and performing 3D target detection on the ROI based on the bird's-eye view features. This application avoids the shortcomings of fixed-region selection in adapting to scene changes, ensures the accuracy of feature selection under different driving conditions, and provides reliable data support for the decision-making of autonomous driving systems (such as deceleration and obstacle avoidance).

Description

Technical Field

[0001] This application relates to the field of vehicle detection technology, and more specifically, to a target detection method, device, and vehicle based on fisheye images. Background Technology

[0002] In the development of autonomous driving technology for commercial vehicles, the accurate perception of the surrounding environment by the vehicle is the core prerequisite for ensuring driving safety. Compared with passenger cars, commercial vehicles (such as cargo trucks and large buses) have significant characteristics such as larger body size, longer wheelbase, and wider turning radius. Their blind spots are much larger than those of passenger cars. To achieve 360° all-around monitoring of the environment around the vehicle without blind spots, fisheye cameras are required.

[0003] However, while the ultra-wide-angle field of view of fisheye cameras brings advantages in monitoring range, it also raises new technical issues, adversely affecting the accuracy of subsequent environmental perception algorithms. Specifically, due to the imaging principle of fisheye images and the structural characteristics of commercial vehicles, the images contain a large number of areas obscured by the vehicle body. These obscured areas do not actually contain any real external environmental targets, but in existing projection processing logic, these obscured areas are included in the camera's effective monitoring range by default, and thus participate in subsequent feature extraction and target detection calculations.

[0004] This problem severely affects the accuracy of identifying key targets such as nearby obstacles, surrounding vehicles, and pedestrians, making it difficult to meet the high reliability requirements of commercial vehicle autonomous driving systems for near-distance obstacle perception. It restricts the improvement of the safety performance of commercial vehicle autonomous driving systems and poses a potential risk to vehicle driving safety. Therefore, there is an urgent need for an optimized solution that can solve the above-mentioned technical problems in order to improve the target detection accuracy and reliability of commercial vehicle surround-view perception systems. Summary of the Invention

[0005] This application addresses the shortcomings of the prior art by providing a target detection method, device, and vehicle based on fisheye images, in order to solve the problems existing in the prior art.

[0006] The technical solution adopted in the embodiments of this application is as follows: In a first aspect, embodiments of this application provide a target detection method based on fisheye images, including: Acquire multiple fisheye images captured by multiple fisheye cameras mounted on the body of the target vehicle; Image feature extraction is performed on the multi-channel fisheye images to obtain multiple sets of initial fisheye image features; Based on the features of each set of initial fisheye images, determine the real-time region of interest in the corresponding fisheye image; The target fisheye image features of the real-time region of interest are determined from each set of initial fisheye image features; The target fisheye image features are mapped to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest. Based on the bird's-eye view features, 3D target detection is performed on the real-time region of interest.

[0007] In one embodiment, determining the real-time region of interest in the corresponding fisheye image based on the features of each set of initial fisheye images includes: A preset vehicle body segmentation network is used to segment the vehicle body region based on the features of each set of initial fisheye images, thereby obtaining the vehicle body region in the corresponding fisheye image; Based on the vehicle body area, a vehicle body area mask map is obtained; Based on the preset region of interest mask and the vehicle body region mask, a real-time region of interest mask is obtained; The real-time region of interest is determined from the corresponding fisheye image based on the real-time region of interest mask map.

[0008] In one embodiment, obtaining a real-time region of interest mask based on a preset region of interest mask and the vehicle body region mask includes: The intersection of the preset region of interest mask and the vehicle body region mask is obtained; The intersection region in the preset region of interest mask is removed to obtain the real-time region of interest mask.

[0009] In one embodiment, before mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest, the method further includes: Based on the real-time region of interest mask, it is determined that the position of each pixel in the real-time region of interest is within the boundary range of the real-time region of interest mask; The step of mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest includes: If the position of each pixel in the real-time region of interest is within the boundary of the mask image of the real-time region of interest, then the target fisheye image features are mapped to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest.

[0010] In one embodiment, mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest includes: Based on a preset three-dimensional spatial range, the target fisheye image features are converted to the preset three-dimensional spatial range to obtain the bird's-eye view features of the real-time region of interest.

[0011] In one embodiment, before converting the target fisheye image features to the preset three-dimensional spatial range to obtain the bird's-eye view features of the real-time region of interest, the method further includes: Using the intrinsic and extrinsic parameters of each fisheye camera, the correspondence between the fisheye image corresponding to each fisheye camera and the three-dimensional voxels in the preset three-dimensional spatial range is determined respectively; The step of converting the target fisheye image features to the preset three-dimensional space range to obtain the bird's-eye view features of the real-time region of interest includes: By using the correspondence between the corresponding fisheye image and the three-dimensional voxels in the preset three-dimensional space, the features of the target fisheye image are converted into the preset three-dimensional space to obtain the bird's-eye view features of the real-time region of interest.

[0012] In one embodiment, mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest includes: The target fisheye image features are converted into a preset three-dimensional space to obtain the three-dimensional spatial features of the real-time region of interest; Based on the three-dimensional spatial features, the bird's-eye view features of the real-time region of interest are obtained.

[0013] In one embodiment, obtaining the bird's-eye view features of the real-time region of interest based on the three-dimensional spatial features includes: Based on the aforementioned three-dimensional spatial features, feature calculations are performed to obtain the bird's-eye view features of the real-time region of interest; or... Based on the three-dimensional spatial features, a preset bird's-eye view feature extraction network is used to extract features to obtain the bird's-eye view features of the real-time region of interest.

[0014] Secondly, embodiments of this application provide a control device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the target detection method based on fisheye images described in any of the above embodiments.

[0015] Thirdly, embodiments of this application provide a vehicle, including: a plurality of fisheye cameras disposed on the vehicle body, and a control device; the plurality of fisheye cameras are respectively connected to the control device; The multiple fisheye cameras are used to acquire multiple fisheye images and send the multiple fisheye images to the control device; The control device is used to execute the target detection method based on fisheye images as described in any of the above embodiments.

[0016] The beneficial effects of this application are as follows: This application provides a target detection method based on fisheye images, including: acquiring multiple fisheye images captured by multiple fisheye cameras mounted on the body of a target vehicle; extracting image features from the multiple fisheye images to obtain multiple sets of initial fisheye image features; determining a real-time region of interest in the corresponding fisheye image based on each set of initial fisheye image features; determining the target fisheye image features of the real-time region of interest from each set of initial fisheye image features; mapping the target fisheye image features to a bird's-eye view BEV space to obtain the bird's-eye view features of the real-time region of interest; and performing three-dimensional target detection on the real-time region of interest based on the bird's-eye view features.

[0017] By filtering the features of each initial fisheye image, the real-time region of interest in the corresponding fisheye image is determined, and 3D target detection is performed on the real-time region of interest. This avoids the shortcomings of fixed region filtering in dealing with scene changes, ensures the accuracy of feature filtering under different driving conditions, and provides reliable data support for the decision-making of autonomous driving systems (such as deceleration and avoidance). Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the target detection method based on fisheye images provided in the embodiments of this application; Figure 2 The second schematic flowchart of the target detection method based on fisheye images provided in the embodiments of this application; Figure 3 A schematic diagram of the vehicle body area in the fisheye image provided in this application; Figure 4 The vehicle body area mask provided in the embodiments of this application; Figure 5 The third schematic flowchart of the target detection method based on fisheye images provided in the embodiments of this application; Figure 6 This is a schematic diagram of the preset region of interest; Figure 7 Preset region of interest mask image; Figure 8 For real-time region of interest mask image; Figure 9 For real-time regions of interest; Figure 10 This is a schematic diagram illustrating 3D object detection without using real-time region of interest. Figure 11 This is a schematic diagram of 3D target detection using real-time region of interest. Figure 12 The fourth schematic flowchart of the target detection method based on fisheye images provided in the embodiments of this application; Figure 13 Fifth schematic flowchart of the target detection method based on fisheye images provided in the embodiments of this application; Figure 14 This is the sixth flowchart illustrating the target detection method based on fisheye images provided in the embodiments of this application. Figure 15 This is a schematic diagram of the target detection device based on fisheye images provided in an embodiment of this application; Figure 16 A schematic diagram of the structure of the control device provided in the embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0024] First, this application provides a vehicle, including a plurality of fisheye cameras arranged on the vehicle body and a control device, wherein the plurality of fisheye cameras are respectively connected to the control device.

[0025] Multiple fisheye cameras are used to acquire multiple fisheye images and send these images to a control device. The control device is used to execute the target detection method based on fisheye images according to any of the above embodiments. The fisheye cameras can be installed on the left, right, front, and rear sides of the vehicle; this application does not limit the specific location or number of fisheye cameras.

[0026] The following examples, in conjunction with the accompanying drawings, provide specific illustrations of the target detection method based on fisheye images provided in this application.

[0027] Figure 1 This is one of the flowcharts illustrating the target detection method based on fisheye images provided in the embodiments of this application, such as... Figure 1 As shown, the method includes: S101. Acquire multiple fisheye images captured by multiple fisheye cameras mounted on the body of the target vehicle.

[0028] The target vehicle of this application can be, for example, a commercial vehicle (such as a cargo truck, a large bus, etc.). Four fisheye cameras are installed at predetermined positions on the vehicle body, facing the front, rear, left, and right directions respectively, to achieve full coverage of the surrounding environment. Multiple fisheye images are simultaneously acquired in real time during the driving process through the fisheye cameras. The image data includes scene information such as obstacles around the vehicle and the road environment.

[0029] S102. Extract image features from multiple fisheye images to obtain multiple sets of initial fisheye image features.

[0030] The acquired multiple fisheye images are input into the image feature extraction module, where a ResNet or FPN network is used to extract features from each fisheye image. Downsampling is performed simultaneously during the extraction process to obtain multiple sets of initial fisheye image features that correspond one-to-one with the input images.

[0031] S103. Based on the features of each initial fisheye image, determine the real-time region of interest in the corresponding fisheye image.

[0032] For each set of initial fisheye image features, a real-time region of interest (ROI) is determined in the corresponding fisheye image. The real-time ROI is a specific region used to filter effective image features.

[0033] S104. Determine the target fisheye image features of the real-time region of interest from each set of initial fisheye image features.

[0034] Based on the mask image of the real-time region of interest, the features of each initial fisheye image are filtered. The feature information of the real-time region of interest identified by the mask image is retained, while the feature data of the vehicle body area and other irrelevant areas are removed, resulting in the target fisheye image features corresponding to each set of initial fisheye image features.

[0035] S105. Map the target fisheye image features to the bird's-eye view BEV space to obtain the real-time bird's-eye view features of the region of interest.

[0036] Specifically, the features of the target fisheye image can be converted into the preset three-dimensional spatial range to obtain the real-time bird's-eye view features of the region of interest.

[0037] For example, the preset 3D spatial range is a 3D spatial range centered on the commercial vehicle, with a length, width, and height of 64m, 64m, and 7m respectively. This range is divided into a 128×128×7 voxel grid. The corresponding pixel position of each 3D voxel on the 2D fisheye image is calculated by combining the camera's intrinsic and extrinsic parameters. Only the pixel features located within the real-time region of interest are inversely mapped to transform the target fisheye image features into the preset 3D spatial range, thus obtaining the bird's-eye view features of the real-time region of interest.

[0038] S106. Based on the bird's-eye view characteristics, perform 3D target detection on the real-time region of interest.

[0039] The bird's-eye view features of the BEV space are input into the FPN network to further extract deep features. The extracted deep features are then input into the detection module, which identifies the categories of other targets (such as other vehicles and obstacles around the target vehicle) in the real-time region of interest based on the feature information. At the same time, it calculates the length, width, and height dimensions of other targets and their distances relative to the target vehicle, and outputs the final 3D target detection results.

[0040] In summary, this embodiment provides a target detection method based on fisheye images. By filtering the features of each initial fisheye image, the real-time region of interest in the corresponding fisheye image is determined, and three-dimensional target detection is performed on the real-time region of interest. This avoids the shortcomings of fixed region filtering in dealing with scene changes, ensures the accuracy of feature filtering under different driving conditions, and provides reliable data support for the decision-making of autonomous driving systems (such as deceleration and avoidance).

[0041] Figure 2 This is a second schematic flowchart of the target detection method based on fisheye images provided in the embodiments of this application, as shown below. Figure 2 As shown, step S103 may specifically include: S201. Using a preset vehicle body segmentation network, vehicle body regions are segmented based on the features of each initial fisheye image to obtain the vehicle body regions in the corresponding fisheye images.

[0042] Each set of initial fisheye image features is input into the U-net segmentation network. This network learns the feature patterns of the vehicle body region to accurately segment the vehicle body in the fisheye image and outputs the corresponding vehicle body region in the fisheye image, clarifying the specific location and range of the vehicle body in the image.

[0043] Taking the image captured by the fisheye camera facing the right side of the vehicle as an example, Figure 3 This is a schematic diagram of the vehicle body area in the fisheye image provided in this application, for example. Figure 3 As shown, the green area represents the specific location and extent of the right side of the vehicle in the image.

[0044] S202. Obtain the vehicle body area mask based on the vehicle body area.

[0045] Figure 4 The vehicle body area mask provided in the embodiments of this application, such as Figure 4 As shown, a binary vehicle body region mask is generated based on the vehicle body region segmented by S201. In the mask, pixels corresponding to the vehicle body region are marked as 1 (white), and pixels corresponding to non-vehicle body regions are marked as 0 (black), thus achieving digital identification of the vehicle body region.

[0046] S203. Obtain the real-time region of interest mask based on the preset region of interest mask and vehicle body region mask.

[0047] Specifically, Figure 5 This is the third flowchart illustrating the target detection method based on fisheye images provided in the embodiments of this application, as shown below. Figure 5 As shown, S203 may include: S301. Perform region intersection on the preset region of interest mask and the vehicle body region mask to obtain the intersection region.

[0048] Figure 6 This is a schematic diagram of the preset region of interest. Figure 7 To obtain a preset region of interest mask, a logical AND operation is performed on the preset region of interest mask and the vehicle body region mask to obtain an intersection region mask, which is the overlapping part of the vehicle body region and the preset region of interest.

[0049] S302. Remove the intersecting regions in the preset region of interest mask to obtain the real-time region of interest mask.

[0050] In the preset region of interest mask image, the pixels corresponding to the intersection region obtained in S301 are removed (that is, the pixels in this part are changed from 1 to 0), and the remaining region constitutes the real-time region of interest mask image, such as... Figure 8 As shown.

[0051] S204. Determine the real-time region of interest from the corresponding fisheye image based on the real-time region of interest mask map.

[0052] The real-time region of interest (ROI) mask obtained in S203 is overlaid and matched with the corresponding original fisheye image. The pixel regions marked with 1 in the mask image represent the real-time ROI in the fisheye image, thus defining the effective range for subsequent feature extraction and object detection. The real-time ROI is as follows: Figure 9 As shown.

[0053] Alternatively, from a formulaic perspective, steps S201 to S204 can be written as the following formula: (1) (2) in, This is the mask image of the segmented vehicle body region. To pre-define the region of interest mask, The intersection of the predefined region of interest mask image and the vehicle body region mask image obtained from segmentation. This refers to the region of interest in real time.

[0054] Figure 10 This is a schematic diagram illustrating the 3D target detection performed in step S106 without using a real-time region of interest, as shown below. Figure 10 As shown, even without using real-time region of interest for 3D target detection, areas obscured by the vehicle body are included in the camera's effective monitoring range by default.

[0055] Figure 11 This is a schematic diagram of 3D object detection using real-time region of interest, as shown below. Figure 11As shown, when using real-time region of interest for 3D target detection, the area occluded by the target vehicle body will not be included in the effective monitoring range of the camera by default. This can effectively improve the accuracy of target detection and the accuracy of feature selection under different driving conditions, providing reliable data support for the decision-making of autonomous driving systems (such as deceleration and avoidance).

[0056] Figure 12 This is the fourth flowchart illustrating the target detection method based on fisheye images provided in the embodiments of this application. Figure 12 As shown, before performing S105, the method of this application further includes: S401. Based on the real-time region of interest mask, determine that the position of each pixel in the real-time region of interest is within the boundary range of the real-time region of interest mask.

[0057] Based on the generated real-time region of interest mask, the boundary range parameters of the mask are first determined, specifically including the minimum x-coordinate (X_min), maximum x-coordinate (X_max), minimum y-coordinate (Y_min), and maximum y-coordinate (Y_max) of the mask in the image coordinate system, forming the rectangular bounding box [X_min,X_max]×[Y_min,Y_max] of the real-time region of interest.

[0058] Iterate through each pixel within the real-time region of interest (ROI) and obtain the coordinates (Px, Py) for each pixel. Verify that each coordinate satisfies the boundary conditions: X_min ≤ Px ≤ X_max and Y_min ≤ Py ≤ Y_max. If all pixels satisfy these conditions, it is determined that all pixel positions within the ROI are within the mask map boundary. If any pixel does not satisfy these conditions, return to step S203 to regenerate the ROI mask map, ensuring the integrity and accuracy of the region boundaries.

[0059] Based on this, step S105 includes: S402. If the position of each pixel in the real-time region of interest is within the boundary of the real-time region of interest mask, then the target fisheye image features are mapped to the bird's-eye view BEV space to obtain the bird's-eye view features of the real-time region of interest.

[0060] When the S401 verification passes, meaning that all pixel positions in the real-time region of interest are within the boundary of the mask image, the BEV space mapping is performed on the target fisheye image features.

[0061] First, the preset three-dimensional spatial range parameters are loaded (centered on the target vehicle, with a length, width, and height of 64m×64m×7m, divided into a 128×128×7 voxel grid). The correspondence between each pixel in the real-time region of interest and the three-dimensional voxels is calculated by combining the camera's intrinsic and extrinsic parameters. Only the effective pixel features within the boundary range are reverse-mapped (excluding interference from invalid pixels outside the boundary). The three-dimensional spatial features are converted into bird's-eye view features in the BEV space to ensure that there is no distortion caused by feature overflow during the mapping process.

[0062] Alternatively, a ray casting method can be used to determine whether a pixel is within the region of interest polygon. For example, for an edge... , Coordinates are , Coordinates are Assuming A ray is considered to have made a valid intersection with another ray only if all of the following conditions are met: 1. The ordinate of a point lies between the two endpoints of the side (vertical range). This is a prerequisite for determining intersection. The ordinate of point P is P_t. y It must be located at the y-coordinates of the two endpoints of the edge. and Between. That is .

[0063] 2. The intersection point is to the right of point P (horizontally). Calculate the x-coordinate of the intersection point of the ray and the edge. Because the ray is horizontal to the right, it only works if the x-coordinate of the intersection point is greater than... Only then is this intersection counted.

[0064] Formula for calculating the X-coordinate of the intersection point (using the two-point form of a straight line):

[0065] The judgment condition is: < .

[0066] For each edge arrive The core judgment code (pseudocode) is as follows: Python if ( # Condition 1: The ordinate lies between the two endpoints. # Calculate the X coordinates of the intersection points

[0067] if ( < Condition 2: The intersection point is to the right of P. .

[0068] Then according to Whether a point is odd or even determines whether it is inside the polygon.

[0069] In one embodiment, Figure 13 This is the fifth flowchart illustrating the target detection method based on fisheye images provided in the embodiments of this application, as shown below. Figure 13 As shown, before converting the target fisheye image features to a preset three-dimensional spatial range according to the above embodiments to obtain the bird's-eye view features of the real-time region of interest, the method of this application further includes: S501. Using the camera intrinsic and extrinsic parameters of each fisheye camera, determine the correspondence between the fisheye image corresponding to each fisheye camera and the three-dimensional voxels in the preset three-dimensional space range.

[0070] For the four fisheye cameras mounted on the body of the commercial vehicle, the intrinsic parameters (including focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (including the translation vector T=[T_x,T_y,T_z] and rotation matrix R of the camera relative to the coordinate system of the target vehicle) of each camera are obtained and stored in the parameter database.

[0071] Based on a preset three-dimensional spatial range (64m×64m×7m, 128×128×7 voxels), the center coordinates (V_x,V_y,V_z) of each three-dimensional voxel are traversed (in the coordinate system of the target vehicle). The voxel coordinates are then converted to coordinates in the camera coordinate system (V'_x,V'_y,V'_z) using camera extrinsic parameters. The formula is: [V'_x,V'_y,V'_z]^T=R×[V_x,V_y,V_z]^T+T.

[0072] Then, by using the in-camera distortion correction model, the coordinates in the camera coordinate system are converted to coordinates (Px, Py) in the fisheye image pixel coordinate system, thus completing the association between a single 3D voxel and its corresponding fisheye image pixel. Repeating the above process, a one-to-one correspondence is established between the fisheye image corresponding to each fisheye camera and all 3D voxels within a preset 3D space, forming multiple sets of "pixel-voxel" mapping tables.

[0073] Based on this, the step of converting the target fisheye image features to a preset three-dimensional space range to obtain the real-time bird's-eye view features of the region of interest includes: S502. By adopting the correspondence between the corresponding fisheye image and the three-dimensional voxels in the preset three-dimensional space, the features of the target fisheye image are converted into the preset three-dimensional space to obtain the bird's-eye view features of the real-time region of interest.

[0074] The "pixel-voxel" mapping table established by S501 is called. For the target fisheye image features in the real-time region of interest, the corresponding three-dimensional voxel position (V_x,V_y,V_z) is quickly matched in the mapping table according to the coordinates (Px,Py) of each feature pixel.

[0075] The feature values ​​(such as texture, brightness, edge features, etc.) of the pixel are assigned to the matched 3D voxel. If multiple pixels correspond to the same 3D voxel, a feature fusion strategy (such as taking the mean or maximum value of features) is used to determine the final feature value of the voxel. After traversing all feature pixels of the target fisheye image, the target features are transformed into a preset 3D space range to generate the 3D spatial features of the real-time region of interest, providing a foundation for subsequent 3D target detection.

[0076] In another embodiment, Figure 14 This is the sixth flowchart illustrating the target detection method based on fisheye images provided in the embodiments of this application, as shown below. Figure 14 As shown, step S105, mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the real-time region of interest's bird's-eye view features, may include: S601. Convert the target fisheye image features into a preset three-dimensional space to obtain the three-dimensional spatial features of the region of interest in real time.

[0077] A 3D spatial transformation is performed on the target fisheye image features. First, target feature pixels within the real-time region of interest are selected. The coordinates of each pixel are used to look up the mapping table to determine its corresponding 3D voxel position. Then, the pixel feature values ​​are assigned to the corresponding voxel. For cases where multiple pixels correspond to the same voxel, the feature values ​​are fused by weighted averaging (the weights are set according to the distance from the pixel to the voxel center, with closer distances resulting in larger weights). Finally, a 3D spatial feature matrix containing all target information from the real-time region of interest is generated.

[0078] S602. Based on the three-dimensional spatial features, obtain the real-time bird's-eye view features of the region of interest.

[0079] Specifically, feature calculations can be performed based on three-dimensional spatial characteristics to obtain real-time bird's-eye view features of the region of interest. For example, three-dimensional spatial features can be converted into bird's-eye view features of the BEV space through feature aggregation methods such as mean, maximum, or weighted average.

[0080] Alternatively, based on the three-dimensional spatial features, a pre-defined bird's-eye view feature extraction network can be used to extract features and obtain the real-time bird's-eye view features of the region of interest. For example, the three-dimensional spatial feature matrix obtained by S601 can be used as input information and fed into the pre-defined bird's-eye view feature extraction network. The output layer of the bird's-eye view feature extraction network outputs a 128×128-dimensional BEV feature map. This feature map contains deep semantic features of the target within the real-time region of interest and can be directly used for category recognition and size and distance calculation in subsequent 3D target detection modules.

[0081] The apparatus, device, and storage medium for implementing the target detection method based on fisheye images provided in any of the above embodiments of this application will be explained below. The specific implementation process and the resulting technical effects are the same as those in the corresponding method embodiments. For the sake of brevity, the parts not mentioned in the following embodiments can be referred to the corresponding content in the method embodiments.

[0082] Figure 15 This is a schematic diagram of the target detection device based on fisheye images provided in the embodiments of this application, as shown below. Figure 15 As shown, this application provides a target detection device based on fisheye images, comprising: The acquisition module 10 is used to acquire multiple fisheye images collected by multiple fisheye cameras mounted on the body of the target vehicle.

[0083] The feature extraction module 20 is used to extract image features from the multi-channel fisheye images to obtain multiple sets of initial fisheye image features.

[0084] The first determining module 30 is used to determine the real-time region of interest in the corresponding fisheye image based on the features of each set of initial fisheye images.

[0085] The second determining module 40 is used to determine the target fisheye image features of the real-time region of interest from each set of initial fisheye image features.

[0086] The mapping module 50 is used to map the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest.

[0087] The detection module 60 is used to perform three-dimensional target detection on the real-time region of interest based on the bird's-eye view features.

[0088] Optionally, the first determining module 30 is used to segment the vehicle body region using a preset vehicle body segmentation network based on the features of each set of initial fisheye images to obtain the vehicle body region in the corresponding fisheye image; obtain a vehicle body region mask map based on the vehicle body region; obtain a real-time region of interest mask map based on the preset region of interest mask map and the vehicle body region mask map; and determine the real-time region of interest from the corresponding fisheye image based on the real-time region of interest mask map.

[0089] Optionally, the first determining module 30 is used to perform region intersection on the preset region of interest mask and the vehicle body region mask to obtain the intersection region; and remove the intersection region from the preset region of interest mask to obtain the real-time region of interest mask.

[0090] Optionally, the mapping module 50 is used to determine, based on the real-time region of interest mask, that the position of each pixel in the real-time region of interest is within the boundary range of the real-time region of interest mask; if the position of each pixel in the real-time region of interest is within the boundary range of the real-time region of interest mask, then the target fisheye image features are mapped to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest.

[0091] Optionally, the mapping module 50 is used to convert the target fisheye image features into the preset three-dimensional space range according to the preset three-dimensional space range, so as to obtain the bird's-eye view features of the real-time region of interest.

[0092] Optionally, the mapping module 50 is used to determine the correspondence between the fisheye image corresponding to each fisheye camera and the three-dimensional voxels in the preset three-dimensional space range by using the camera intrinsic and extrinsic parameters of each fisheye camera; and to convert the target fisheye image features into the preset three-dimensional space by using the correspondence between the corresponding fisheye image and the three-dimensional voxels in the preset three-dimensional space range to obtain the bird's-eye view features of the real-time region of interest.

[0093] Optionally, the mapping module 50 is used to convert the target fisheye image features into a preset three-dimensional space to obtain the three-dimensional spatial features of the real-time region of interest; and to obtain the bird's-eye view features of the real-time region of interest based on the three-dimensional spatial features.

[0094] Optionally, the mapping module 50 is used to perform feature calculation based on the three-dimensional spatial features to obtain the bird's-eye view features of the real-time region of interest; or, based on the three-dimensional spatial features, a preset bird's-eye view feature extraction network is used to extract features to obtain the bird's-eye view features of the real-time region of interest.

[0095] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0096] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0097] Figure 16 This is a schematic diagram of the structure of the control device provided in the embodiments of this application, such as... Figure 16 As shown, this application also provides a control device, including a processor 100, a storage medium 200 and a bus 300. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the target detection method based on fisheye images described in any of the above embodiments.

[0098] This application also provides a readable storage medium storing program instructions, which, when executed by a processor, implement the target detection method based on fisheye images described in any of the above embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0102] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target detection method based on fisheye images, characterized in that, include: Acquire multiple fisheye images captured by multiple fisheye cameras mounted on the body of the target vehicle; Image feature extraction is performed on the multi-channel fisheye images to obtain multiple sets of initial fisheye image features; Based on the features of each set of initial fisheye images, determine the real-time region of interest in the corresponding fisheye image; The target fisheye image features of the real-time region of interest are determined from each set of initial fisheye image features; The target fisheye image features are mapped to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest. Based on the bird's-eye view features, 3D target detection is performed on the real-time region of interest; The step of determining the real-time region of interest in the corresponding fisheye image based on the features of each initial fisheye image includes: A preset vehicle body segmentation network is used to segment the vehicle body region based on the features of each set of initial fisheye images, thereby obtaining the vehicle body region in the corresponding fisheye image; Based on the vehicle body area, a vehicle body area mask map is obtained; Based on the preset region of interest mask and the vehicle body region mask, a real-time region of interest mask is obtained; The real-time region of interest is determined from the corresponding fisheye image based on the real-time region of interest mask map; The step of obtaining a real-time region of interest mask based on a preset region of interest mask and the vehicle body region mask includes: The intersection of the preset region of interest mask and the vehicle body region mask is obtained; The intersection region in the preset region of interest mask is removed to obtain the real-time region of interest mask.

2. The method according to claim 1, characterized in that, Before mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest, the method further includes: Based on the real-time region of interest mask, it is determined that the position of each pixel in the real-time region of interest is within the boundary range of the real-time region of interest mask; The step of mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest includes: If the position of each pixel in the real-time region of interest is within the boundary of the mask image of the real-time region of interest, then the target fisheye image features are mapped to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest.

3. The method according to claim 1, characterized in that, The step of mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest includes: Based on a preset three-dimensional spatial range, the target fisheye image features are converted to the preset three-dimensional spatial range to obtain the bird's-eye view features of the real-time region of interest.

4. The method according to claim 3, characterized in that, Before converting the target fisheye image features to the preset three-dimensional spatial range according to the preset three-dimensional spatial range to obtain the bird's-eye view features of the real-time region of interest, the method further includes: Using the intrinsic and extrinsic parameters of each fisheye camera, the correspondence between the fisheye image corresponding to each fisheye camera and the three-dimensional voxels in the preset three-dimensional spatial range is determined respectively; The step of converting the target fisheye image features to the preset three-dimensional space range to obtain the bird's-eye view features of the real-time region of interest includes: By using the correspondence between the corresponding fisheye image and the three-dimensional voxels in the preset three-dimensional space, the features of the target fisheye image are converted into the preset three-dimensional space to obtain the bird's-eye view features of the real-time region of interest.

5. The method according to claim 1, characterized in that, The step of mapping the target fisheye image features to the bird's-eye view (BEV) space to obtain the bird's-eye view features of the real-time region of interest includes: The target fisheye image features are converted into a preset three-dimensional space to obtain the three-dimensional spatial features of the real-time region of interest; Based on the three-dimensional spatial features, the bird's-eye view features of the real-time region of interest are obtained.

6. The method according to claim 5, characterized in that, The step of obtaining the bird's-eye view features of the real-time region of interest based on the three-dimensional spatial features includes: Based on the aforementioned three-dimensional spatial features, feature calculations are performed to obtain the bird's-eye view features of the real-time region of interest; or... Based on the three-dimensional spatial features, a preset bird's-eye view feature extraction network is used to extract features to obtain the bird's-eye view features of the real-time region of interest.

7. A control device, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the target detection method based on fisheye images as described in any one of claims 1 to 6.

8. A vehicle, characterized in that, include: Multiple fisheye cameras and control devices are mounted on the vehicle body; The plurality of fisheye cameras are respectively connected to the control device; The multiple fisheye cameras are used to acquire multiple fisheye images and send the multiple fisheye images to the control device; The control device is used to execute the target detection method based on fisheye images as described in any one of claims 1 to 6.

Images