Target detection and semantic segmentation method, device and equipment, and storage medium

Abstract

The application relates to the field of image processing and discloses a target detection and semantic segmentation method, device and equipment and a storage medium, the method comprising the following steps: acquiring a same frame monocular image photographed by multiple cameras, and performing depth labeling to obtain a depth map corresponding to the monocular image; converting the image coordinates of each pixel in the depth map to a camera coordinate system according to the camera internal parameter, to obtain pseudo point clouds of each pixel in the depth map under the camera coordinate system; converting the pseudo point clouds and corresponding image features to a preset bird's-eye view coordinate system to obtain corresponding bird's-eye view point clouds and bird's-eye view features; and performing target detection and semantic segmentation under the bird's-eye view perspective based on the bird's-eye view point clouds and the corresponding bird's-eye view features. The method can break the strong dependence on ranging sensors under the premise of ensuring safety, reduces the hardware cost, effectively utilizes the results of 2D image perception tasks in 3D perception tasks, extracts image information and transforms the image information to a 3D space, and improves the performance of a perception algorithm.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method, apparatus, device, and storage medium for object detection and semantic segmentation. Background Technology

[0002] Because cameras are inexpensive and technologically mature, Level 2 / 3 assisted driving systems often employ a pure vision system composed of multiple cameras to provide input for perception algorithms in order to reduce the hardware costs associated with sensors, especially lidar. However, in Level 4 autonomous driving systems, to avoid single points of failure caused by the malfunction of a single sensor, in addition to perception schemes based on sensors with ranging capabilities such as lidar, a pure vision-based scheme is also needed as redundancy. This ensures that the perception module continues to function normally when sensors such as lidar fail, thereby guaranteeing the safety of the autonomous driving system.

[0003] In driver assistance and autonomous driving systems, the planning and control modules rely on the perception modules to provide 3D spatial detection results. To achieve stable output and overcome blind spots of a single camera, pure vision-based perception schemes need to transform the surround view image information composed of multiple cameras into a bird's-eye view coordinate system for processing. Therefore, driver assistance and autonomous driving systems also need to avoid strong reliance on ranging sensors for bird's-eye view. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem of the heavy reliance on ranging sensors such as lidar in various tasks involving bird's-eye view mapping.

[0005] The first aspect of this invention provides a method for object detection and semantic segmentation, comprising: acquiring a monocular image of the same frame captured by multiple cameras, and performing depth annotation on the monocular image to obtain a depth map corresponding to the monocular image; transforming the image coordinates of each pixel in the depth map to a corresponding camera coordinate system according to the camera intrinsic parameters of the multiple cameras, to obtain multiple pseudo point clouds of each pixel in the depth map under the camera coordinate system; transforming the multiple pseudo point clouds and corresponding image features to a preset bird's-eye view coordinate system to obtain corresponding bird's-eye view point clouds and bird's-eye view features; and performing object detection and semantic segmentation from a bird's-eye view perspective based on the bird's-eye view point clouds and corresponding bird's-eye view features.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the camera intrinsic parameters include focal length and the coordinates of the image center point; the focal length includes horizontal focal length and vertical focal length; and the image coordinates include horizontal image coordinates, vertical image coordinates, and depth image coordinates. The step of transforming the image coordinates of each pixel in the depth map to the corresponding camera coordinate system based on the camera intrinsic parameters of the multiple cameras to obtain multiple pseudo-point clouds of each pixel in the depth map under the corresponding camera coordinate system includes: calculating the horizontal camera coordinates of the pseudo-point clouds in the corresponding camera coordinate system based on the horizontal image coordinates, the horizontal focal length, and the coordinates of the image center point; calculating the vertical camera coordinates of the pseudo-point clouds in the corresponding camera coordinate system based on the vertical image coordinates, the vertical focal length, and the coordinates of the image center point; using the depth image coordinates as the depth camera coordinates of the pseudo-point clouds in the corresponding camera coordinate system; and generating the pseudo-point cloud corresponding to each pixel based on the horizontal camera coordinates, the vertical camera coordinates, and the depth image coordinates.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the depth map further carries the confidence level of each pixel. After generating the pseudo point cloud corresponding to each pixel based on the horizontal camera coordinates, the vertical camera coordinates, and the depth image coordinates, the method further includes: determining whether there are pixels in the depth map with a confidence level less than a preset confidence threshold; if so, deleting the pseudo point cloud corresponding to the pixel with a confidence level less than the preset confidence threshold.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of converting the plurality of pseudo point clouds and corresponding image features to a preset bird's-eye view coordinate system to obtain corresponding bird's-eye view point clouds and bird's-eye view features includes: acquiring camera extrinsic parameters of the plurality of cameras; converting the pseudo point clouds corresponding to each monocular image to a unified bird's-eye view coordinate system based on the camera extrinsic parameters to obtain dense bird's-eye view point clouds; and converting the image features corresponding to each pixel to the bird's-eye view coordinate system according to the first correspondence between each pixel in each monocular image and the pseudo point cloud to obtain bird's-eye view features of the bird's-eye view point cloud.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the step of transforming the pseudo-point clouds corresponding to each monocular image to a unified bird's-eye view coordinate system based on the camera extrinsic parameters to obtain dense bird's-eye view point clouds includes: constructing a transformation matrix from the camera coordinate system to the bird's-eye view coordinate system based on the camera extrinsic parameters; and transforming all pseudo-point clouds from their corresponding camera coordinate systems to the bird's-eye view coordinate system based on the transformation matrix of each camera to obtain dense bird's-eye view point clouds.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the step of converting the image features corresponding to each pixel in each monocular image to the bird's-eye view coordinate system based on the first correspondence between each pixel in each monocular image and the pseudo-point cloud to obtain the bird's-eye view features of the bird's-eye view point cloud includes: determining a second correspondence between the pseudo-point cloud and the bird's-eye view grid points in the bird's-eye view coordinate system based on the transformation matrix; converting the image features corresponding to each pixel in the depth map to the bird's-eye view grid points in the bird's-eye view coordinate system based on the first correspondence and the second correspondence; and performing rasterization processing on the image features converted to the bird's-eye view grid points to obtain the bird's-eye view features.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the step of rasterizing the image features converted to the bird's-eye view grid points to obtain bird's-eye view features includes: determining whether the bird's-eye view grid points correspond to multiple pixels based on the first correspondence and the second correspondence; if so, calculating the average value of the image features in the same bird's-eye view grid points or determining the maximum value of the image features in the same bird's-eye view grid points, and using the average value or the maximum value as the bird's-eye view feature of the bird's-eye view grid points.

[0012] A second aspect of the present invention provides a target detection and semantic segmentation apparatus, comprising: a depth annotation module for acquiring a monocular image of the same frame captured by multiple cameras and performing depth annotation on the monocular image to obtain a depth map corresponding to the monocular image; a pseudo point cloud generation module for converting the image coordinates of each pixel in the depth map to a corresponding camera coordinate system based on the camera intrinsic parameters of the multiple cameras to obtain multiple pseudo point clouds of each pixel in the depth map under the camera coordinate system; a conversion module for converting the multiple pseudo point clouds and corresponding image features to a preset bird's-eye view coordinate system to obtain corresponding bird's-eye view point clouds and bird's-eye view features; and a perception task execution module for performing target detection and semantic segmentation from a bird's-eye view perspective based on the bird's-eye view point clouds and corresponding bird's-eye view features.

[0013] Optionally, in a first implementation of the second aspect of the present invention, the camera intrinsic parameters include focal length and the coordinates of the image center point; the focal length includes a horizontal focal length and a vertical focal length; and the image coordinates include horizontal image coordinates, vertical image coordinates, and depth image coordinates. The pseudo-point cloud generation module is specifically used to: calculate the horizontal camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system based on the horizontal image coordinates, the horizontal focal length, and the coordinates of the image center point; calculate the vertical camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system based on the vertical image coordinates, the vertical focal length, and the coordinates of the image center point; use the depth image coordinates as the depth camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system; and generate a pseudo-point cloud corresponding to each pixel based on the horizontal camera coordinates, the vertical camera coordinates, and the depth image coordinates.

[0014] Optionally, in a second implementation of the second aspect of the present invention, the target detection and semantic segmentation device further includes a point cloud deletion module, which is specifically used to: determine whether there are pixels in the depth map with a confidence level less than a preset confidence threshold; if so, delete the pseudo point cloud corresponding to the pixels with a confidence level less than the preset confidence threshold.

[0015] Optionally, in a third implementation of the second aspect of the present invention, the conversion module specifically includes: an acquisition unit for acquiring camera extrinsic parameters of multiple cameras; a point cloud conversion unit for converting the pseudo point cloud corresponding to each monocular image to a unified bird's-eye view coordinate system based on the camera extrinsic parameters, thereby obtaining a dense bird's-eye view point cloud; and a feature conversion unit for converting the image features corresponding to each pixel to the bird's-eye view coordinate system according to a first correspondence between each pixel in each monocular image and the pseudo point cloud, thereby obtaining the bird's-eye view features of the bird's-eye view point cloud.

[0016] Optionally, in a fourth implementation of the second aspect of the present invention, the point cloud conversion unit is specifically used to: construct a conversion matrix from the camera coordinate system to the bird's-eye view coordinate system based on the camera extrinsic parameters; and convert all pseudo point clouds from the corresponding camera coordinate system to the bird's-eye view coordinate system based on the conversion matrix of each camera to obtain a dense bird's-eye view point cloud.

[0017] Optionally, in a fifth implementation of the second aspect of the present invention, the feature conversion unit specifically includes: a relation determination subunit, used to determine a second correspondence between the pseudo-point cloud and the bird's-eye view grid points in the bird's-eye view coordinate system according to the conversion matrix; a relation conversion subunit, used to convert the image features corresponding to each pixel in the depth map to the bird's-eye view grid points in the bird's-eye view coordinate system according to the first correspondence and the second correspondence; and a rasterization subunit, used to rasterize the image features converted to the bird's-eye view grid points to obtain bird's-eye view features.

[0018] Optionally, in a sixth implementation of the second aspect of the present invention, the rasterization subunit is further configured to: determine whether the bird's-eye view grid point corresponds to multiple pixels based on the first correspondence and the second correspondence; if so, calculate the average value of the image features in the same bird's-eye view grid point or determine the maximum value of the image features in the same bird's-eye view grid point, and use the average value or the maximum value as the bird's-eye view feature of the bird's-eye view grid point.

[0019] A third aspect of the present invention provides an object detection and semantic segmentation device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the object detection and semantic segmentation device to perform the steps of the object detection and semantic segmentation method described above.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described object detection and semantic segmentation method.

[0021] In the technical solution of this invention, a depth map corresponding to the monocular image is obtained by acquiring the same frame of monocular image captured by multiple cameras and performing depth annotation on the monocular image. Based on the camera intrinsic parameters of the multiple cameras, the image coordinates of each pixel in the depth map are transformed to the camera coordinate system to obtain a pseudo-point cloud of each pixel in the depth map under the camera coordinate system. The pseudo-point cloud and its corresponding image features are then transformed into a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features. Target detection and semantic segmentation are performed based on the bird's-eye view point cloud and the corresponding bird's-eye view features. This technical solution can eliminate the strong dependence of the perception algorithm on the ranging sensor while ensuring security, greatly reducing the cost of hardware kits. In 3D perception tasks, it directly and effectively utilizes the results of 2D image perception tasks to extract rich image information and transform it to 3D space, thereby improving the performance of the perception algorithm, especially its ability to handle long-tail scenes. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the first embodiment of the object detection and semantic segmentation method in this invention;

[0023] Figure 2 This is a schematic diagram of a second embodiment of the object detection and semantic segmentation method in this invention;

[0024] Figure 3This is a schematic diagram of one embodiment of the target detection and semantic segmentation device in this invention;

[0025] Figure 4 This is a schematic diagram of another embodiment of the target detection and semantic segmentation device in this invention;

[0026] Figure 5 This is a schematic diagram of one embodiment of the target detection and semantic segmentation device in this invention. Detailed Implementation

[0027] In the technical solution of this invention, a depth map corresponding to the monocular image is obtained by acquiring the same frame of monocular image captured by multiple cameras and performing depth annotation on the monocular image. Based on the camera intrinsic parameters of the multiple cameras, the image coordinates of each pixel in the depth map are transformed to the camera coordinate system to obtain a pseudo-point cloud of each pixel in the depth map under the camera coordinate system. The pseudo-point cloud and its corresponding image features are then transformed into a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features. Target detection and semantic segmentation are performed based on the bird's-eye view point cloud and the corresponding bird's-eye view features. This technical solution can eliminate the strong dependence of the perception algorithm on the ranging sensor while ensuring security, greatly reducing the cost of hardware kits. In 3D perception tasks, it directly and effectively utilizes the results of 2D image perception tasks to extract rich image information and transform it to 3D space, thereby improving the performance of the perception algorithm, especially its ability to handle long-tail scenes.

[0028] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the target detection and semantic segmentation method in this invention includes:

[0030] 101. Acquire the same frame of monocular image captured by multiple cameras, and perform depth annotation on the monocular image to obtain the depth map corresponding to the monocular image;

[0031] It is understood that the executing entity of this invention can be an object detection and semantic segmentation device, or it can be a terminal or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0032] In practical applications, camera technology is mature, stable, inexpensive, and rich in information, making cameras an important sensing element for autonomous driving perception. Compared with laser point clouds, camera images can provide richer details and texture information. The objects captured by the camera are mainly areas that need to be represented using point clouds, such as any area in an urban road environment that includes vehicles, pedestrians, traffic signs, or billboards. During autonomous driving, the area captured by the camera is the field of view area in front of the vehicle. In this embodiment, the result captured by the camera is a monocular image.

[0033] In practical applications, in L2 / L3 level assisted driving systems, to reduce the hardware cost of sensors, especially LiDAR, a pure vision system composed of multiple cameras is often used to provide input for the perception algorithm. However, in L4 level autonomous driving systems, to avoid single-point failures caused by the failure of a single sensor, in addition to perception schemes based on sensors with ranging capabilities such as LiDAR, a pure vision-based scheme is also needed as redundancy. This ensures the normal operation of the perception module when sensors such as LiDAR fail, thus ensuring the safety of the autonomous driving system. This method is used in scenarios where multiple cameras are installed for perception algorithms in L2 / L3 systems, or in L4 level autonomous driving systems as a cloud fusion scenario in addition to perception schemes based on sensors with ranging capabilities such as LiDAR. Therefore, multiple cameras are needed to capture the same frame of monocular image.

[0034] In this embodiment, a pre-trained monocular depth estimation model is used to estimate the depth of a monocular image, obtaining the depth value and confidence value of each pixel in the monocular image. Based on the depth value of each pixel, a corresponding depth map is generated. In practical applications, the monocular depth estimation model can be trained using supervised and unsupervised algorithms. The supervised algorithm directly uses a 2D image as input and the depth map as output for training. The unsupervised algorithm uses binocular image data collected by two cameras for joint training. The binocular data can predict each other to obtain corresponding disparity data, which is then evolved based on the relationship between disparity and depth. Alternatively, the correspondence problem of each pixel in the binocular image can be treated as a stereo matching problem for training. In this embodiment, the monocular depth estimation model uses dense image depth estimation based on semi-automatic annotation, which effectively solves the problem of poor depth annotation quality in industry. Based on the model design of VIP-Deeplab, the trained monocular depth estimation model can achieve relatively accurate depth estimation for obstacles and ground pixels. The depth value is defined as the z-axis value of the 3D point corresponding to each pixel in the image in the camera coordinate system.

[0035] 102. Based on the camera intrinsic parameters of multiple cameras, transform the image coordinates of each pixel in the depth map to the corresponding camera coordinate system to obtain multiple pseudo point clouds of each pixel in the depth map under the camera coordinate system.

[0036] In this embodiment, there are multiple coordinate systems, including a camera coordinate system, a pixel coordinate system, and a bird's-eye view coordinate system. The camera coordinate system takes the optical center as its origin, and the x and y directions parallel to the image as the Xc and Yc axes, respectively. The Zc axis is parallel to the optical axis, and the Xc, Yc, and Zc axes are perpendicular to each other. The unit is a length unit. The pixel coordinate system takes the upper left vertex of the imaging plane as its origin and is introduced to describe the coordinate position of a pixel in a digital image.

[0037] In this embodiment, for any pixel pi(u, v) in an image, the depth estimation model predicts its depth d and confidence level c, where d ∈ [0, dmax], c ∈ (0, 1), and dmax is the predefined maximum depth predicted by the model. Based on the calibrated camera intrinsic parameters, including the focal lengths in two directions (horizontal focal length fx and vertical focal length fy) and the image center cx, cy, the normalized coordinates pi(u', v') of pixel pi(u, v) are first calculated, where u' = (u - cx) / fx, v' = (v - cy) / fy. Then, based on the depth, the coordinates Pi(u'd, v'd, d) of the corresponding 3D point in the camera coordinate system are obtained. It should be noted that for pixels with low confidence, especially background pixels with large depth variations, when the confidence level c < 0.5, the depth predicted by the model for that pixel will not be accepted, i.e., the corresponding 3D point will be ignored. By processing each pixel in the image using the method described above, a pseudo-point cloud in the camera coordinate system will be obtained.

[0038] 103. Transform multiple pseudo point clouds and their corresponding image features into a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features;

[0039] In this embodiment, the pseudo point cloud in each camera coordinate system is transformed to a unified bird's-eye view coordinate system based on the extrinsic parameters of each camera. This results in a dense bird's-eye view point cloud generated by multiple cameras. Although its positional accuracy is inferior to that of the point cloud obtained by the range sensor, it can achieve several times the point cloud density and resolution of the range sensor, depending on the image resolution and the quality of the image depth annotation.

[0040] In this embodiment, during the generation of the pseudo-point cloud, a one-to-one correspondence between each pixel in each camera image and each 3D point in the resulting pseudo-point cloud can be obtained simultaneously. Based on this correspondence, image information at various pixel granularities, including pixel RGB values, image semantics and instance segmentation results, and parametric image detection results based on line segments or rectangles, can be transformed into bird's-eye view coordinates.

[0041] In this embodiment, considering that bird's-eye view features usually need to be rasterized to adapt to the structure of convolutional neural networks and reduce computation, during the transformation process, a bird's-eye view grid point may correspond to multiple different pixels at the same time. Therefore, an order-independent aggregation method is required, such as taking the average value or the maximum value, to process the image features corresponding to different pixels in the feature dimension. Finally, the rasterized feature map is obtained by fusion.

[0042] 104. Perform target detection and semantic segmentation from the perspective of the bird's-eye view based on the point cloud of the bird's-eye view and the corresponding bird's-eye view features.

[0043] In this embodiment, the obtained bird's-eye view features can be used to design a multi-task model based on the bird's-eye view. This scheme adopts a U-Net-based network structure design, which, after extracting high-dimensional multi-scale features, further completes tasks such as obstacle detection based on bounding boxes, drivable area segmentation based on the bird's-eye view grid, and semantic map element segmentation. The aforementioned multi-tasks can use the same backbone network, namely the U-Net network, to share computation. The detection task is based on a two-stage network dependent on region proposal, while drivable area and semantic map element segmentation are based on a simple pixel segmentation network.

[0044] In the technical solution of this invention, a depth map corresponding to the monocular image is obtained by acquiring the same frame of monocular image captured by multiple cameras and performing depth annotation on the monocular image. Based on the camera intrinsic parameters of the multiple cameras, the image coordinates of each pixel in the depth map are transformed to the camera coordinate system to obtain a pseudo-point cloud of each pixel in the depth map under the camera coordinate system. The pseudo-point cloud and its corresponding image features are then transformed into a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features. Target detection and semantic segmentation are performed based on the bird's-eye view point cloud and the corresponding bird's-eye view features. This technical solution can eliminate the strong dependence of the perception algorithm on the ranging sensor while ensuring security, greatly reducing the cost of hardware kits. In 3D perception tasks, it directly and effectively utilizes the results of 2D image perception tasks to extract rich image information and transform it to 3D space, thereby improving the performance of the perception algorithm, especially its ability to handle long-tail scenes.

[0045] Please see Figure 2 The second embodiment of the target detection and semantic segmentation method in this invention includes:

[0046] 201. Acquire the same frame of monocular image captured by multiple cameras, and perform depth annotation on the monocular image to obtain the depth map corresponding to the monocular image;

[0047] 202. Calculate the horizontal camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system based on the horizontal image coordinates, horizontal focal length, and coordinates of the image center point;

[0048] 203. Calculate the vertical camera coordinates of the pseudo point cloud in the corresponding camera coordinate system based on the vertical image coordinates, vertical focal length, and coordinates of the image center point;

[0049] 204. Use the depth image coordinates as the depth camera coordinates of the pseudo point cloud in the corresponding camera coordinate system, and generate the pseudo point cloud corresponding to each pixel based on the horizontal camera coordinates, vertical camera coordinates and depth image coordinates.

[0050] In this embodiment, multiple coordinate systems exist, including a camera coordinate system, a pixel coordinate system, and a bird's-eye view coordinate system. An image coordinate system exists before the camera and pixel coordinate systems. The image coordinate system uses the intersection point of the camera's optical axis and the imaging plane (principal point) as its origin and describes the position of an object projected onto the imaging plane, typically in millimeters. In this embodiment, the depth image is first transformed from the pixel coordinate system to the camera coordinate system, and then from the camera coordinate system to the bird's-eye view coordinate system. The image coordinate system exists between the pixel and camera coordinate systems, and the transformation relationship between the image and pixel coordinate systems is as follows:

[0051]

[0052] Converted into matrix form:

[0053]

[0054] Where (u, v) represent the number of rows and columns of the pixel, (u0, v0) represent the coordinates of the origin of the image coordinate system in the pixel coordinate system, and dx and dy represent the physical dimensions of a single pixel on the x-axis and y-axis, respectively. The transformation relationship between the image coordinate system and the camera coordinate system can be obtained from the principle of triangle similarity:

[0055]

[0056] Converted into matrix form:

[0057]

[0058] Substituting the conversion formula between pixel coordinates and image coordinates and simplifying, we can obtain:

[0059]

[0060] Among them, f x =f / dx, f y =f / dy represents the camera's focal length along the x-axis and y-axis, respectively, which are the horizontal and vertical focal lengths. Knowing the camera's focal lengths along the x-axis and y-axis, as well as the coordinates of the image coordinate system origin in the pixel coordinate system, allows for conversion between the camera coordinate system and the pixel coordinate system. Through calibration, the aforementioned (u0, v0) and f can be obtained. x and f y .

[0061] In this embodiment, x can be obtained based on the transformation matrix described above. c =(u-u0) / f x y c =(v-v0) / f yBased on the coordinates of each pixel in the pixel coordinate system, the x-axis and y-axis coordinates of each pixel in the camera coordinate system can be calculated. The depth value d is used as the z-axis coordinate in the camera coordinate system, thus converting the pixel in the depth value to the camera coordinate system. The 3D point of each coordinate is used as a pseudo point cloud, thus obtaining the pseudo point cloud of each pixel in the camera coordinate system.

[0062] In this embodiment, the method further includes: determining whether there are pixels in the depth map with a confidence level less than a preset confidence threshold; if so, deleting the pseudo point cloud corresponding to the pixels with a confidence level less than the preset confidence threshold.

[0063] Specifically, the depth estimation model predicts the depth value d and confidence value c for each pixel, where d∈[0,dmax], c∈(0,1), dmax is the predefined maximum depth predicted by the model, and confidence refers to the accuracy and reliability of the depth estimation, i.e. the quality of the point cloud. For pixels with low confidence, especially background pixels with large depth variations, when the confidence c<0.5, the depth predicted by the model for that pixel will not be accepted, i.e., its corresponding pseudo point cloud will be ignored.

[0064] 205. Obtain camera extrinsic parameters from multiple cameras;

[0065] In practical applications, in order to transform the pseudo point clouds corresponding to multiple cameras to a unified bird's-eye view coordinate system, it is necessary to obtain the camera extrinsic parameters corresponding to different cameras. The camera extrinsic parameters include the deflection angle and translation amount of the camera coordinate system corresponding to each camera relative to the bird's-eye view coordinate system, denoted as R and T respectively.

[0066] 206. Based on camera extrinsic parameters, the pseudo point clouds corresponding to each monocular image are transformed to a unified bird's-eye view coordinate system to obtain dense bird's-eye view point clouds;

[0067] In this embodiment, the step of transforming the pseudo point clouds corresponding to each monocular image to a unified bird's-eye view coordinate system based on the camera extrinsic parameters to obtain dense bird's-eye view point clouds includes: constructing a transformation matrix from the camera coordinate system to the bird's-eye view coordinate system based on the camera extrinsic parameters; and transforming all pseudo point clouds from their corresponding camera coordinate systems to the bird's-eye view coordinate system based on the transformation matrix of each camera to obtain dense bird's-eye view point clouds.

[0068] Specifically, the transformation relationship from the camera coordinate system to the bird's-eye view coordinate system is as follows:

[0069]

[0070] in, Let x be the transformation matrix. w y w z wThese are the coordinates of the pseudo-point cloud transformed into the bird's-eye view coordinate system.

[0071] 207. Based on the first correspondence between each pixel in each monocular image and the pseudo point cloud, the image features corresponding to the pixels are transformed to the bird's-eye view coordinate system to obtain the bird's-eye view features of the bird's-eye view point cloud.

[0072] In this embodiment, the step of converting the image features corresponding to each pixel in each monocular image to the bird's-eye view coordinate system based on the first correspondence between each pixel in each monocular image and the pseudo-point cloud to obtain the bird's-eye view features of the bird's-eye view point cloud includes: determining the second correspondence between the pseudo-point cloud and the bird's-eye view grid points in the bird's-eye view coordinate system based on the transformation matrix; converting the image features corresponding to each pixel in the depth map to the bird's-eye view grid points in the bird's-eye view coordinate system based on the first and second correspondences; and performing rasterization processing on the image features converted to the bird's-eye view grid points to obtain the bird's-eye view features.

[0073] Specifically, based on the correspondence between pixels and pseudo-point clouds, and the correspondence between pseudo-point clouds and bird's-eye view grid points, the correspondence between pixels and bird's-eye view grid points can be obtained. Since the bird's-eye view grid points are in a unified coordinate system and there are monocular images of the same frame captured by multiple cameras, there are multiple pixels and pseudo-point clouds corresponding to the bird's-eye view grid points. Based on this correspondence, image information of various pixel granularities, including the RGB values ​​of pixels, image semantics and instance segmentation results, and parametric image detection results based on line segments or rectangles, can be transformed into bird's-eye view coordinate system.

[0074] In this embodiment, the step of rasterizing the image features converted to the bird's-eye view grid points to obtain bird's-eye view features includes: determining whether the bird's-eye view grid point corresponds to multiple pixels based on the first correspondence and the second correspondence; if so, calculating the average value of the image features in the same bird's-eye view grid point or determining the maximum value of the image features in the same bird's-eye view grid point, and using the average value or the maximum value as the bird's-eye view feature of the bird's-eye view grid point.

[0075] Specifically, considering that bird's-eye view features typically require rasterization to adapt to the structure of convolutional neural networks and reduce computational cost, a single bird's-eye view grid point may correspond to multiple different pixels during the transformation process. Therefore, an order-independent aggregation method, such as averaging or maximizing, is needed to process the image features corresponding to different pixels along the feature dimension. Finally, the fused rasterized feature map can be used as input to the bird's-eye view model to complete various perception tasks.

[0076] 208. Perform target detection and semantic segmentation from the perspective of the bird's-eye view based on the point cloud of the bird's-eye view and the corresponding bird's-eye view features.

[0077] This embodiment, based on the previous embodiment, details the generation process of pseudo-point clouds and the process of transforming the pseudo-point clouds and corresponding image features to a bird's-eye view coordinate system. Specifically, it involves acquiring the camera extrinsic parameters of multiple cameras; transforming the pseudo-point clouds corresponding to each monocular image to a unified bird's-eye view coordinate system based on these parameters, resulting in a dense bird's-eye view point cloud; and, according to the first correspondence between each pixel in each monocular image and the pseudo-point cloud, transforming the image features corresponding to each pixel to the bird's-eye view coordinate system, thus obtaining the bird's-eye view features of the bird's-eye view point cloud. This method, while ensuring security, eliminates the strong dependence of perception algorithms on ranging sensors, significantly reducing hardware costs. In 3D perception tasks, it directly and effectively utilizes the results of 2D image perception tasks, extracting rich image information and transforming it to 3D space to improve the performance of perception algorithms, especially their ability to handle long-tailed scenes.

[0078] The above describes the object detection and semantic segmentation method in the embodiments of the present invention. The following describes the object detection and semantic segmentation apparatus in the embodiments of the present invention. Please refer to [link / reference]. Figure 3 One embodiment of the target detection and semantic segmentation device in this invention includes:

[0079] The depth annotation module 301 is used to acquire the same frame of monocular image captured by multiple cameras, and to perform depth annotation on the monocular image to obtain the depth map corresponding to the monocular image.

[0080] The pseudo point cloud generation module 302 is used to transform the image coordinates of each pixel in the depth map to the corresponding camera coordinate system according to the camera intrinsic parameters of the multiple cameras, so as to obtain multiple pseudo point clouds of each pixel in the depth map under the camera coordinate system.

[0081] The conversion module 303 is used to convert the multiple pseudo point clouds and their corresponding image features to a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features.

[0082] The perception task execution module 304 is used to perform target detection and semantic segmentation from the perspective of the bird's-eye view based on the bird's-eye view point cloud and the corresponding bird's-eye view features.

[0083] In this embodiment of the invention, the target detection and semantic segmentation device runs the aforementioned target detection and semantic segmentation method. The device acquires the same frame of monocular image captured by multiple cameras and performs depth annotation on the monocular image to obtain a depth map corresponding to the monocular image. Based on the camera intrinsic parameters of the multiple cameras, it transforms the image coordinates of each pixel in the depth map to the camera coordinate system, obtaining a pseudo-point cloud of each pixel in the depth map under the camera coordinate system. The pseudo-point cloud and its corresponding image features are then mapped to a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features. Based on the bird's-eye view point cloud and the corresponding bird's-eye view features, target detection and semantic segmentation are performed from the bird's-eye view perspective. The technical solution of this application can eliminate the strong dependence of the perception algorithm on the ranging sensor while ensuring security, greatly reducing the cost of hardware kits. In 3D perception tasks, it directly and effectively utilizes the results of 2D image perception tasks, extracts rich image information, and transforms it to 3D space to improve the performance of the perception algorithm, especially its ability to handle long-tail scenes.

[0084] Please see Figure 4 A second embodiment of the target detection and semantic segmentation device in this invention includes:

[0085] The depth annotation module 301 is used to acquire the same frame of monocular image captured by multiple cameras, and to perform depth annotation on the monocular image to obtain the depth map corresponding to the monocular image.

[0086] The pseudo point cloud generation module 302 is used to transform the image coordinates of each pixel in the depth map to the corresponding camera coordinate system according to the camera intrinsic parameters of the multiple cameras, so as to obtain multiple pseudo point clouds of each pixel in the depth map under the camera coordinate system.

[0087] The conversion module 303 is used to convert the multiple pseudo point clouds and their corresponding image features to a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features.

[0088] The perception task execution module 304 is used to perform target detection and semantic segmentation from the perspective of the bird's-eye view based on the bird's-eye view point cloud and the corresponding bird's-eye view features.

[0089] In this embodiment, the camera intrinsic parameters include focal length and the coordinates of the image center point. The focal length includes horizontal focal length and vertical focal length, and the image coordinates include horizontal image coordinates, vertical image coordinates, and depth image coordinates. The pseudo-point cloud generation module 302 is specifically used to: calculate the horizontal camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system based on the horizontal image coordinates, the horizontal focal length, and the coordinates of the image center point; calculate the vertical camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system based on the vertical image coordinates, the vertical focal length, and the coordinates of the image center point; use the depth image coordinates as the depth camera coordinates of the pseudo-point cloud in the corresponding camera coordinate system, and generate the pseudo-point cloud corresponding to each pixel based on the horizontal camera coordinates, the vertical camera coordinates, and the depth image coordinates.

[0090] In this embodiment, the target detection and semantic segmentation device further includes a point cloud deletion module 305, which is specifically used to: determine whether there are pixels in the depth map with a confidence level less than a preset confidence threshold; if so, delete the pseudo point cloud corresponding to the pixels with a confidence level less than the preset confidence threshold.

[0091] In this embodiment, the conversion module 303 specifically includes: an acquisition unit 3031, used to acquire camera extrinsic parameters of multiple cameras; a point cloud conversion unit 3032, used to convert the pseudo point cloud corresponding to each monocular image to a unified bird's-eye view coordinate system based on the camera extrinsic parameters, to obtain a dense bird's-eye view point cloud; and a feature conversion unit 3033, used to convert the image features corresponding to each pixel to the bird's-eye view coordinate system according to the first correspondence between each pixel in each monocular image and the pseudo point cloud, to obtain the bird's-eye view features of the bird's-eye view point cloud.

[0092] In this embodiment, the point cloud conversion unit 3032 is specifically used to: construct a conversion matrix from the camera coordinate system to the bird's-eye view coordinate system based on the camera extrinsic parameters; and convert all pseudo point clouds from the corresponding camera coordinate system to the bird's-eye view coordinate system according to the conversion matrix of each camera to obtain a dense bird's-eye view point cloud.

[0093] In this embodiment, the feature conversion unit specifically includes: a relationship determination subunit 30331, used to determine a second correspondence between the pseudo-point cloud and the bird's-eye view grid points in the bird's-eye view coordinate system according to the conversion matrix; a relationship conversion subunit 30332, used to convert the image features corresponding to each pixel in the depth map to the bird's-eye view grid points in the bird's-eye view coordinate system according to the first correspondence and the second correspondence; and a rasterization subunit 30333, used to perform rasterization processing on the image features converted to the bird's-eye view grid points to obtain bird's-eye view features.

[0094] In this embodiment, the rasterization subunit 30333 is further configured to: determine whether the bird's-eye view grid point corresponds to multiple pixels based on the first correspondence and the second correspondence; if so, calculate the average value of the image features in the same bird's-eye view grid point or determine the maximum value of the image features in the same bird's-eye view grid point, and use the average value or the maximum value as the bird's-eye view feature of the bird's-eye view grid point.

[0095] This implementation details the specific functions of each module of the target detection and semantic segmentation device and the unit composition of some modules. Through the modules and units of this device, the perception algorithm can be freed from its strong dependence on the ranging sensor while ensuring safety, which greatly reduces the cost of the hardware kit. In 3D perception tasks, the results of 2D image perception tasks can be directly and effectively utilized to extract rich image information and transform it into 3D space to improve the performance of the perception algorithm, especially its ability to handle long-tailed scenes.

[0096] above Figure 3 and Figure 4 The target detection and semantic segmentation device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The target detection and semantic segmentation device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0097] Figure 5 This is a schematic diagram of the structure of a target detection and semantic segmentation device 500 provided in an embodiment of the present invention. The target detection and semantic segmentation device 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the target detection and semantic segmentation device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the target detection and semantic segmentation device 500 to implement the steps of the above-described target detection and semantic segmentation method.

[0098] The object detection and semantic segmentation device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated target detection and semantic segmentation device structure does not constitute a limitation on the target detection and semantic segmentation device provided in this application. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0099] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the object detection and semantic segmentation method.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for object detection and semantic segmentation, characterized in that, The object detection and semantic segmentation method includes: Acquire the same frame of monocular image captured by multiple cameras, and perform depth annotation on the monocular image to obtain the depth map corresponding to the monocular image; Based on the camera intrinsic parameters of the multiple cameras, the image coordinates of each pixel in the depth map are transformed to the corresponding camera coordinate system to obtain multiple pseudo point clouds of each pixel in the depth map under the camera coordinate system. The multiple pseudo point clouds and their corresponding image features are transformed into a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point clouds and bird's-eye view features. Based on the bird's-eye view point cloud and the corresponding bird's-eye view features, target detection and semantic segmentation are performed from the bird's-eye view perspective. The step of converting the multiple pseudo point clouds and their corresponding image features to a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point clouds and bird's-eye view features includes: Obtain the camera extrinsic parameters of multiple cameras; construct a transformation matrix from the camera coordinate system to the bird's-eye view coordinate system based on the camera extrinsic parameters; transform all pseudo point clouds from their corresponding camera coordinate systems to the bird's-eye view coordinate system according to the transformation matrix of each camera, to obtain a dense bird's-eye view point cloud; based on the first correspondence between each pixel in each monocular image and the pseudo point cloud, transform the image features corresponding to the pixels to the bird's-eye view coordinate system, to obtain the bird's-eye view features of the bird's-eye view point cloud.

2. The target detection and semantic segmentation method according to claim 1, characterized in that, The camera intrinsic parameters include focal length and the coordinates of the image center point. The focal length includes horizontal focal length and vertical focal length. The image coordinates include horizontal image coordinates, vertical image coordinates, and depth image coordinates. The step of transforming the image coordinates of each pixel in the depth map to the corresponding camera coordinate system based on the camera intrinsic parameters of the multiple cameras, to obtain multiple pseudo point clouds of each pixel in the depth map in the camera coordinate system, includes: Calculate the horizontal camera coordinates of the pseudo point cloud in the corresponding camera coordinate system based on the horizontal image coordinates, the horizontal focal length, and the coordinates of the image center point. Calculate the vertical camera coordinates of the pseudo point cloud in the corresponding camera coordinate system based on the vertical image coordinates, the vertical focal length, and the coordinates of the image center point. The depth image coordinates are used as the depth camera coordinates of the pseudo point cloud in the corresponding camera coordinate system, and the pseudo point cloud corresponding to each pixel is generated based on the horizontal camera coordinates, the vertical camera coordinates and the depth image coordinates.

3. The target detection and semantic segmentation method according to claim 2, characterized in that, The depth map also carries the confidence score for each pixel. After generating the pseudo-point cloud corresponding to each pixel based on the horizontal camera coordinates, the vertical camera coordinates, and the depth image coordinates, the method further includes: Determine whether there are pixels in the depth map with a confidence level lower than a preset confidence threshold; If they exist, the pseudo point clouds corresponding to pixels with confidence scores lower than the preset confidence threshold will be deleted.

4. The target detection and semantic segmentation method according to claim 1, characterized in that, The step of transforming the image features corresponding to each pixel to the bird's-eye view coordinate system based on the first correspondence between each pixel in each monocular image and the pseudo point cloud to obtain the bird's-eye view features of the bird's-eye view point cloud includes: Based on the transformation matrix, a second correspondence is determined between the pseudo-point cloud and the bird's-eye view grid points in the bird's-eye view coordinate system; Based on the first correspondence and the second correspondence, the image features corresponding to each pixel in the depth map are transformed into the bird's-eye view grid points in the bird's-eye view coordinate system; The image features converted to the bird's-eye view grid points are rasterized to obtain the bird's-eye view features.

5. The target detection and semantic segmentation method according to claim 4, characterized in that, The process of rasterizing the image features converted to the bird's-eye view grid points to obtain bird's-eye view features includes: Based on the first correspondence and the second correspondence, determine whether the bird's-eye view grid point corresponds to multiple pixels; If they exist, calculate the average value of the image features in the same bird's-eye view grid point or determine the maximum value of the image features in the same bird's-eye view grid point, and use the average value or the maximum value as the bird's-eye view feature of the bird's-eye view grid point.

6. A target detection and semantic segmentation device, characterized in that, The target detection and semantic segmentation device includes: The depth annotation module is used to acquire the same frame of monocular image captured by multiple cameras, and to perform depth annotation on the monocular image to obtain the depth map corresponding to the monocular image. The pseudo point cloud generation module is used to transform the image coordinates of each pixel in the depth map to the corresponding camera coordinate system based on the camera intrinsic parameters of multiple cameras, so as to obtain multiple pseudo point clouds of each pixel in the depth map in the camera coordinate system. The conversion module is used to convert the multiple pseudo point clouds and their corresponding image features to a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point cloud and bird's-eye view features. The perception task execution module is used to perform target detection and semantic segmentation from the perspective of the bird's-eye view based on the bird's-eye view point cloud and the corresponding bird's-eye view features. The step of converting the multiple pseudo point clouds and their corresponding image features to a preset bird's-eye view coordinate system to obtain the corresponding bird's-eye view point clouds and bird's-eye view features includes: Obtain the camera extrinsic parameters of multiple cameras; construct a transformation matrix from the camera coordinate system to the bird's-eye view coordinate system based on the camera extrinsic parameters; transform all pseudo point clouds from their corresponding camera coordinate systems to the bird's-eye view coordinate system according to the transformation matrix of each camera, to obtain a dense bird's-eye view point cloud; based on the first correspondence between each pixel in each monocular image and the pseudo point cloud, transform the image features corresponding to the pixels to the bird's-eye view coordinate system, to obtain the bird's-eye view features of the bird's-eye view point cloud.

7. A target detection and semantic segmentation device, characterized in that, The target detection and semantic segmentation device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the object detection and semantic segmentation device to perform the steps of the object detection and semantic segmentation method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the object detection and semantic segmentation method as described in any one of claims 1-5.

Images