Object recognition method, device, vehicle and storage medium

Abstract

The application discloses an object recognition method and device, a vehicle and a storage medium. The method comprises the following steps: acquiring a to-be-detected image captured by a vehicle camera and a depth image of the to-be-detected image; inputting the to-be-detected image and the depth image into a trained detection model to obtain an object recognition result, wherein the trained detection model comprises an image encoder, a bird's-eye view decoder and a connecting part; the image encoder is used for determining a feature image of the to-be-detected image and a three-dimensional view cone image corresponding to the feature image and the depth image; the connecting part is used for determining a three-dimensional mapping point array; based on the relationship between the three-dimensional view cone image and the three-dimensional mapping point array and the three-dimensional mapping point array, a view cone pixel feature is extracted from the three-dimensional view cone image to obtain a three-dimensional bird's-eye view feature; and the bird's-eye view decoder is used for determining the object recognition result based on the three-dimensional bird's-eye view feature. The application solves the problem of low accuracy of the existing object recognition method based on a neural network model.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to object recognition methods, devices, vehicles, and storage media. Background Technology

[0002] Object recognition in three-dimensional space can be used in fields such as scene reconstruction, robotics, and autonomous driving. For example, in the field of autonomous driving, by identifying objects on the road (such as cars, people, or other obstacles), vehicles can be helped to perceive the environment, providing information for subsequent path planning and control decisions.

[0003] Existing technologies convert depth images into point cloud data and use neural network models to identify objects based on the point cloud data. However, they cannot simultaneously utilize both the feature information and depth information of the image. Therefore, the accuracy of existing object recognition based on neural network models is low. Summary of the Invention

[0004] This invention provides an object recognition method, device, vehicle, and storage medium to solve the problem of low accuracy in existing object recognition based on neural network models.

[0005] According to one aspect of the present invention, an object recognition method is provided, the method comprising:

[0006] Acquire the image to be detected captured by the vehicle's camera and the depth image of the image to be detected;

[0007] The image to be detected and the depth image are input into a trained detection model to obtain the object recognition result. The trained detection model includes an image encoder, a bird's-eye view decoder, and a connection part that connects the image encoder and the bird's-eye view decoder.

[0008] The image encoder is used to determine the feature image of the image to be detected, as well as the three-dimensional frustum image corresponding to the feature image and the depth image.

[0009] The connection part is used to determine the 3D mapping point matrix; determine the correspondence between the 3D frustum image and the 3D mapping point matrix, and extract frustum pixel features from the 3D frustum image based on the correspondence and the 3D mapping point matrix to obtain 3D bird's-eye view features; the 3D mapping point matrix is ​​determined based on the 3D detection mesh created in the camera's orientation direction, camera extrinsic parameters, and camera intrinsic parameters.

[0010] The bird's-eye view decoder is used to determine object recognition results based on the features of a 3D bird's-eye view.

[0011] According to another aspect of the present invention, an object recognition device is provided, the device comprising:

[0012] The acquisition module is used to acquire the image to be detected captured by the vehicle's camera and the depth image of the image to be detected;

[0013] The recognition module is used to input the image to be detected and the depth image into the trained detection model to obtain the object recognition result. The trained detection model includes an image encoder, a bird's-eye view decoder, and a connection part connecting the image encoder and the bird's-eye view decoder.

[0014] The image encoder is used to determine the feature image of the image to be detected, as well as the three-dimensional frustum image corresponding to the feature image and the depth image.

[0015] The connection part is used to determine the 3D mapping point matrix; determine the correspondence between the 3D frustum image and the 3D mapping point matrix, and extract frustum pixel features from the 3D frustum image based on the correspondence and the 3D mapping point matrix to obtain 3D bird's-eye view features; the 3D mapping point matrix is ​​determined based on the 3D detection mesh created in the camera's orientation direction, camera extrinsic parameters, and camera intrinsic parameters.

[0016] The bird's-eye view decoder is used to determine object recognition results based on the features of a 3D bird's-eye view.

[0017] According to another aspect of the present invention, a vehicle is provided, the vehicle comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the object recognition method of any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the object recognition method of any embodiment of the present invention.

[0022] The technical solution of this invention converts image features into three-dimensional bird's-eye view features through a connection part, and performs object recognition based on the three-dimensional bird's-eye view features, thereby improving the accuracy of object recognition.

[0023] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily apparent from the following description. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of an object recognition method provided according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the structure of an image encoder according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of a connection portion provided according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a bird's-eye view decoder according to an embodiment of the present invention;

[0029] Figure 5 This is a structural block diagram of an object recognition device according to an embodiment of the present invention;

[0030] Figure 6 This is a structural block diagram of another object recognition device provided according to an embodiment of the present invention;

[0031] Figure 7 This is a structural block diagram of a vehicle provided according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0034] Figure 1This is a flowchart of an object recognition method provided according to an embodiment of the present invention. This embodiment is applicable to scenarios where object recognition is performed based on a neural network model. The method can be configured in an object recognition device, which can be implemented in hardware and / or software, or it can be configured in a vehicle's processor.

[0035] like Figure 1 As shown, an object recognition method includes the following steps:

[0036] S110: Acquire the image to be detected captured by the vehicle's camera and the depth image of the image to be detected.

[0037] Understandably, the vehicle's current state could be straight, turning, reversing, or stopped. To identify objects around the vehicle, cameras are installed on the vehicle and used to capture a video of the surrounding environment. Each frame of this video serves as the image to be detected. For example, cameras are installed at the front and rear vehicle logos and the left and right rearview mirrors. After the vehicle starts, the cameras capture images of the surrounding environment to obtain the images to be detected.

[0038] A depth image reflects the distance (depth) from the camera to various points in the scene being captured. Methods for acquiring depth images can be divided into two categories: passive ranging sensing and active depth sensing. The most common method in passive ranging sensing is binocular stereo vision. This method uses two cameras spaced a certain distance apart to simultaneously acquire two images of the same scene. A stereo matching algorithm is used to find corresponding pixels in the two images, and the disparity information of these pixels is calculated based on triangulation principles. This disparity information is then converted into depth information to represent objects in the scene. Optionally, based on stereo matching algorithms, a set of images can be captured from different angles to obtain a depth image of the same scene. Active depth sensing requires devices that emit energy to acquire depth images. The advantage of this is that it ensures that the acquisition of depth images is independent of the acquisition of the images to be detected.

[0039] In one embodiment, the image to be detected is input into a trained depth image model to obtain the corresponding depth image. Specifically, for the same scene, two images to be detected are acquired from two cameras positioned at different locations on the vehicle; a stereo matching algorithm is used to determine the pixels corresponding to the same position in the two images; based on the intrinsic and extrinsic parameters of the two cameras at different positions and the pixels corresponding to the same position in the two images, the three-dimensional spatial coordinates of that position can be obtained; the three-dimensional spatial coordinates of all pixels in the image to be detected are determined to obtain the depth image corresponding to that image.

[0040] Optionally, in order to train the depth image model, a dataset of sensor data collected by the cameras of autonomous vehicles is set up. For the dataset of scene images in the dataset, the dataset of scene images is divided into a training dataset and a test dataset. Each image in the training dataset is used as a sample, and the 3D bounding box of the object in the real scene in each image and the depth image of the object are used as labels to train the depth image model, so as to obtain the trained depth image model.

[0041] Furthermore, before entering the model training, the images can be preprocessed to augment the image data using data augmentation techniques, such as random horizontal flipping, random cropping, and / or random rotation. The advantage of this is that, without substantially increasing the image data, limited data can generate value equivalent to more data.

[0042] Optionally, the image to be detected can be converted into a corresponding depth image using an existing image processing model that has reached the current state of the arts (Sota). This embodiment does not specifically limit this.

[0043] S120. Input the image to be detected and the depth image into the trained detection model to obtain the object recognition result. The trained detection model includes an image encoder, a bird's-eye view decoder, and a connection part connecting the image encoder and the bird's-eye view decoder.

[0044] The image encoder is used to determine the feature image of the image to be detected, as well as the 3D frustum image corresponding to the feature image and the depth image; the connection part is used to determine the 3D mapping matrix; determine the correspondence between the 3D frustum image and the 3D mapping matrix, and extract the frustum pixel features from the 3D frustum image based on the correspondence and the 3D mapping matrix to obtain the 3D bird's-eye view features; the 3D mapping matrix is ​​determined based on the 3D detection grid created in the camera's orientation direction, camera extrinsic parameters, and camera intrinsic parameters; the bird's-eye view decoder is used to determine the object recognition result based on the 3D bird's-eye view features.

[0045] The view frustum is the shape of the area that a vehicle's camera can see and render. It's a cube-shaped structure that resembles a pyramid, with the top of the frustum cut off so that the top plane is parallel to the bottom plane. Within the view frustum, any object falling outside its boundaries is invisible to the camera. In other words, any point in the image captured by the camera actually corresponds to a line in world space, and only one point on that line is visible in the view frustum image; everything behind that point is occluded.

[0046] Specifically, the depth image is input into an image encoder to extract its feature data. This determines the frustum range of the vehicle camera when capturing the corresponding image to be detected, thus obtaining a 3D frustum image corresponding to the depth image. For example, the depth image is input into the image encoder (see...). Figure 2 First, features are extracted using ResNet. The last layer feature map of size H×W is upsampled and added to the previous layer feature map. Then, a 1×1 convolution kernel is used to obtain a 3D frustum image with downsampled by d times and C channels.

[0047] The connection part establishes a correspondence between the 3D frustum image and the 3D mapping point matrix using the intrinsic and extrinsic parameters of the camera and the depth information provided by the depth image. This ensures that every feature in the 3D frustum image corresponds to a depth position, achieving a combination of depth information and image features. Based on the 3D mapping point matrix and this correspondence, the pixel points in the 3D frustum image corresponding to each point in the 3D mapping point matrix are determined. Using the pixel points corresponding to each point in the 3D frustum image and the 3D mapping point matrix, frustum pixel features are extracted from the 3D frustum image to obtain the 3D bird's-eye view features. For example, the connection part (see...) Figure 3 A coordinate grid is generated in front of the front-facing camera as a 3D detection grid. The center point coordinates of each grid voxel are first projected onto the front-facing camera's coordinate system through camera extrinsic parameters, and then mapped onto the pixel coordinate system through camera intrinsic parameters. During the mapping process, the depth value is saved as a new variable. After mapping, the 2D mapped points on the pixel are concatenated with the depth values ​​to obtain a 3D mapped point matrix. Using trilinear interpolation, each 3D mapped point samples a feature from the previously generated 3D frustum image. The sampled feature is placed in the corresponding grid voxel, ultimately generating a 3D bird's eye view (BEV) feature with a resolution of 376×280×61.

[0048] The 3D bird's-eye view features are input into the bird's-eye view decoder for object recognition, and the output includes the object type, quantity, and location, yielding the object recognition result. For example, the bird's-eye view features are input into the bird's-eye view decoder (see...). Figure 4 This process generates bird's-eye view features on four horizontal planes. The resolution of the 3D mapping point array is adjusted to match the resolution of the 3D frustum image (188×140×128). Then, the 2D bird's-eye view features on all horizontal planes are stacked to obtain the current 3D bird's-eye view features. Finally, different 3D target detection heads are set for different categories. These detection heads are used to perform object recognition based on the current 3D bird's-eye view features to obtain the object recognition results.

[0049] Furthermore, before training the detection model, firstly, the framework of the detection model needs to be determined. OpenPCDet, Det3D, or Paddle3D can be used as the framework; this embodiment chooses OpenPCDet. Then, for the detection model framework, configuration files and training scripts required for computation are written. When training the detection model, the training scripts in OpenPCDet can be run directly, specifying the pre-written configuration file and the number of Graphics Processing Units (GPUs) required for computation. Finally, the test scripts in OpenPCDet are run according to the configuration file and the required number of GPUs to train the detection model. Furthermore, the performance of this detection model in an in-vehicle environment can be evaluated using the Computer Vision Algorithm Evaluation Dataset for Autonomous Driving Scenarios (KITTI).

[0050] In one embodiment, the image encoder is specifically used to: extract a feature map of the image to be detected; downsample the depth image to obtain a current depth image, the size of which is the same as the feature map; convert the depth value corresponding to each pixel in the feature map in the current depth map into a one-hot vector; and perform a dot product between the one-hot vector and the feature map to obtain a frustum feature map. First, a 101-layer residual network is used to extract features from the image to be detected, resulting in a feature map. The depth image of the image to be detected is downsampled using bilinear interpolation to obtain a depth image of the same size as the feature map, which serves as the current depth image. Second, the depth value of each pixel in the feature map is converted into a one-hot vector, which extends the values ​​of discrete features to Euclidean space, making the values ​​of discrete features correspond to points in Euclidean space, thus making the distance calculation between features more reasonable and improving the accuracy of object recognition. Then, the one-hot vector is dot-producted with the feature map of the image to be detected to obtain a frustum feature map, ensuring that each feature has a corresponding depth position.

[0051] Furthermore, the three-dimensional mapping lattice is determined through the following steps:

[0052] Step a1: Create a 3D detection mesh in the vehicle body coordinate system in the direction the camera is facing.

[0053] The vehicle body coordinate system is a coordinate system established with the vehicle itself as a reference, and the center point of the rear axle or the center of mass of the vehicle is selected as the origin.

[0054] Specifically, a three-dimensional coordinate system is created with the camera's position as the origin and the camera's orientation as one of the coordinate axes. This system serves as the three-dimensional detection mesh, located within the vehicle body coordinate system. For example, in the vehicle body coordinate system, the camera is positioned at the front of the vehicle's emblem, facing directly forward. A three-dimensional space is defined as the three-dimensional detection mesh. The coordinate range of this mesh in the vehicle body coordinate system (unit: meters) is: length 2-46.8, width -30-30, height -3-1. This mesh contains 280×375×25 voxels, each voxel having a length, width, and height of 0.16 meters.

[0055] Step a2: Project the 3D detection mesh onto the vehicle's camera coordinate system based on the camera extrinsic parameters to obtain a 2D mapped point matrix.

[0056] The camera coordinate system has its origin at the camera's optical center. Its x and y axes are parallel to the x and y axes of the plane coordinate system of the image to be detected, while its z-axis is parallel to the camera's optical axis and perpendicular to the plane of the image to be detected. Therefore, by using the camera's extrinsic parameter matrix, the 3D detection mesh can be transformed from the vehicle body coordinate system to the camera coordinate system.

[0057] For example, the coordinates of the center point of each voxel in the 3D detection mesh in the vehicle body coordinate system are projected onto the camera coordinate system using camera extrinsic parameters to obtain a 2D mapped point matrix. The advantage of doing this is that the features extracted by the image encoder are projected onto the bird's-eye view based on the camera parameters, thus improving the resolution of the features at the bird's-eye view.

[0058] Step a3: Based on the camera intrinsic parameters, map the two-dimensional mapping point matrix and the depth values ​​corresponding to each point in the two-dimensional mapping point matrix to the vehicle coordinate system to obtain the three-dimensional mapping point matrix.

[0059] Among them, the camera intrinsic parameters are used to realize the projection from the camera coordinate system to the planar coordinate system of the image to be detected.

[0060] Specifically, the two-dimensional mapping point array is mapped from the camera coordinate system to the planar coordinate system of the image to be detected using camera intrinsic parameters, resulting in a two-dimensional mapping point array containing image pixels. During the mapping process, the depth value corresponding to each point in the two-dimensional mapping point array is saved as a new variable. The two-dimensional mapping point array containing image pixels is then concatenated with the depth value to obtain a three-dimensional mapping point array.

[0061] In one embodiment, before determining the correspondence between the view frustum image and the 3D mapping point array, if the resolution of the 3D mapping point array is less than the resolution of the 3D view frustum image, then trilinear interpolation is performed on the mapping point array to make the resolution of the mapping point array the same as the resolution of the 3D view frustum image; if the resolution of the mapping point array is greater than the resolution of the 3D view frustum image, then trilinear interpolation is performed on the 3D view frustum image to make the resolution of the mapping point array the same as the resolution of the 3D view frustum image.

[0062] Specifically, each point in the 3D mapping point array needs to be upsampled in the 3D frustum image to obtain a feature, so the resolution of the 3D mapping point array is the same as the resolution of the 3D frustum image; that is, when the resolution of the 3D mapping point array is less than or greater than the resolution of the 3D frustum image, the resolution of the 3D mapping point array needs to be the same as the resolution of the 3D frustum image.

[0063] Understandably, during image resizing, some new pixels are generated, requiring interpolation to calculate their pixel values. This means fitting known pixel information to unknown pixel information. For example, in 3D space, linear interpolation is performed on the sampling point using pixels surrounding it, achieving upsampling. Alternatively, for sampling points in the mapped point matrix, trilinear interpolation is used to determine the pixel value of each sampling point by performing 3D linear interpolation based on the pixel values ​​at the eight vertices of the cubic mesh cell of the voxel containing that sampling point. Furthermore, the result of trilinear interpolation is independent of the interpolation order of the three axes of the 3D detection mesh; that is, regardless of whether interpolation is performed first along the x-axis, y-axis, or z-axis, the same value will be produced.

[0064] In one embodiment, based on the correspondence, the view frustum pixel features corresponding to the center points of each voxel in the 3D mapping point matrix are determined in the 3D view frustum image to obtain the 3D bird's-eye view features corresponding to the 3D mapping point matrix. Specifically, for the center point of each voxel in the 3D mapping point matrix, according to the correspondence between the 3D view frustum image and the 3D mapping point matrix, a feature is sampled from the previously generated image view frustum features for each 3D mapping point to obtain the view frustum pixel features corresponding to the center points of each voxel in the 3D view frustum image. The sampled features are then placed in the corresponding voxel to obtain the 3D bird's-eye view features corresponding to the 3D mapping point matrix.

[0065] In one embodiment, the bird's-eye view decoder is used to: determine the two-dimensional bird's-eye view features of the three-dimensional bird's-eye view features on each horizontal plane; stack all the two-dimensional bird's-eye view features on all horizontal planes along the vehicle height direction to obtain the current three-dimensional bird's-eye view features; and determine the object recognition result based on the current three-dimensional bird's-eye view features. Specifically, first, the bird's-eye view feature decoder determines the two-dimensional bird's-eye view features corresponding to each horizontal grid in the three-dimensional detection grid; then, it stitches all the two-dimensional bird's-eye view features on all horizontal planes along the vehicle height direction to obtain the three-dimensional bird's-eye view features, which are used as the current three-dimensional bird's-eye view features; finally, it inputs the current three-dimensional bird's-eye view features into a trained detection model to perform object recognition and obtain the object recognition result. Optionally, the 3D bird's-eye view features are obtained, and then the channel dimension is reduced to 64 using a 1×1 convolutional kernel, resulting in a 64-channel bird's-eye view feature. Further, multiple convolutional layers, batch normalization (BN) layers, activation layers, and upsampling fusion are used to obtain aggregated bird's-eye view features, which serve as the current 3D bird's-eye view features. Finally, based on the current 3D bird's-eye view features, a prediction network is developed to predict the object's category, score, and regress the 3D detection box. The advantage of this approach is that it allows the feature map to possess multi-scale information.

[0066] The specific steps of upsampling fusion are as follows: First, the bird's-eye view features of the last layer are upsampled by a factor of 2 using a 3×3 convolution kernel; second, they are added to the features of the previous layer in pixels; then, the above process is repeated twice more for upsampling until the output bird's-eye view features have the same resolution as the original input image. The convolution operation still uses a 3×3 convolution kernel, and each upsampling operation upsamples the height and width of the features to twice the input of that operation, with a unified output channel for each upsampling. This process can be described by the following formula:

[0067] X u ′ p =C 1×1 (C 3×3 (X))

[0068] X = X u ′ p +C 1×1 (X)

[0069] Among them, C 3×3 and C 1×1 These represent convolutions with kernel sizes of 3×3 and 1×1, respectively, where X represents the features before upsampling. u ′ p These are the features after upsampling.

[0070] The technical solution of this embodiment combines the depth information of the depth image with the characteristics of the bird's-eye view, and uses a trained detection model to perform object recognition, thereby improving the accuracy of object recognition.

[0071] Figure 5 This is a structural block diagram of an object recognition device according to an embodiment of the present invention. The device can be implemented in hardware and / or software and integrated into the vehicle's processor.

[0072] like Figure 5 As shown, the object recognition device includes:

[0073] The acquisition module 501 is used to acquire the image to be detected captured by the vehicle camera and the depth image of the image to be detected;

[0074] The recognition module 502 is used to input the image to be detected and the depth image into the trained detection model to obtain the object recognition result. The trained detection model includes an image encoder, a bird's-eye view decoder, and a connection part connecting the image encoder and the bird's-eye view decoder.

[0075] The image encoder is used to determine the feature image of the image to be detected, as well as the three-dimensional frustum image corresponding to the feature image and the depth image.

[0076] The connection part is used to determine the 3D mapping point matrix; determine the correspondence between the 3D frustum image and the 3D mapping point matrix, and extract frustum pixel features from the 3D frustum image based on the correspondence and the 3D mapping point matrix to obtain 3D bird's-eye view features; the 3D mapping point matrix is ​​determined based on the 3D detection mesh created in the camera's orientation direction, camera extrinsic parameters, and camera intrinsic parameters.

[0077] The bird's-eye view decoder is used to determine object recognition results based on the features of a 3D bird's-eye view.

[0078] Optional, such as Figure 6 As shown, the trained detection model 5021 includes an image encoder 50211, a connection part 20212, and a bird's-eye view feature decoder 50213.

[0079] The connecting part 50212 is used for:

[0080] Create a 3D detection mesh in the vehicle body coordinate system in the direction the camera is facing;

[0081] Based on camera extrinsic parameters, a 3D detection mesh is projected onto the vehicle's camera coordinate system to obtain a 2D mapped point matrix;

[0082] Based on camera intrinsic parameters, the two-dimensional mapping point matrix and the depth values ​​corresponding to each point in the two-dimensional mapping point matrix are mapped to the vehicle coordinate system to obtain the three-dimensional mapping point matrix.

[0083] Optionally, module 501 is specifically used for:

[0084] The image to be detected is input into a trained depth image model to obtain the corresponding depth image.

[0085] Optionally, the image encoder 50211 is specifically used for:

[0086] Extract feature maps from the image to be detected;

[0087] The depth image is downsampled to obtain the current depth image, and the size of the current depth image is the same as the size of the feature map;

[0088] Convert the depth value of each pixel in the feature map into a one-hot vector in the current depth map;

[0089] The one-hot vector is dot-producted with the feature map to obtain the view frustum feature map.

[0090] Optionally, the identification module 502 is specifically used for:

[0091] If the resolution of the 3D mapping point array is less than the resolution of the 3D frustum image, then trilinear interpolation is performed on the 3D mapping point array to make the resolution of the 3D mapping point array the same as the resolution of the 3D frustum image.

[0092] If the resolution of the 3D mapping point array is greater than the resolution of the 3D frustum image, then trilinear interpolation is performed on the 3D frustum image to make the resolution of the 3D mapping point array the same as the resolution of the 3D frustum image.

[0093] Optionally, the connection portion 50212 is used for:

[0094] Based on the correspondence, the frustum pixel features corresponding to the center points of each voxel in the 3D mapping matrix are determined in the 3D frustum image, so as to obtain the 3D bird's-eye view features corresponding to the 3D mapping matrix.

[0095] Optionally, the bird's-eye view decoder 50213 is used for:

[0096] Determine the 2D bird's-eye view features of the 3D bird's-eye view features on each horizontal plane;

[0097] All two-dimensional bird's-eye view features on the horizontal plane are stacked in the vehicle height direction to obtain the current three-dimensional bird's-eye view features;

[0098] The object recognition result is determined based on the features of the current 3D bird's-eye view.

[0099] The technical solution of this embodiment, through the cooperation of various modules, inputs the images and depth images captured by the camera into the trained detection model to obtain object recognition results, thereby improving the accuracy of object recognition.

[0100] The object recognition device provided in the embodiments of the present invention can execute the object recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0101] Figure 7 This is a structural block diagram of a vehicle according to an embodiment of the present invention, such as... Figure 7 As shown, vehicle 10 includes at least one processor 11 and a memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer program stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of vehicle 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. Input / output (I / O) interface 15 is also connected to bus 14.

[0102] Multiple components in vehicle 10 are connected to I / O interface 15, including: input unit 16, such as a keyboard or mouse; output unit 17, such as various types of displays or speakers; storage unit 18, such as a disk or optical disk; and communication unit 19, such as a network card, modem, or wireless transceiver. Communication unit 19 allows vehicle 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), any suitable processor, controller, or microcontroller, etc. Processor 11 performs the various methods and processes described above, such as object recognition methods.

[0104] In some embodiments, the object recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on vehicle 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the object recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the object recognition method by any other suitable means (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on the machine or partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0108] To provide interaction with the user, the systems and techniques described herein can be implemented on vehicle 10, which has: a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An object recognition method, characterized in that, include: Acquire the image to be detected captured by the vehicle's camera and the depth image of the image to be detected; The image to be detected and the depth image are input into a trained detection model to obtain object recognition results. The trained detection model includes an image encoder, a bird's-eye view decoder, and a connection part connecting the image encoder and the bird's-eye view decoder. The image encoder is used to determine the feature image of the image to be detected, and the three-dimensional frustum image corresponding to the feature image and the depth image; The connection portion is used to determine the three-dimensional mapping point matrix; determine the correspondence between the three-dimensional frustum image and the three-dimensional mapping point matrix, and extract frustum pixel features from the three-dimensional frustum image based on the correspondence and the three-dimensional mapping point matrix to obtain three-dimensional bird's-eye view features; the three-dimensional mapping point matrix is ​​determined based on the three-dimensional detection grid created in the camera's orientation direction, camera extrinsic parameters, and camera intrinsic parameters. The bird's-eye view decoder is used to determine the object recognition result based on the features of the three-dimensional bird's-eye view; The three-dimensional mapping point matrix is ​​determined through the following steps: A three-dimensional detection mesh located in the vehicle body coordinate system is created in the orientation direction of the camera; The three-dimensional detection mesh is projected onto the vehicle's camera coordinate system based on the camera's extrinsic parameters to obtain a two-dimensional mapping matrix; Based on the camera intrinsic parameters, the two-dimensional mapping point matrix and the depth value corresponding to each point in the two-dimensional mapping point matrix are mapped to the vehicle coordinate system to obtain the three-dimensional mapping point matrix; Specifically, the image encoder is used for: Extract the feature map of the image to be detected; The depth image is downsampled to obtain a current depth image, the size of which is the same as the size of the feature map; Convert the depth value of each pixel in the feature map in the current depth image into a one-hot vector; The one-hot vector is dot-producted with the feature map to obtain the view frustum feature map; Determining the depth image of the image to be detected includes: inputting the image to be detected into a trained depth image model to obtain the corresponding depth image.

2. The method according to claim 1, characterized in that, Before determining the correspondence between the three-dimensional frustum image and the three-dimensional mapping point matrix, the method further includes: If the resolution of the 3D mapping point array is less than the resolution of the 3D frustum image, then trilinear interpolation is performed on the 3D mapping point array to make the resolution of the 3D mapping point array the same as the resolution of the 3D frustum image. If the resolution of the 3D mapping point array is greater than the resolution of the 3D frustum image, then the 3D frustum image is subjected to trilinear interpolation so that the resolution of the 3D mapping point array is the same as the resolution of the 3D frustum image.

3. The method according to claim 1, characterized in that, The step of extracting frustum pixel features from the 3D frustum image based on the correspondence and the 3D mapping matrix to obtain 3D bird's-eye view features includes: Based on the correspondence, the center point of each voxel in the three-dimensional mapping matrix is ​​determined to correspond to the frustum pixel features in the three-dimensional frustum image, so as to obtain the three-dimensional bird's-eye view features corresponding to the three-dimensional mapping matrix.

4. The method according to claim 1, characterized in that, The bird's-eye view decoder is used for: Determine the two-dimensional bird's-eye view features of the three-dimensional bird's-eye view features on each horizontal plane; All two-dimensional bird's-eye view features on the horizontal plane are stacked in the vehicle height direction to obtain the current three-dimensional bird's-eye view features; The object recognition result is determined based on the features of the current 3D bird's-eye view.

5. An object recognition device, characterized in that, include: The acquisition module is used to acquire the image to be detected captured by the vehicle camera and the depth image of the image to be detected; The recognition module is used to input the image to be detected and the depth image into a trained detection model to obtain an object recognition result, wherein the trained detection model includes an image encoder, a bird's-eye view decoder, and a connection part connecting the image encoder and the bird's-eye view decoder; The image encoder is used to determine the feature image of the image to be detected, and the three-dimensional frustum image corresponding to the feature image and the depth image; The connection portion is used to determine the three-dimensional mapping point matrix; determine the correspondence between the three-dimensional frustum image and the three-dimensional mapping point matrix, and extract frustum pixel features from the three-dimensional frustum image based on the correspondence and the three-dimensional mapping point matrix to obtain three-dimensional bird's-eye view features; the three-dimensional mapping point matrix is ​​determined based on the three-dimensional detection grid created in the camera's orientation direction, camera extrinsic parameters, and camera intrinsic parameters. The bird's-eye view decoder is used to determine the object recognition result based on the features of the three-dimensional bird's-eye view; The connecting portion is used for: Create a 3D detection mesh in the vehicle body coordinate system in the direction the camera is facing; Based on camera extrinsic parameters, a 3D detection mesh is projected onto the vehicle's camera coordinate system to obtain a 2D mapped point matrix; Based on the camera intrinsic parameters, the two-dimensional mapping point matrix and the depth value corresponding to each point in the two-dimensional mapping point matrix are mapped to the vehicle coordinate system to obtain the three-dimensional mapping point matrix. Specifically, the image encoder is used for: Extract feature maps from the image to be detected; The depth image is downsampled to obtain the current depth image, and the size of the current depth image is the same as the size of the feature map; Convert the depth value of each pixel in the feature map into a one-hot vector in the current depth map; The one-hot vector is dot-producted with the feature map to obtain the view frustum feature map; Specifically, the acquisition module is used for: The image to be detected is input into a trained depth image model to obtain the corresponding depth image.

6. A vehicle, characterized in that, The vehicles include: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the object recognition method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the object recognition method according to any one of claims 1-4.

Images