Target detection method, electronic device and storage medium

Abstract

An embodiment of the present application discloses a target detection method, an electronic device, and a storage medium. The method includes: extracting semantic features of images in an image group to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras; estimating depth information on the two-dimensional semantic feature map to obtain a two-dimensional depth feature map; performing feature compression on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and performing feature compression on the two-dimensional depth feature map to obtain a one-dimensional depth feature map; generating a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map; and performing target detection based on the two-dimensional bird's-eye view feature map.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a target detection method, electronic device, and storage medium. Background Art

[0002] With the continuous development of machine vision technology, 3D object detection technology is widely used in the fields of autonomous driving and robotics. Taking autonomous driving as an example, autonomous vehicles need to identify not only the type of obstacles but also their precise location and orientation to provide information to the planning and control module to plan a reasonable route. 3D object detection aims to use data from multiple sensors such as cameras, radar, and lidar to enable autonomous vehicles to detect objects such as vehicles, pedestrians, and obstacles, thereby ensuring driving safety.

[0003] Currently, multi-camera surround-view 3D object detection technology has experienced rapid development due to its high performance and support for multiple tasks such as object detection, object segmentation, and lane detection. Compared to traditional detectors, such as FCOS3D, this type of 3D object detection technology primarily improves upon the conversion of image features from the camera's perspective to a bird's-eye view, while also integrating multiple features simultaneously.

[0004] Related technologies use data-driven conversion schemes, such as BEVFormer, to convert image features from a camera perspective to a bird's-eye view. This implicit feature conversion is achieved by training neural network models, such as the Transformer model. However, these methods use large neural network models, require a large amount of training data, and occupy a large amount of video memory, making them expensive to deploy on automotive chipsets. Summary of the Invention

[0005] The embodiments of the present application provide a target detection method, an electronic device, and a storage medium to solve the technical problem of high deployment cost of 3D target detection technology on vehicle-mounted chips.

[0006] According to a first aspect of the present application, a target detection method is disclosed, the method comprising:

[0007] Extracting semantic features of images in an image group to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras;

[0008] Estimating depth information of the two-dimensional semantic feature map to obtain a two-dimensional depth feature map;

[0009] In the height dimension of the feature map, feature compression is performed on the two-dimensional semantic feature map to obtain a one-dimensional semantic feature map, and feature compression is performed on the two-dimensional depth feature map to obtain a one-dimensional depth feature map;

[0010] Generating a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map;

[0011] Target detection is performed based on the two-dimensional bird's-eye view feature map.

[0012] According to a second aspect of the present application, a target detection device is disclosed, comprising:

[0013] an extraction module, configured to extract semantic features of images in an image group to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras;

[0014] An estimation module, configured to estimate depth information of the two-dimensional semantic feature map to obtain a two-dimensional depth feature map;

[0015] A compression module is used to perform feature compression on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and to perform feature compression on the two-dimensional depth feature map to obtain a one-dimensional depth feature map;

[0016] A generating module, configured to generate a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map;

[0017] A detection module is used to perform target detection based on the two-dimensional bird's-eye view feature map.

[0018] According to a third aspect of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the target detection method as in the first aspect.

[0019] According to a fourth aspect of the present application, a computer-readable storage medium is disclosed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the target detection method as in the first aspect is implemented.

[0020] According to a fifth aspect of the present application, a computer program product is disclosed, comprising a computer program / instruction, which implements the target detection method in the first aspect when executed by a processor.

[0021] In an embodiment of the present application, semantic features of images in an image group are extracted to obtain a two-dimensional semantic feature map, depth information is estimated on the two-dimensional semantic feature map to obtain a two-dimensional depth feature map, feature compression is performed on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and feature compression is performed on the two-dimensional depth feature map to obtain a one-dimensional depth feature map; based on the one-dimensional semantic feature map and the one-dimensional depth feature map, a two-dimensional bird's-eye view feature map is generated, and target detection is performed based on the two-dimensional bird's-eye view feature map.

[0022] Compared with the related art, in the embodiment of the present application, in the process of converting image features from the camera perspective to the bird's-eye view, considering that there is information redundancy in the two-dimensional image in the height dimension, the two-dimensional image features and two-dimensional depth information under the camera perspective can be compressed into one-dimensional image features and one-dimensional depth information in the height dimension, and a two-dimensional bird's-eye view feature map for 3D target detection is generated based on the one-dimensional image features and one-dimensional depth information. Since the conversion process only involves compression processing of redundant information in the height dimension, the information in the width dimension will not be compressed, and converting the two-dimensional information into one-dimensional information can greatly reduce the amount of data in the conversion process. Therefore, the occupancy of the video memory can be reduced while ensuring the accuracy of the feature conversion results, thereby reducing the cost of deployment on the vehicle chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is one of the flow charts of the target detection method provided in the embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the feature compression process provided in an embodiment of the present application;

[0025] Figure 3 This is the second flow chart of the target detection method provided in the embodiment of the present application;

[0026] Figure 4 2. It is a schematic diagram of a process for generating a two-dimensional bird's-eye view feature map provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a target detection system provided in an embodiment of the present application;

[0028] Figure 6 This is the third flow chart of the target detection method provided in the embodiment of the present application;

[0029] Figure 7 is a structural diagram of a target detection device provided in an embodiment of the present application;

[0030] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0033] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, techniques, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, aims to enable machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0034] Taking autonomous driving as an example, 3D object detection technology has been widely used in this field. Among various sensor solutions, camera-based 3D object detection has gradually attracted considerable attention due to its low cost. Among camera-based solutions, multi-camera (i.e., multi-camera) bird's-eye view (BEV) detection solutions (referred to as surround-view BEV detection) have recently experienced rapid development due to their high performance and support for the fusion of multiple tasks such as object detection, object segmentation, and lane detection. Compared to traditional detectors such as FCOS3D, this type of object detection technology primarily improves upon the conversion of image features from the camera's perspective to a bird's-eye view, with multi-camera feature fusion also performed simultaneously during this step.

[0035] In related technologies, there are two main types of feature conversion schemes for surround view BEV detection technology:

[0036] One type of method is a data-driven conversion solution represented by BEVFormer, which achieves implicit feature conversion by training neural network models such as the Transformer model. Since this type of method uses a large neural network model, it requires a large amount of training data and occupies a large amount of video memory space, so the deployment cost on automotive chips is relatively high.

[0037] Another approach uses predicted depth information to project 2D semantic features into 3D space. These 3D features are then integrated and extracted through sampling or pooling to produce 2D bird's-eye-view features. Representative work includes BEVDet. While these approaches reduce computational complexity, the operations used in the 3D feature extraction and integration process are often complex, requiring significant graphics memory usage, resulting in high deployment costs on automotive chipsets.

[0038] It can be seen that the existing surround view 3D object detection algorithms in related technologies either introduce complex models or operators that are not conducive to deployment during the perspective conversion process, which will occupy a large amount of video memory space and make it impossible to deploy them in mass production on a variety of automotive chips.

[0039] In order to solve the above technical problems, the embodiments of the present application provide a target detection method, an electronic device and a storage medium, which realize perspective conversion and multi-view feature fusion through a simple and efficient method, so that the surround-view BEV detection technology can achieve high efficiency and easy deployment without reducing performance, wherein the multi-view feature fusion is completed during the perspective conversion process.

[0040] To facilitate understanding, some concepts involved in the embodiments of this application are first introduced.

[0041] Bird's Eye View (BEV), also known as God's perspective, is a perspective or coordinate system used to describe the perceived world. BEV is also used to refer to an end-to-end technology in the field of computer vision that uses neural networks to convert visual information from image space to BEV space.

[0042] The intrinsic parameter information of a camera is a matrix used to convert from the camera coordinate system to the pixel coordinate system. Generally speaking, the intrinsic parameters of a camera are determined after leaving the factory and can be manually calculated through camera calibration.

[0043] The camera's extrinsic parameter information is determined by the camera's pose (that is, placement). Different poses have different extrinsic parameter information, which is used to convert from the world coordinate system to the camera coordinate system.

[0044] Taking the driving scenario as an example, the multiple cameras in the embodiments of the present application are generally arranged around the vehicle body, for example, six cameras are arranged around the vehicle body, and therefore can also be referred to as surround view cameras, or surround view cameras. The external parameter information of each of the multiple cameras is different, and the internal parameter information can be the same or different.

[0045] Depth information estimation refers to obtaining the distance information from each point in the scene in the image to the camera (also called depth information).

[0046] Next, a target detection method provided in an embodiment of the present application is introduced.

[0047] Figure 1 This is one of the flow charts of the target detection method provided in the embodiment of the present application, such as Figure 1 As shown, the method may include the following steps: step 101, step 102, step 103, step 104 and step 105;

[0048] In step 101, semantic features of images in an image group are extracted to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras.

[0049] In the embodiment of the present application, one of the common image classification networks such as AlexNet, ResNet, VGG, or MobileNet can be used to extract semantic features of images in the image group, wherein the semantic features can be high-level semantic features. The two-dimensional semantic feature map can be represented by a tensor.

[0050] In the embodiment of the present application, each image in the image group can be an image in RGB format, or an image in other formats, such as an image in YUV format, which is not limited in the present application.

[0051] In an example, an image group contains six images captured by six cameras. For ease of description, the shape of the image group is expressed in the form of a tensor of 6*3*256*704, where 6 represents the number of images, 3 represents the number of color channels, 256 represents the image height, and 704 represents the image width.

[0052] Then, the semantic features of the images in the image group are extracted to obtain a two-dimensional semantic feature map. The shape of the two-dimensional semantic feature map is 6*256*16*44, where 6 represents the number of images, 256 represents the number of feature channels, 16 represents the height of the feature map (the height after downsampling 16 times the original image height), and 44 represents the width of the feature map (the width after downsampling 16 times the original image width).

[0053] In step 102, depth information is estimated on the two-dimensional semantic feature map to obtain a two-dimensional depth feature map.

[0054] In the embodiment of the present application, the size of the two-dimensional semantic feature map can be the same as the size of the two-dimensional depth feature map. In this case, depth information can be estimated for each point in the two-dimensional semantic feature map to obtain a two-dimensional depth feature map. Alternatively, the size of the two-dimensional semantic feature map can be different from the size of the two-dimensional depth feature map.

[0055] In some embodiments, in order to improve the speed and computational complexity of depth estimation, depth information estimation may be performed based only on a two-dimensional semantic feature map to obtain a two-dimensional depth feature map.

[0056] In some embodiments, in order to improve the accuracy of depth estimation, the physical depth information of the corresponding position can be estimated from the two-dimensional semantic feature map with the help of camera intrinsic parameter information. Accordingly, the above step 102 includes the following steps: step 1021;

[0057] In step 1021, the intrinsic reference information of multiple cameras is used as auxiliary reference information to estimate the corresponding depth information from the two-dimensional semantic feature map to obtain a two-dimensional depth feature map.

[0058] In an embodiment of the present application, one or more convolutional layers and fully connected operators may be used to construct a depth estimation network, and the processing of step 1021 may be implemented through the constructed depth estimation network.

[0059] It can be seen that in the embodiments of the present application, depth information can be estimated from the two-dimensional semantic feature map in a variety of ways to obtain a two-dimensional depth feature map. Users can choose which depth information estimation method to use based on actual conditions and their own needs to meet their diverse needs.

[0060] In an example, the shape of a two-dimensional semantic feature map is 6*256*16*44. Depth information is estimated on the two-dimensional semantic feature map to obtain a two-dimensional depth feature map. The shape of the two-dimensional depth feature map is 6*112*16*44, where 6 represents the number of images, 112 represents the number of depth channels in the image, 16 represents the height of the feature map, and 44 represents the width of the feature map.

[0061] In step 103, feature compression is performed on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and feature compression is performed on the two-dimensional depth feature map to obtain a one-dimensional depth feature map.

[0062] In an embodiment of the present application, compared with a two-dimensional semantic feature map, a one-dimensional semantic feature map becomes one-dimensional in the height dimension; similarly, compared with a two-dimensional depth feature map, a one-dimensional depth feature map becomes one-dimensional in the height dimension.

[0063] Considering that in the autonomous driving scenario, it is impossible for multiple objects to exist at different heights in the same direction in an image. For example, if there is a car in the front left, there cannot be another car on top of this car, that is, it is impossible for two cars to be stacked on top of each other. Therefore, it is believed that there is information redundancy in the image at height. Therefore, the height dimension can be reduced from two dimensions to one dimension.

[0064] In some embodiments, there is provided Figure 2 The feature compression process shown, Figure 2 "H" represents the height direction, "W" represents the width direction, "C" and "D" represent the color direction and the depth direction respectively. Accordingly, the above step 103 includes the following steps: step 1031 and step 1032;

[0065] In step 1031, maximum pooling is performed on the two-dimensional semantic feature map in the height dimension to obtain a one-dimensional semantic feature map.

[0066] In the embodiment of the present application, a convolution operation can also be used to obtain a one-dimensional semantic feature map.

[0067] In step 1032, the weight of the depth information corresponding to each point in the two-dimensional semantic feature map is predicted, the depth information of each point in the two-dimensional depth feature map is multiplied by the weight of the corresponding position, and the sum is taken in the height dimension to obtain a one-dimensional depth feature map.

[0068] In an embodiment of the present application, when predicting the weight of depth information, the weight of the depth information can be obtained by convolving the two-dimensional semantic feature map through an existing convolutional network to reduce the feature dimension to 1.

[0069] Considering that the core point of 3D target detection is the detection of target objects in the image rather than the detection of background objects, and the weight value of the depth of the target object in the two-dimensional semantic feature map is high, and the weight value of the depth of the background object is low, therefore, in an embodiment of the present application, the depth information of each point in the two-dimensional depth feature map can be multiplied by the weight of the corresponding position, and the sum is taken in the height dimension to obtain a one-dimensional depth feature map. The one-dimensional depth feature map only contains the depth of the target object and does not contain the depth of the background object. The one-dimensional depth feature map after dimensionality reduction will not cause the loss of key information.

[0070] In an example, the shape of the two-dimensional semantic feature map is 6*256*16*44, and the shape of the two-dimensional depth feature map is 6*112*16*44. The two-dimensional semantic feature map is convolved to reduce the feature dimension to 1 to obtain the depth information weight, whose shape is 6*1*16*44. The two-dimensional depth feature map is multiplied by the weight and summed in the height dimension to obtain a one-dimensional depth feature map, whose shape is 6*112*44. It can be seen that the data of the height dimension in the one-dimensional depth feature map is gone; the two-dimensional semantic feature map is maximum pooled in the height dimension to obtain a one-dimensional semantic feature map, whose shape is 6*256*44. It can be seen that the data of the height dimension in the one-dimensional semantic feature map is also gone.

[0071] In an embodiment of the present application, compressing a two-dimensional semantic feature map into a one-dimensional semantic feature map and compressing a two-dimensional depth feature map into a one-dimensional depth feature map can greatly reduce the amount of data in the conversion process and reduce the occupancy of video memory.

[0072] In step 104, a two-dimensional bird's-eye view feature map is generated based on the one-dimensional semantic feature map and the one-dimensional depth feature map.

[0073] In the embodiments of the present application, to reduce computational complexity and video memory usage, a two-dimensional bird's-eye view feature map can be generated based on a one-dimensional semantic feature map and a one-dimensional depth feature map through matrix operations. Alternatively, to reduce R&D costs, shorten construction time, and increase technology reuse, an existing computing network or module can be used to process a one-dimensional semantic feature map and a one-dimensional depth feature map to generate a two-dimensional bird's-eye view feature map.

[0074] As can be seen, in the embodiments of the present application, a two-dimensional bird's-eye view feature map can be generated based on a one-dimensional semantic feature map and a one-dimensional depth feature map in a variety of ways. Users can choose which conversion method to use based on actual conditions and their own needs to meet their diverse needs.

[0075] In step 105, target detection is performed based on the two-dimensional bird's-eye view feature map.

[0076] In the embodiment of the present application, the 2D bird's-eye view feature map generated in step 104 can be used to directly predict the 3D object detection result. For example, the 2D bird's-eye view feature map is convolved on an existing object detection network such as Fbev to obtain the 3D object detection result, wherein the object detection network can be composed of one or more convolutional layers.

[0077] It can be seen from the above embodiments that in this embodiment, the semantic features of the images in the image group are extracted to obtain a two-dimensional semantic feature map, the depth information of the two-dimensional semantic feature map is estimated to obtain a two-dimensional depth feature map, the two-dimensional semantic feature map is feature compressed in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and the two-dimensional depth feature map is feature compressed to obtain a one-dimensional depth feature map; based on the one-dimensional semantic feature map and the one-dimensional depth feature map, a two-dimensional bird's-eye view feature map is generated, and based on the two-dimensional bird's-eye view feature map, target detection is performed.

[0078] Compared with the related art, in the embodiment of the present application, in the process of converting image features from the camera perspective to the bird's-eye view, considering that there is information redundancy in the two-dimensional image in the height dimension, the two-dimensional image features and two-dimensional depth information under the camera perspective can be compressed into one-dimensional image features and one-dimensional depth information in the height dimension, and a two-dimensional bird's-eye view feature map for 3D target detection is generated based on the one-dimensional image features and one-dimensional depth information. Since the conversion process only involves compression processing of redundant information in the height dimension, the information in the width dimension will not be compressed, and converting the two-dimensional information into one-dimensional information can greatly reduce the amount of data in the conversion process. Therefore, the occupancy of the video memory can be reduced while ensuring the accuracy of the feature conversion results, thereby reducing the cost of deployment on the vehicle chip.

[0079] Figure 3 This is the second flow chart of the target detection method provided in the embodiment of the present application. In the embodiment of the present application, a two-dimensional bird's-eye view feature map can be generated based on a one-dimensional semantic feature map and a one-dimensional depth feature map by means of matrix operation, such as Figure 3 As shown, the method may include the following steps: step 301, step 302, step 303, step 304 and step 305;

[0080] In step 301, semantic features of images in an image group are extracted to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras.

[0081] In step 302, depth information is estimated on the two-dimensional semantic feature map to obtain a two-dimensional depth feature map.

[0082] In step 303, feature compression is performed on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and feature compression is performed on the two-dimensional depth feature map to obtain a one-dimensional depth feature map.

[0083] Steps 301 to 303 in the embodiment of the present application are the same as Figure 1 Steps 101 to 103 in the embodiment shown are similar and will not be described in detail. Figure 1 The corresponding contents in the illustrated embodiment.

[0084] Considering that when matrix operations are used for feature conversion, if it is a two-dimensional feature, the matrix will be very large during the intermediate conversion process, and will occupy a large amount of video memory. In the embodiment of the present application, in order to reduce the amount of matrix data, the two-dimensional feature can be reduced to a one-dimensional feature. The use of one-dimensional features will reduce the intermediate data a lot, making the matrix feature conversion method possible.

[0085] In step 304 , based on the static transformation matrix and guided by the depth information in the one-dimensional depth feature map, the features in the one-dimensional semantic feature map are projected into the two-dimensional bird's-eye view feature map.

[0086] Considering that the image coordinate system is combined with depth, it can be regarded as a polar coordinate system with two scales of distance and direction. However, the bird's-eye view feature map is a Cartesian coordinate system. The conversion of one-dimensional features into two-dimensional bird's-eye view features involves a conversion between coordinate systems. In the embodiment of the present application, a static transformation matrix for conversion between coordinates can be pre-generated based on the intrinsic and extrinsic parameter information of multiple cameras.

[0087] In an embodiment of the present application, a static transformation matrix is generated based on the intrinsic parameter information and extrinsic parameter information of multiple cameras. The static transformation matrix contains coordinate conversion information between the polar coordinate system and the Cartesian coordinate system. The coordinate conversion information is used to convert the semantic features and depth information in the polar coordinate system into two-dimensional bird's-eye view features in the Cartesian coordinate system.

[0088] In an embodiment of the present application, the size and number of feature channels of the two-dimensional bird's-eye view feature map can be predefined so as to project the features in the one-dimensional semantic feature map into the two-dimensional bird's-eye view feature map.

[0089] In this embodiment of the present application, a static transformation matrix may include a ring matrix and a ray matrix. The ring matrix contains depth information for multiple cameras, while the ray matrix contains directional information for multiple cameras. Since the ring matrix lacks directional information, the ray matrix does, so the ring matrix and ray matrix must be used together.

[0090] The above two matrices are only generated during the first calculation. In subsequent calculations, if the camera's internal and external parameters remain unchanged, they do not need to be generated again.

[0091] In an embodiment of the present application, in an autonomous driving scenario, the ring matrix can be viewed as a set of concentric rings centered on the vehicle, and the ray matrix can be viewed as a set of directional rays centered on the vehicle and emitted toward the concentric rings.

[0092] In some embodiments, the process of generating the ring matrix may include the following steps: step 401, step 402, and step 403;

[0093] In step 401, a first initial matrix of size L*L is constructed, and M depth values d are set, where L is the size of the two-dimensional bird's-eye view feature map, M is the number of channels of depth information in the one-dimensional semantic feature map, and both L and M are integers greater than 1.

[0094] In step 402, for each depth value d, at each position of the first initial matrix, if the depth from the actual coordinates of the position to the camera is equal to d, the value of the position is set to 1, otherwise it is set to 0, thereby obtaining M matrices of size L*L;

[0095] In step 403, M matrices of size L*L are superimposed to obtain a ring matrix.

[0096] In one example, the shape of a one-dimensional semantic feature map is 6*256*44, the shape of a one-dimensional depth feature map is 6*112*44, and the shape of a ring matrix is 112*16384. The generation process is as follows: for each depth value d of the 112 depths, at each position in the 128*128 matrix, if the actual coordinate corresponding to the position is equal to the depth d of the camera, then the value of the position is set to 1, otherwise it is set to 0. In this way, 112 128*128 matrices can be generated, superimposed together, and reshaped to 112*16384.

[0097] In some embodiments, the process of generating the ray matrix may include the following steps: step 501, step 502, and step 503;

[0098] In step 501, a second initial matrix with a size of L*L is constructed.

[0099] In step 502, for each viewing direction of each camera, at each position in the second initial matrix, if the position is in the viewing direction, the value of the position is set to 1, otherwise it is set to 0, and N matrices of size L*L are obtained, where N=S*T, S is the number of cameras, T is the number of viewing directions of a single camera, and both S and T are integers greater than 1.

[0100] In step 503, N matrices of size L*L are superimposed to obtain a ray matrix.

[0101] In one example, the shape of a one-dimensional semantic feature map is 6*256*44, the shape of a one-dimensional depth feature map is 6*112*44, and the shape of a ray matrix is 264*16384. The generation process is as follows: for each column vector c of the one-dimensional semantic feature map of the six cameras (each camera has 44 column vectors, each corresponding to a direction), at each position in the 128*128 matrix, if the position is on the line of sight corresponding to the column vector c, the value of the position is set to 1, otherwise it is set to 0. In this way, 44*6=264 128*128 matrices can be generated, superimposed together, and reshaped to 264*16384.

[0102] It should be noted that, in addition to the above-described methods of generating ring matrices and ray matrices, other methods of generating static transformation matrices and using them for the same or equivalent matrix multiplication operations are also within the scope of protection of this application.

[0103] In some embodiments, feature conversion is achieved by cooperating with a ring matrix and a ray matrix. Specifically, a matrix cross product can be performed between the ring matrix and the one-dimensional depth feature map (to embed the depth information of the images in the image group into the ring matrix), and the result is matrix dot producted with the ray matrix (to obtain direction, because the ring matrix only contains depth information but not direction information, while the ray matrix contains direction information). Finally, a matrix cross product is performed between the matrix and the one-dimensional semantic feature map to obtain a two-dimensional bird's-eye view feature map. Accordingly, the above step 304 includes the following steps: step 3041, step 3042, and step 3043;

[0104] In step 3041, a matrix cross multiplication operation is performed on the ring matrix and the depth information in the one-dimensional depth feature map to obtain an intermediate matrix.

[0105] In an embodiment of the present application, a matrix cross multiplication operation is performed on the ring matrix and the depth information in the one-dimensional depth feature map, which is essentially a weighted summation process of the ring matrix and the depth information in the one-dimensional depth feature map. The intermediate matrix obtained after the weighted summation is the ring matrix carrying the depth information of the images in the image group.

[0106] In step 3042, a matrix dot multiplication operation is performed on the intermediate matrix and the ray matrix to obtain a projection matrix.

[0107] In the embodiment of the present application, a matrix dot multiplication operation is performed on the intermediate matrix and the ray matrix, which is essentially a mask operation on the intermediate matrix to add directional information to the intermediate matrix. The resulting projection matrix is a ring matrix that carries depth information and directional information.

[0108] In step 3043, a matrix cross multiplication operation is performed on the projection matrix and the features in the one-dimensional semantic feature map to obtain a two-dimensional bird's-eye view feature map.

[0109] In an example, the one-dimensional semantic feature map is transposed to a shape of 6*44*112 and reshaped to 264*112, the one-dimensional depth feature map is transposed to a shape of 256*6*44 and reshaped to 256*264, the reshaped one-dimensional depth feature map and the ring matrix are cross-producted to obtain an intermediate matrix with a shape of 264*16384; the intermediate matrix and the ray matrix are dot-producted to obtain a projection matrix with a shape of 264*16384; the reshaped one-dimensional semantic feature map and the projection matrix are cross-producted to obtain a two-dimensional bird's-eye view feature map with a shape of 256*16384, which is reshaped to 256*128*128 and can be used as a two-dimensional feature representation.

[0110] For ease of understanding, combined Figure 4 The schematic diagram shown describes the feature conversion process.

[0111] like Figure 4 As shown, ① is the ring matrix, D×(L×L) is the shape of ①, ② is the one-dimensional depth feature map, W×D is the shape of ②, ③ is the ray matrix, W×(L×L) is the shape of ③, ④ is the one-dimensional semantic feature map, W×C is the shape of ④, first, perform matrix cross product on the ring matrix ① and the one-dimensional depth feature map ② to obtain the intermediate matrix ⑤, the shape is W×(L×L), perform dot product on the intermediate matrix ⑤ and the ray matrix ③ to obtain the projection matrix ⑥, the shape is W×(L×L), perform matrix cross product on the projection matrix ⑥ and the one-dimensional semantic feature map ④ to obtain the two-dimensional bird's-eye view feature ⑦, merge the two-dimensional bird's-eye view feature ⑦ to obtain the two-dimensional bird's-eye view feature map ⑧, the shape is C×(L×L).

[0112] In step 305, target detection is performed based on the two-dimensional bird's-eye view feature map.

[0113] Under the premise of using ResNet-50 as the backbone network, the performance comparison between the technical solution of the present application and the existing technical solution on the NuScenes dataset is shown in Table 1 below.

[0114]

[0115] Table 1

[0116] Through comparison, it can be seen that the technical solution of this application is on par with the most advanced BEVDepth algorithm in terms of two key indicators, mAP and NDS, and the technical solution of this application has the advantages of significantly reducing video memory usage and being easier to deploy on chip devices.

[0117] It can be seen from the above embodiments that in this embodiment, the semantic features of the images in the image group are extracted to obtain a two-dimensional semantic feature map, the depth information of the two-dimensional semantic feature map is estimated to obtain a two-dimensional depth feature map, the feature compression of the two-dimensional semantic feature map is performed on the height dimension of the feature map to obtain a one-dimensional semantic feature map, and the feature compression of the two-dimensional depth feature map is performed to obtain a one-dimensional depth feature map; based on the static transformation matrix, guided by the depth information in the one-dimensional depth feature map, the features in the one-dimensional semantic feature map are projected into a two-dimensional bird's-eye view feature map; and target detection is performed based on the two-dimensional bird's-eye view feature map.

[0118] Compared with related technologies, in the embodiment of the present application, in the process of converting image features from the camera perspective to the bird's-eye view, considering the information redundancy of the two-dimensional image in the height dimension, the two-dimensional image features and two-dimensional depth information under the camera perspective can be compressed into one-dimensional image features and one-dimensional depth information in the height dimension. Based on the static transformation matrix, the one-dimensional image features and the one-dimensional depth information, a two-dimensional bird's-eye view feature map for 3D target detection is generated. Since the conversion process only involves the compression processing of redundant information in the height dimension, the information in the width dimension will not be compressed, and converting the two-dimensional information into one-dimensional information can significantly reduce the amount of data in the conversion process. Therefore, it is possible to reduce the occupation of video memory while ensuring the accuracy of the feature conversion results, thereby reducing the cost of deployment on the vehicle chip. In addition, since only basic matrix multiplication and feature transposition operations are used in the conversion process, operators such as feature sampling or pooling that are difficult to deploy or difficult to accelerate are not used, making it easier to deploy on the vehicle chip.

[0119] Corresponding to Figure 3 The target detection method shown in the embodiment of the present application can also provide a target detection system, such as Figure 5 As shown, the target detection system may include the following modules: a feature extraction module, a depth estimation module, a feature and depth compression module, a feature projection module and a target detection module; wherein,

[0120] The feature extraction module is used to extract high-level semantic features from the input image group. It is usually one of the common image classification architectures such as AlexNet, ResNet, VGG, or MobileNet. Its input is an image group and its output is a two-dimensional semantic feature map.

[0121] The depth estimation module is used to estimate the physical depth of the corresponding position from the image features with the help of camera intrinsic parameter information. The module consists of one or more convolutional layers and fully connected operators. Its input is a two-dimensional semantic feature map and its output is a two-dimensional depth feature map.

[0122] The feature and depth compression module is used to perform convolution or height-dimensional pooling operations on image feature information, and to predict weights from the corresponding feature information for depth information, and to perform weighted summation at the height. Its input is a two-dimensional semantic feature map and a two-dimensional depth feature map, and its output is a one-dimensional semantic feature map and a one-dimensional depth feature map.

[0123] The feature projection module is used to use a pre-generated static transformation matrix and, guided by the depth in the one-dimensional depth feature map, project the features in the one-dimensional semantic feature map into the two-dimensional bird's-eye view feature map to form two-dimensional features from a bird's-eye view. Specifically, a matrix cross product is performed between the static "ring matrix" and the one-dimensional depth feature map. The result is then matrix dot-producted with the static "ray matrix". Finally, this matrix is matrix cross-producted with the one-dimensional semantic feature map to obtain the two-dimensional bird's-eye view features. Its input is a one-dimensional semantic feature map, a one-dimensional depth feature map, and a static transformation matrix, and its output is a two-dimensional bird's-eye view feature map.

[0124] The target detection module is used to extract two-dimensional bird's-eye view features through one or more convolutions. Its input is a two-dimensional bird's-eye view feature map, and its output is a 3D detection result.

[0125] It can be seen that the embodiments of the present application can have the following advantages: high performance and easy deployment. Among them, high performance is reflected in the use of one-dimensional semantic features and one-dimensional depth features for perspective conversion, which significantly reduces video memory usage and deployment costs compared to existing methods; easy deployment is reflected in the absence of operators such as feature sampling or pooling that are difficult to deploy or difficult to accelerate. All modules are composed of basic matrix multiplication and feature transposition operations. In summary, the technical solution of this application has broad application value in the fields of autonomous driving and robotics.

[0126] Figure 6 This is the third flow chart of the target detection method provided in the embodiment of the present application. In the embodiment of the present application, the one-dimensional semantic feature map and the one-dimensional depth feature map can be combined with other existing computing networks or modules to achieve a significant reduction in video memory usage, such as Figure 6 As shown, the method may include the following steps: step 601, step 602, step 603, step 604 and step 605;

[0127] In step 601, semantic features of images in an image group are extracted to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras.

[0128] In step 602, depth information is estimated on the two-dimensional semantic feature map to obtain a two-dimensional depth feature map.

[0129] In step 603, feature compression is performed on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and feature compression is performed on the two-dimensional depth feature map to obtain a one-dimensional depth feature map.

[0130] Steps 601 to 603 in the embodiment of the present application are the same as Figure 1 Steps 101 to 103 in the embodiment shown are similar and will not be described in detail. Figure 1 The content in the real-time example is shown.

[0131] In step 604, a matrix outer product operation is performed on the features in the one-dimensional semantic feature map and the depth information in the one-dimensional depth feature map to obtain a one-dimensional bird's-eye view feature map; feature sampling or voxel pooling is performed on the one-dimensional bird's-eye view feature map to generate a two-dimensional bird's-eye view feature map.

[0132] In an embodiment of the present application, a matrix outer product operation is performed on the features in the one-dimensional semantic feature map and the depth information in the one-dimensional depth feature map to obtain a one-dimensional bird's-eye view feature map with a perspective effect, and then the feature map is processed using feature sampling (GridSample) or voxel pooling (VoxelPool) to obtain a two-dimensional bird's-eye view feature map.

[0133] In an example, the shape of a one-dimensional semantic feature map is 6*256*44, the shape of a one-dimensional depth feature map is 6*112*44, and a matrix outer product operation is performed on the one-dimensional semantic feature map and the one-dimensional depth feature map to obtain a one-dimensional bird's-eye view feature map, whose shape is 6*256*112*44; the one-dimensional bird's-eye view feature map is subjected to feature sampling or voxel pooling processing to obtain a two-dimensional bird's-eye view feature map, whose shape is 256*128*128, where 256 represents the number of feature channels and 128 represents the height and width of the feature map.

[0134] In step 605, target detection is performed based on the two-dimensional bird's-eye view feature map.

[0135] It can be seen from the above embodiments that in this embodiment, the semantic features of the images in the image group are extracted to obtain a two-dimensional semantic feature map, the depth information of the two-dimensional semantic feature map is estimated to obtain a two-dimensional depth feature map, the two-dimensional semantic feature map is feature compressed in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and the two-dimensional depth feature map is feature compressed to obtain a one-dimensional depth feature map; a matrix outer product operation is performed on the features in the one-dimensional semantic feature map and the depth information in the one-dimensional depth feature map to obtain a one-dimensional bird's-eye view feature map; feature sampling or voxel pooling is performed on the one-dimensional bird's-eye view feature map to generate a two-dimensional bird's-eye view feature map; and target detection is performed based on the two-dimensional bird's-eye view feature map.

[0136] Compared with related technologies, in the embodiment of the present application, in the process of converting image features from the camera perspective to the bird's-eye view, considering the information redundancy of the two-dimensional image in the height dimension, the two-dimensional image features and two-dimensional depth information under the camera perspective can be compressed into one-dimensional image features and one-dimensional depth information in the height dimension. Based on the one-dimensional image features, one-dimensional depth information and operators such as feature sampling or voxel pooling, a two-dimensional bird's-eye view feature map for 3D target detection is generated. Since the conversion process only involves the compression processing of redundant information in the height dimension, the information in the width dimension will not be compressed, and converting the two-dimensional information into one-dimensional information can significantly reduce the amount of data in the conversion process. Therefore, it is possible to reduce the usage of video memory while ensuring the accuracy of the feature conversion results, reducing the cost of deployment on the vehicle chip. In addition, since the conversion process can be coordinated with other existing computing networks or modules, it can reduce R&D costs, shorten the construction period, and improve the reuse rate of technology.

[0137] Figure 7 is a schematic diagram of the structure of a target detection device provided in an embodiment of the present application, such as Figure 7 As shown, the target detection device 700 may include: an extraction module 701, an estimation module 702, a compression module 703, a generation module 704 and a detection module 705;

[0138] An extraction module 701 is configured to extract semantic features of images in an image group to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras;

[0139] An estimation module 702 is configured to estimate depth information of the two-dimensional semantic feature map to obtain a two-dimensional depth feature map;

[0140] A compression module 703 is configured to perform feature compression on the two-dimensional semantic feature map in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and to perform feature compression on the two-dimensional depth feature map to obtain a one-dimensional depth feature map;

[0141] A generating module 704 is configured to generate a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map;

[0142] The detection module 705 is used to perform target detection based on the two-dimensional bird's-eye view feature map.

[0143] It can be seen from the above embodiments that in this embodiment, the semantic features of the images in the image group are extracted to obtain a two-dimensional semantic feature map, the depth information of the two-dimensional semantic feature map is estimated to obtain a two-dimensional depth feature map, the two-dimensional semantic feature map is feature compressed in the height dimension of the feature map to obtain a one-dimensional semantic feature map, and the two-dimensional depth feature map is feature compressed to obtain a one-dimensional depth feature map; based on the one-dimensional semantic feature map and the one-dimensional depth feature map, a two-dimensional bird's-eye view feature map is generated, and based on the two-dimensional bird's-eye view feature map, target detection is performed.

[0144] Compared with the related art, in the embodiment of the present application, in the process of converting image features from the camera perspective to the bird's-eye view, considering that there is information redundancy in the two-dimensional image in the height dimension, the two-dimensional image features and two-dimensional depth information under the camera perspective can be compressed into one-dimensional image features and one-dimensional depth information in the height dimension, and a two-dimensional bird's-eye view feature map for 3D target detection is generated based on the one-dimensional image features and one-dimensional depth information. Since the conversion process only involves compression processing of redundant information in the height dimension, the information in the width dimension will not be compressed, and converting the two-dimensional information into one-dimensional information can greatly reduce the amount of data in the conversion process. Therefore, the occupancy of the video memory can be reduced while ensuring the accuracy of the feature conversion results, thereby reducing the cost of deployment on the vehicle chip.

[0145] Optionally, as an embodiment, the generating module 704 may include:

[0146] a projection submodule, configured to project features in the one-dimensional semantic feature map into a two-dimensional bird's-eye view feature map based on a static transformation matrix and guided by the depth information in the one-dimensional depth feature map;

[0147] In which, the static transformation matrix is generated based on the intrinsic parameter information and extrinsic parameter information of the multiple cameras, and the static transformation matrix contains coordinate conversion information between the polar coordinate system and the Cartesian coordinate system. The coordinate conversion information is used to convert the semantic features and depth information under the polar coordinate system into two-dimensional bird's-eye view features under the Cartesian coordinate system.

[0148] Optionally, as an embodiment, the static transformation matrix includes: a ring matrix and a ray matrix, wherein the ring matrix contains depth information of the multiple cameras, and the ray matrix contains direction information of the multiple cameras.

[0149] Optionally, as an embodiment, the projection submodule may include:

[0150] A first operation unit is configured to perform a matrix cross multiplication operation on the ring matrix and the depth information in the one-dimensional depth feature map to obtain an intermediate matrix;

[0151] A second operation unit is used to perform a matrix dot multiplication operation on the intermediate matrix and the ray matrix to obtain a projection matrix;

[0152] The third operation unit is used to perform a matrix cross multiplication operation on the projection matrix and the features in the one-dimensional semantic feature map to obtain a two-dimensional bird's-eye view feature map.

[0153] Optionally, as an embodiment, the process of generating the ring matrix may include:

[0154] Constructing a first initial matrix of size L*L, and setting M depth values d, where L is the size of the two-dimensional bird's-eye view feature map, M is the number of channels of depth information in the one-dimensional semantic feature map, and L and M are both integers greater than 1;

[0155] For each depth value d, at each position of the first initial matrix, if the depth from the actual coordinates of the position to the camera is equal to d, then the value of the position is set to 1, otherwise it is set to 0, resulting in M matrices of size L*L;

[0156] The M matrices of size L*L are superimposed to obtain the ring matrix.

[0157] Optionally, as an embodiment, the process of generating the ray matrix may include:

[0158] Construct a second initial matrix of size L*L;

[0159] For each viewing direction of each camera, at each position in the second initial matrix, if the position is in the viewing direction, the value of the position is set to 1, otherwise it is set to 0, resulting in N matrices of size L*L, where N=S*T, S is the number of cameras, T is the number of viewing directions of a single camera, and both S and T are integers greater than 1;

[0160] The N matrices of size L*L are superimposed to obtain the ray matrix.

[0161] Optionally, as an embodiment, the generating module 704 may include:

[0162] A first generating submodule is configured to perform a matrix outer product operation on the features in the one-dimensional semantic feature map and the depth information in the one-dimensional depth feature map to obtain a one-dimensional bird's-eye view feature map;

[0163] The second generating submodule is used to perform feature sampling or voxel pooling processing on the one-dimensional bird's-eye view feature map to generate a two-dimensional bird's-eye view feature map.

[0164] Optionally, as an embodiment, the compression module 703 may include:

[0165] The first compression submodule is used to perform maximum pooling processing on the two-dimensional semantic feature map in the height dimension to obtain a one-dimensional semantic feature map.

[0166] Optionally, as an embodiment, the compression module 703 may include:

[0167] A prediction submodule, configured to predict the weight of the depth information corresponding to each point in the two-dimensional semantic feature map;

[0168] The second compression submodule is used to perform a point multiplication operation on the depth information of each point in the two-dimensional depth feature map and the weight of the corresponding position, and sum them in the height dimension to obtain a one-dimensional depth feature map.

[0169] Optionally, as an embodiment, the estimation module 702 may include:

[0170] The depth estimation submodule is used to estimate the corresponding depth information from the two-dimensional semantic feature map using the intrinsic reference information of the multiple cameras as auxiliary reference information to obtain a two-dimensional depth feature map.

[0171] Any step in the embodiment of the target detection method provided in the present application and the specific operation in any step can be completed by the corresponding module in the target detection device. The process of the corresponding operation completed by each module in the target detection device refers to the process of the corresponding operation described in the embodiment of the target detection method.

[0172] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] Figure 8 8 is a block diagram of an electronic device according to an embodiment of the present application. The electronic device includes a processing component 822, which further includes one or more processors, and memory resources represented by memory 832 for storing instructions executable by the processing component 822, such as applications. The applications stored in memory 832 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 822 is configured to execute the instructions to perform the above-described method.

[0174] The electronic device may further include a power supply component 826 configured to perform power management of the electronic device, a wired or wireless network interface 850 configured to connect the electronic device to a network, and an input / output (I / O) interface 858. The electronic device may operate based on an operating system stored in the memory 832, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0175] According to another embodiment of the present application, the present application further provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps in the target detection method described in any of the above embodiments.

[0176] According to another embodiment of the present application, the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the target detection method described in any of the above embodiments.

[0177] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0178] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0182] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0183] The above is a detailed introduction to the target detection method, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Extracting semantic features of images in an image group to obtain a two-dimensional semantic feature map, wherein the image group includes two-dimensional images captured by multiple cameras; Estimating depth information of the two-dimensional semantic feature map to obtain a two-dimensional depth feature map; In the height dimension of the feature map, feature compression is performed on the two-dimensional semantic feature map to obtain a one-dimensional semantic feature map, and feature compression is performed on the two-dimensional depth feature map to obtain a one-dimensional depth feature map; Generating a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map; Performing target detection based on the two-dimensional bird's-eye view feature map; The step of compressing the two-dimensional depth feature map to obtain a one-dimensional depth feature map includes: Predicting the weight of the depth information corresponding to each point in the two-dimensional semantic feature map; The depth information of each point in the two-dimensional depth feature map is multiplied by the weight of the corresponding position, and the sum is calculated in the height dimension to obtain a one-dimensional depth feature map.

2. The method according to claim 1, characterized in that The generating of a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map includes: Based on a static transformation matrix, guided by the depth information in the one-dimensional depth feature map, the features in the one-dimensional semantic feature map are projected into a two-dimensional bird's-eye view feature map; In which, the static transformation matrix is generated based on the intrinsic parameter information and extrinsic parameter information of the multiple cameras, and the static transformation matrix contains coordinate conversion information between the polar coordinate system and the Cartesian coordinate system. The coordinate conversion information is used to convert the semantic features and depth information under the polar coordinate system into two-dimensional bird's-eye view features under the Cartesian coordinate system.

3. The method according to claim 2, characterized in that The static transformation matrix includes: a ring matrix and a ray matrix, wherein the ring matrix includes depth information of the multiple cameras, and the ray matrix includes direction information of the multiple cameras.

4. The method according to claim 3, characterized in that The method of projecting the features in the one-dimensional semantic feature map into a two-dimensional bird's-eye view feature map based on a static transformation matrix and guided by the depth information in the one-dimensional depth feature map includes: Performing a matrix cross product operation on the ring matrix and the depth information in the one-dimensional depth feature map to obtain an intermediate matrix; Performing a matrix dot multiplication operation on the intermediate matrix and the ray matrix to obtain a projection matrix; A matrix cross product operation is performed on the projection matrix and the features in the one-dimensional semantic feature map to obtain a two-dimensional bird's-eye view feature map.

5. The method according to claim 3, characterized in that The generation process of the ring matrix includes: Constructing a first initial matrix of size L*L, and setting M depth values d, where L is the size of the two-dimensional bird's-eye view feature map, M is the number of channels of depth information in the one-dimensional semantic feature map, and L and M are both integers greater than 1; For each depth value d, at each position of the first initial matrix, if the depth from the actual coordinates of the position to the camera is equal to d, then the value of the position is set to 1, otherwise it is set to 0, resulting in M matrices of size L*L; The M matrices of size L*L are superimposed to obtain the ring matrix.

6. The method according to claim 3, characterized in that The generation process of the ray matrix includes: Construct a second initial matrix of size L*L; For each viewing direction of each camera, at each position in the second initial matrix, if the position is in the viewing direction, the value of the position is set to 1, otherwise it is set to 0, to obtain N matrices of size L*L, where N = S*T, S is the number of cameras, T is the number of viewing directions of a single camera, and both S and T are integers greater than 1; The N matrices of size L*L are superimposed to obtain the ray matrix.

7. The method according to claim 1, characterized in that The generating of a two-dimensional bird's-eye view feature map based on the one-dimensional semantic feature map and the one-dimensional depth feature map includes: Performing a matrix outer product operation on the features in the one-dimensional semantic feature map and the depth information in the one-dimensional depth feature map to obtain a one-dimensional bird's-eye view feature map; Feature sampling or voxel pooling is performed on the one-dimensional bird's-eye view feature map to generate a two-dimensional bird's-eye view feature map.

8. The method according to claim 1, characterized in that The step of performing feature compression on the two-dimensional semantic feature map to obtain a one-dimensional semantic feature map includes: Perform maximum pooling processing on the two-dimensional semantic feature map in the height dimension to obtain a one-dimensional semantic feature map.

9. The method according to claim 1, characterized in that The step of estimating depth information of the two-dimensional semantic feature map to obtain a two-dimensional depth feature map includes: Using the intrinsic reference information of the multiple cameras as auxiliary reference information, corresponding depth information is estimated from the two-dimensional semantic feature map to obtain a two-dimensional depth feature map.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images