Detection method, detection device, vehicle and readable storage medium

Abstract

The invention relates to a detection method, a detection device, a vehicle and a readable storage medium. The detection method comprises the steps of obtaining a scene image; projecting the two-dimensional feature map of the scene image to a preset view to obtain a feature map under the preset view, the preset view comprising a plurality of views; determining a three-dimensional feature map corresponding to the scene image according to the feature map under each view; and carrying out detection by using the three-dimensional feature map. Thus, the purpose of extracting the three-dimensional features can be achieved only by using the scene image collected by the pure vision sensor, the detection task in the three-dimensional space can be achieved without increasing the hardware cost, the convenience of executing the detection task in the three-dimensional space is improved, and the use experience of a user is improved. Besides, the three-dimensional feature map corresponding to the scene image is determined by using the feature maps under the plurality of views, so that the accuracy of the determined three-dimensional feature map is improved, and the detection accuracy is further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a detection method, a detection device, a vehicle and a readable storage medium. Background Art

[0002] In recent years, with the development of intelligent vehicle technology by automobile manufacturers, various levels of assisted driving technology have developed rapidly. Based on sensors such as visual sensors, laser rangefinders, ultrasonic and infrared sensors, the perception and detection of the surrounding environment and moving obstacles is one of the key technologies for realizing vehicle automatic navigation, and is also one of the key capabilities for realizing automatic driving and automatic parking.

[0003] Among them, the target object detection task needs to detect the location, size, heading angle and other attributes of the target box. The grid occupancy detection task needs to detect the occupancy status and obstacle category of the three-dimensional space grid.

[0004] At present, most of the features in the image space are projected into the BEV (Bird's Eye View) space, and feature fusion is performed in the BEV space, and then object detection is performed in the BEV space. However, the BEV space is a two-dimensional plane space, which is suitable for two-dimensional object detection tasks and two-dimensional plane occupancy grid prediction tasks, but cannot perform three-dimensional object detection and occupancy prediction. If you want to perform three-dimensional object detection and occupancy prediction, you need to extract features in the three-dimensional space. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides a detection method, a detection device, a vehicle and a readable storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, a detection method is provided, the detection method comprising:

[0007] Get scene image;

[0008] Projecting the two-dimensional feature map of the scene image onto a preset view to obtain a feature map under the preset view, wherein the preset view includes multiple views;

[0009] Determine a three-dimensional feature map corresponding to the scene image according to the feature map under each view;

[0010] The three-dimensional feature map is used for detection.

[0011] Optionally, projecting the two-dimensional feature map of the scene image to a preset view to obtain the feature map under the preset view includes:

[0012] For each of the views, the two-dimensional feature map of the scene image is input into the projection sub-model corresponding to the view to obtain a feature map under the view output by the projection sub-model.

[0013] Optionally, the projection sub-model is an attention mechanism model, and each of the projection sub-models generates a feature map under a view corresponding to the projection sub-model in the following manner:

[0014] For each feature position of the feature map under the view to be generated, select n position points in the vertical direction of the position, and determine the projection point coordinates of each position point on the scene image;

[0015] According to the projection point coordinates of each position point on the scene image, the query vector corresponding to the feature position and the two-dimensional feature map of the scene image, based on the attention mechanism, the feature value at the feature position is determined to obtain the feature map under the view, where n is an integer greater than 1.

[0016] Optionally, determining the feature value at the feature position based on an attention mechanism according to the projection point coordinates of each position point on the scene image, the query vector corresponding to the feature position, and the two-dimensional feature map of the scene image includes:

[0017] The features at the feature positions are determined by the following formula:

[0018]

[0019] Wherein, Q represents the feature value at the feature position, N represents the number of the scene images, c represents the cth frame scene image, q represents the query vector corresponding to the feature position, and p ic Represents the projection point of the i-th position point on the c-th frame scene image, F c A two-dimensional feature map representing the c-th frame scene image, and DeformAttn() represents the attention mechanism algorithm.

[0020] Optionally, the detection method further comprises:

[0021] The scene image is input into an image processing model to obtain a two-dimensional feature map of the scene image output by the image processing model.

[0022] Optionally, the image processing model includes a coding module and a feature extraction module; and inputting the scene image into the image processing model to obtain a two-dimensional feature map of the scene image output by the image processing model includes:

[0023] Inputting the scene image into the encoding module to obtain a plurality of feature maps of different scales output by the encoding module;

[0024] The multiple feature maps of different scales are input into a feature extraction module, so that the feature extraction module performs feature fusion on the multiple feature maps of different scales to generate a two-dimensional feature map of the scene image.

[0025] Optionally, the scene image is a scene image captured by a camera arranged on a vehicle, and / or the scene image includes multiple frame images.

[0026] Optionally, the preset views include three mutually perpendicular views.

[0027] According to a second aspect of an embodiment of the present disclosure, a detection device is provided, including:

[0028] An acquisition module, configured to acquire a scene image;

[0029] A projection module is configured to project the two-dimensional feature map of the scene image to a preset view to obtain a feature map under the preset view, wherein the preset view includes multiple views;

[0030] A determination module is configured to determine a three-dimensional feature map corresponding to the scene image according to the feature map under each view;

[0031] The detection module is configured to perform detection according to the three-dimensional feature map.

[0032] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising:

[0033] processor;

[0034] a memory for storing processor-executable instructions;

[0035] Wherein, the processor is configured to execute the steps of the detection method as described in the first aspect of the present disclosure.

[0036] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the detection method provided in the first aspect of the present disclosure are implemented.

[0037] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0038] Using the above technical solution, first, a scene image is acquired, then the two-dimensional feature map of the scene image is projected to multiple views to obtain the feature map under each view, and the three-dimensional feature map corresponding to the scene image is determined based on the feature map under each view, and finally, the three-dimensional feature map is used for detection. In this way, the purpose of extracting three-dimensional features can be achieved by using only the scene image collected by a pure visual sensor, and the detection task in three-dimensional space can be achieved without increasing the hardware cost, which improves the convenience of performing the detection task in three-dimensional space and enhances the user experience. In addition, the three-dimensional feature map corresponding to the scene image is determined using the feature maps under multiple views, which improves the accuracy of the determined three-dimensional feature map, thereby improving the accuracy of the detection.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0041] Figure 1 The figure is a flow chart of a detection method according to an exemplary embodiment.

[0042] Figure 2 is a flow chart of another detection method according to an exemplary embodiment.

[0043] Figure 3 is a schematic diagram of a detection method according to an exemplary embodiment.

[0044] Figure 4 The present invention is a frame of a detection device according to an exemplary embodiment.

[0045] Figure 5 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0047] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0048] As mentioned in the background technology, currently most methods are to project the features of the image space into the BEV (Bird's Eye View) space, fuse the features in the BEV space, and then perform target detection in the BEV space. This method is not suitable for detection tasks in three-dimensional space. Therefore, in order to perform detection tasks in three-dimensional space, it is necessary to extract the three-dimensional features of the image space. In related technologies, most of them use a combination of visual sensors and laser rangefinders to extract three-dimensional features from images. This will result in higher hardware costs for extracting three-dimensional features and a large amount of computation.

[0049] In view of this, the present disclosure provides a detection method, a detection device, a vehicle and a readable storage medium, which extract three-dimensional features from images collected by a pure visual sensor and use the extracted three-dimensional features for detection, thereby achieving detection tasks in three-dimensional space without increasing hardware costs.

[0050] Figure 1 FIG. 1 is a flow chart of a detection method according to an exemplary embodiment. Figure 1 As shown, the detection method may include the following steps.

[0051] In step S11, a scene image is acquired.

[0052] The scene image may be a scene image of a vehicle. For example, the scene image may be acquired by an image acquisition device such as a camera disposed on the vehicle. The scene image may be one frame or multiple frames. In practical applications, considering that the acquisition viewing angle of the image acquisition device is limited, in order to obtain as much scene information as possible, the scene image may include multiple frames of images.

[0053] For example, an autonomous driving vehicle is generally equipped with four surround-view cameras. Therefore, in the present disclosure, if the detection task in the three-dimensional space is a task related to the autonomous driving vehicle, for example, three-dimensional target object detection and three-dimensional grid occupancy detection tasks, then the above-mentioned step S11 of acquiring the scene image can be four frames of images captured by four surround-view cameras installed on the vehicle.

[0054] It should be understood that the scene image in step S11 may be an original image captured by a camera, or an image obtained by enhancing the original image, and the present disclosure does not specifically limit this.

[0055] In step S12, the two-dimensional feature map of the scene image is projected onto a preset view to obtain a feature map under the preset view.

[0056] The preset view may include multiple views. In one embodiment, the multiple views are perpendicular to each other, for example, the preset view may include at least two of a bird's eye view BEV, a perspective view PV, and a side view that are perpendicular to each other. For example, in the present disclosure, the preset view may include three mutually perpendicular views, for example, a bird's eye view BEV, a perspective view PV, and a side view. In another embodiment, the multiple views may also not be perpendicular.

[0057] It should be understood that when multiple views are not perpendicular, it is necessary to further determine the angle between the views, so as to facilitate the subsequent determination of the three-dimensional feature map corresponding to the scene image based on the feature map under each view and the angle between the views.

[0058] In step S13, a three-dimensional feature map corresponding to the scene image is determined according to the feature map in each view.

[0059] It should be understood that if the three-dimensional feature map of the scene image is determined using a feature map under one view, at least two-thirds of the information in the determined three-dimensional feature map will be lost, resulting in poor accuracy of the determined three-dimensional feature map, and thus poor detection accuracy.

[0060] In the present disclosure, the three-dimensional feature map corresponding to the scene image is determined by using feature maps under multiple views. Compared with determining the three-dimensional feature map corresponding to the scene image using feature maps under one view, the accuracy of the determined three-dimensional feature map is improved, thereby improving the accuracy of detection.

[0061] In step S14, detection is performed using the three-dimensional feature map.

[0062] First, it should be understood that no matter whether the number of scene images is one frame or multiple frames, the three-dimensional feature map corresponding to the determined scene image is one frame. Secondly, it should be understood that the more frames of the scene image, the more accurate the scene information included in the determined three-dimensional feature map, and then the three-dimensional feature map can be used to accurately perform detection, thereby improving the accuracy of detection.

[0063] In the present disclosure, target object detection and / or grid occupancy detection can be performed using a three-dimensional feature map. For example, the movement of other vehicles or pedestrians around the vehicle can be detected using a three-dimensional feature map. For another example, the three-dimensional feature map can be used to detect whether a three-dimensional grid is occupied, and the type of the occupied obstacle, etc.

[0064] Using the above technical solution, first, a scene image is acquired, then the two-dimensional feature map of the scene image is projected to multiple views to obtain the feature map under each view, and the three-dimensional feature map corresponding to the scene image is determined based on the feature map under each view, and finally, the three-dimensional feature map is used for detection. In this way, the purpose of extracting three-dimensional features can be achieved by using only the scene image collected by a pure visual sensor, and the detection task in three-dimensional space can be achieved without increasing the hardware cost, which improves the convenience of performing the detection task in three-dimensional space and enhances the user experience. In addition, the three-dimensional feature map corresponding to the scene image is determined using the feature maps under multiple views, which improves the accuracy of the determined three-dimensional feature map, thereby improving the accuracy of the detection.

[0065] In order to facilitate those skilled in the art to better understand the detection method provided by the present disclosure, the detection method is described below with a complete embodiment.

[0066] Figure 2 FIG. 1 is a flow chart of another detection method according to an exemplary embodiment. Figure 2 As shown, the detection method may include the following steps.

[0067] In step S21, a scene image is acquired.

[0068] In step S22, the scene image is input into the image processing model to obtain a two-dimensional feature map of the scene image output by the image processing model.

[0069] In this embodiment, a two-dimensional feature map of a scene image is obtained by an image processing model, and the image processing model is a pre-trained model. The training method of the model is a relatively mature technology, and the present disclosure does not make any specific limitations on this.

[0070] In one embodiment, the image processing model may include an encoding module and a feature extraction module. The specific implementation of step S22 is: inputting the scene image into the encoding module to obtain a plurality of feature maps of different scales output by the encoding module; inputting the plurality of feature maps of different scales into the feature extraction module, so that the feature extraction module performs feature fusion on the plurality of feature maps of different scales to generate a two-dimensional feature map of the scene image.

[0071] The encoding module may be an encoder. For example, in the present disclosure, RegNet800M may be used as an image space encoder. RegNet800M is a network structure obtained based on NAS (neural network search) search, which takes into account both the feature extraction capability and feature extraction computational complexity of the model.

[0072] For example, Figure 3 FIG. 1 is a schematic diagram of a detection method according to an exemplary embodiment. Figure 3As shown, the encoding module is RegNet800M, and RegNet800M includes 6 sequentially connected convolutional neural network layers, and each convolutional neural network layer outputs an encoded feature map. Figure 3 , the image data imag(4,3,576,768) of the scene image is input into the encoding module, wherein imag(4,3,576,768) represents 4 frames of scene images, each frame of the image includes 3 channels, such as an R channel, a G channel, and a B channel, and the length and width of each frame of the image are 576 pixels and 768 pixels, respectively.

[0073] like Figure 3 Each convolutional neural network layer encodes the image data of the scene image in turn. For example, the i+1th convolutional neural network layer encodes the feature map output by the ith convolutional neural network layer, where the value of i ranges from [1,5]. In addition, the convolutional neural network layers other than the first convolutional neural network layer output feature maps of different sizes.

[0074] In the present disclosure, the feature extraction module can perform feature fusion on multiple feature maps of different scales output by the encoding module. For example, the feature extraction module can be a bidirectional BiFPN. Figure 3 As shown in Figure 2, the bidirectional BiFPN is used to fuse the five feature maps of different sizes output by RegNet800M. The specific fusion method is shown in Figure 2. Figure 3 As shown by the arrows, for example, RegNet800M includes 6 layers of convolutional neural network layers, which are marked as 11-16 respectively, and BiFPN includes 10 layers of convolutional neural network layers, which are marked as 20-29 respectively, wherein convolutional neural network layer 20 performs feature extraction on the feature map output by convolutional neural network layer 12, and then inputs the extracted feature map into convolutional neural network layer 21, which fuses the feature map output by convolutional neural network layer 20 and the feature map output by convolutional neural network layer 13 to obtain a feature map, and inputs it into convolutional neural network layer 22, similarly, convolutional neural network layer 22 fuses the feature map output by convolutional neural network layer 21 and the feature map output by convolutional neural network layer 14 to obtain a feature map, and inputs it into convolutional neural network layer 23, and so on.

[0075] according to Figure 3 The fusion method shown by the middle arrow, after the bidirectional BiFPN is finally fused, the feature map (4,128,144,192) can be obtained. It should be understood that Figure 3 In the figure, the four numbers in each bracket represent the feature map output by the convolutional layer, and among the four numbers, the first number represents the number of frames of the feature map, the second number represents the number of channels of the feature map, and the third and fourth numbers represent the length and width of the feature map.

[0076] It should be understood that in the image processing model, encoding and feature fusion are performed on each frame of image separately during encoding and feature fusion, and feature fusion is not performed between multiple frames of scene images.

[0077] At this point, a two-dimensional feature map with better encoding effect can be obtained according to the above method, that is, the obtained two-dimensional feature map integrates multiple information in the scene image, thereby improving the accuracy of the determined two-dimensional feature map.

[0078] Below Figure 1 In step S12, the two-dimensional feature map of the scene image is projected onto a preset view, and a specific method of obtaining the feature map under the preset view is described.

[0079] In one implementation, the two-dimensional feature map can be projected onto a preset view according to a preset coordinate conversion matrix to obtain a feature map under each view. For details, please refer to the introduction in the conventional technology, which will not be described in detail here.

[0080] In another embodiment, if Figure 2 As shown, Figure 1 Step S12 may include step S23.

[0081] In step S23, for each view, the two-dimensional feature map of the scene image is input into the projection sub-model corresponding to the view to obtain the feature map under the view output by the projection sub-model.

[0082] In the present disclosure, the preset view includes three mutually perpendicular views as an example, for example, the preset view includes a bird's eye view BEV, a front view PV (perspective view) and a side view, and each view corresponds to a projection sub-model. Each projection sub-model can be a pre-trained model.

[0083] In this embodiment, the feature map under each view is obtained by using a pre-trained projection sub-model, and there is no need to use a coordinate transformation matrix to calculate the feature map under each view, which effectively reduces the amount of calculation in the projection process.

[0084] For example, each projection sub-model is an attention mechanism model, where the attention mechanism model refers to a model trained based on the attention mechanism. Each projection sub-model generates a feature map under the corresponding view of the projection sub-model in the following way:

[0085] First, for each feature position of the feature map under the view to be generated, n position points are selected in the vertical direction of the position, and the projection point coordinates of each position point on the scene image are determined.

[0086] It should be understood that when the projection sub-model is trained, the size of the feature map under the corresponding view of the projection sub-model can be determined. For example, taking the projection of two-dimensional features to the bird's-eye view BEV as an example, the tensor size of the feature map under the bird's-eye view BEV is 128x96x96, the size of the feature map is 96x96, and the number of channels is 128.

[0087] For each feature position in the feature map, n position points are selected in the vertical direction of the position and on the bird's-eye view BEV plane, where the height of each position point from the feature position can be determined when the position is selected. Assuming that for the position (4.8, 4.8) in the feature map, 5 positions are selected in the vertical direction of the position, and the coordinates of the 5 position points are (48, 48, 0.1), (48, 48, 0.2), (48, 48, 0.3), (48, 48, 0.4) and (48, 48, 0.5), respectively. Then, the coordinates of the projection point of each position point on the scene image are determined. For example, the coordinates of the projection point of the position point on the scene image can be determined based on the coordinates of the position point and the relevant parameters of the camera that collects the scene image, such as external parameters, internal parameters and distortion parameters. Among them, the specific method of determining the coordinates of the projection point based on the coordinates of the position point and the relevant parameters of the camera that collects the scene image can refer to the introduction in the conventional technology, and the present disclosure does not limit this.

[0088] It should be understood that if the scene image is multiple frames, it is necessary to determine the coordinates of the projection point of the position point on each frame of the image based on the coordinates of the position point and relevant parameters of the camera that captures the frame of the image.

[0089] Then, according to the projection point coordinates of each position point on the scene image, the query vector corresponding to the feature position and the two-dimensional feature map of the scene image, based on the attention mechanism, the feature value at the feature position is determined to obtain the feature map under the view, where n is an integer greater than 1.

[0090] In one approach, the features at that location can be determined by

[0091]

[0092] Wherein, Q represents the feature value at the feature position, N represents the number of the scene images, c represents the cth frame scene image, q represents the query vector corresponding to the feature position, and p ic Represents the projection point of the i-th position point on the c-th frame scene image, F c A two-dimensional feature map representing the c-th frame scene image, and DeformAttn() represents the attention mechanism algorithm.

[0093] For example, take the scene image as four frames, each frame of the scene image corresponds to a frame of two-dimensional feature map. Assume that the query vector of the position (h, w) is q, select n position points in the vertical direction at the position, and the projection point coordinates of the i-th position point on the c-th frame of the scene image are marked as p ic , with p ic As the query vector q, the 2D feature map F of the scene image in the cth frame c The coordinates of the upper reference point, at F c Perform feature extraction based on the attention mechanism to obtain the query extraction result DeformAttn(q,p ic ,F c ). After that, the query extraction results of the n positions and four frames of images in the vertical direction are summed to obtain the feature value at the feature position.

[0094] By calculating each feature position on the feature map in the above manner, the feature map of the view can be obtained. For other views, the feature maps of each view can also be obtained in the above manner.

[0095] return Figure 3 , assuming that the preset view includes three mutually perpendicular views, then Figure 3 There are three projection sub-models, among which projection sub-model 1 is the model corresponding to the bird's-eye view BEV, projection sub-model 2 is the model corresponding to the side view side, and projection sub-model 3 is the model corresponding to the front view PV. Among them, the tensor size bevgrid query of the feature map under the bird's-eye view BEV is 128x96x96, recorded as bev grid query (128, 96, 96), the tensor size side grid query of the feature map under the side view side is 128x16x96, recorded as side grid query (128, 16, 96), and the tensor size pv grid query of the feature map under the front view PV is 128x16x96, recorded as pv grid query (128, 16, 96).

[0096] like Figure 3 As shown, each projection sub-model includes a deformable attention mechanism module, a first addition and normalization operator module, a feedforward network layer, and a second addition and normalization operator module. The specific functions of each module can refer to the introduction in the conventional technology, and the present disclosure does not limit this.

[0097] In addition, if Figure 3As shown, camera-related parameters, such as extrinsic parameters, intrinsic parameters, and distortion parameters, are also input into the deformable attention mechanism module in each projection sub-model, so that the deformable attention mechanism module can determine the coordinates of the projection point based on the camera's extrinsic parameters, intrinsic parameters, and distortion parameters as well as the coordinates of the position point.

[0098] By adopting the above technical solution, when the two-dimensional feature map of the scene image is projected to the preset view, an attention mechanism is adopted, which can fully and effectively fuse and extract the two-dimensional features of the scene image.

[0099] After obtaining the feature map in each view in the above manner, a three-dimensional feature map corresponding to the scene image is further determined, wherein the three-dimensional feature map may also be referred to as a three-dimensional voxel feature map.

[0100] For example, a specific implementation method for determining a three-dimensional feature map corresponding to a scene image based on a feature map under each view is as follows: for each feature map under each view, determine the feature position corresponding to the same voxel in the three-dimensional space, and determine the three-dimensional feature value corresponding to the voxel based on the feature value at the feature position in each feature map.

[0101] Assume that the voxel (d, h, w) in the three-dimensional space corresponds to the feature position (h, w) in the feature map under the BEV view, corresponds to the feature position (d, h) in the feature map under the side view, and corresponds to the feature position (d, w) in the feature map under the PV view, and the eigenvalue at the feature position (h, w) is Q_hw, the eigenvalue at the feature position (d, h) is Q_dh, and the eigenvalue at the feature position (d, w) is Q_dw, then the eigenvalue corresponding to the voxel is obtained according to Q_hw, Q_dh and Q_dw.

[0102] By way of example, Q_hw, Q_dh and Q_dw can be added to obtain the eigenvalue corresponding to the voxel. By way of another example, the eigenvalue corresponding to the voxel can be obtained by the concat() function, for example, the eigenvalue corresponding to the voxel Q_dhw=concat(Q_dh,Q_dw,Q_hw). By way of another example, the eigenvalue corresponding to the voxel can also be obtained by the concat() function and the convolution operation, that is, after the numerical value is obtained by the concat() function, the convolution operation is performed to obtain the eigenvalue corresponding to the voxel. It is also possible to directly perform a convolution operation on Q_hw, Q_dh and Q_dw to obtain the eigenvalue corresponding to the voxel, and so on. The present disclosure does not make specific limitations on this.

[0103] Thus, according to the above method, the three-dimensional feature value corresponding to each voxel in the three-dimensional space can be obtained, and then the three-dimensional feature map corresponding to the scene image can be obtained. Figure 3 As shown in FIG. 1 , the size of the 3D feature map corresponding to the final scene image is 16×96×96 and the number of channels is 128.

[0104] return Figure 2 , in step S24, detection is performed using the three-dimensional feature map.

[0105] For example, in Figure 3 In the method, the 3D feature map corresponding to the scene image is input into the target object detection head and the grid occupancy detection head respectively, and the final 3D surround view target object detection result and grid occupancy result can be obtained.

[0106] Based on the same inventive concept, the present disclosure also provides a detection device. Figure 4 is a frame of a detection device according to an exemplary embodiment. Figure 4 As shown, the detection device 400 may include:

[0107] An acquisition module 401 is configured to acquire a scene image;

[0108] A projection module 402 is configured to project the two-dimensional feature map of the scene image to a preset view to obtain a feature map under the preset view, where the preset view includes multiple views;

[0109] A determination module 403 is configured to determine a three-dimensional feature map corresponding to the scene image according to the feature map in each view;

[0110] The detection module 404 is configured to perform detection using the three-dimensional feature map.

[0111] Optionally, the projection module 402 is configured to input the two-dimensional feature map of the scene image into a projection sub-model corresponding to the view for each of the views, and obtain a feature map under the view output by the projection sub-model.

[0112] Optionally, the projection sub-model is an attention mechanism model, and the projection module 402 is configured to: for each feature position of the feature map under the view to be generated, select n position points in the vertical direction of the position, and determine the projection point coordinates of each position point on the scene image;

[0113] According to the projection point coordinates of each position point on the scene image, the query vector corresponding to the feature position and the two-dimensional feature map of the scene image, based on the attention mechanism, the feature value at the feature position is determined to obtain the feature map under the view, where n is an integer greater than 1.

[0114] Optionally, the projection module 402 is configured to determine the feature at the feature position by using the following formula:

[0115]

[0116] Wherein, Q represents the feature value at the feature position, N represents the number of the scene images, c represents the cth frame scene image, q represents the query vector corresponding to the feature position, and p ic Represents the projection point of the i-th position point on the c-th frame scene image, F c A two-dimensional feature map representing the c-th frame scene image, and DeformAttn() represents the attention mechanism algorithm.

[0117] Optionally, the detection device 400 further includes:

[0118] The input module is configured to input the scene image into the image processing model to obtain a two-dimensional feature map of the scene image output by the image processing model.

[0119] Optionally, the image processing model includes an encoding module and a feature extraction module;

[0120] The input module is configured to: input the scene image into the encoding module to obtain a plurality of feature maps of different scales output by the encoding module; input the plurality of feature maps of different scales into a feature extraction module so that the feature extraction module performs feature fusion on the plurality of feature maps of different scales to generate a two-dimensional feature map of the scene image.

[0121] Optionally, the scene image is a scene image captured by a camera arranged on a vehicle, and / or the scene image includes multiple frame images.

[0122] Optionally, the preset views include three mutually perpendicular views.

[0123] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0124] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and the program instructions, when executed by a processor, implement the steps of the detection method provided by the present disclosure.

[0125] Figure 5 600 is a block diagram of a vehicle according to an exemplary embodiment. For example, vehicle 600 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. Vehicle 600 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0126] Reference Figure 5, the vehicle 600 may include various subsystems, for example, an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. The vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 600 may be interconnected by wire or wireless means.

[0127] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, and a navigation system, among others.

[0128] The perception system 620 may include several sensors for sensing information about the environment around the vehicle 600. For example, the perception system 620 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0129] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0130] The drive system 640 may include components that provide powered motion for the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0131] Some or all functions of the vehicle 600 are controlled by a computing platform 650. The computing platform 650 may include at least one processor 651 and a memory 652, and the processor 651 may execute instructions 653 stored in the memory 652.

[0132] The processor 651 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0133] The memory 652 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0134] In addition to the instructions 653 , the memory 652 may also store data, such as road maps, route information, and data such as the location, direction, and speed of the vehicle. The data stored in the memory 652 may be used by the computing platform 650 .

[0135] In the embodiment of the present disclosure, the processor 651 may execute instruction 653 to complete all or part of the steps of the above-mentioned detection method.

[0136] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above detection method when executed by the programmable device.

[0137] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0138] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A detection method, characterized in that: The detection method comprises: Get scene image; Projecting the two-dimensional feature map of the scene image onto a preset view to obtain a feature map under the preset view, wherein the preset view includes multiple views; Determine a three-dimensional feature map corresponding to the scene image according to the feature map under each view; The three-dimensional feature map is used for detection.

2. The detection method according to claim 1, characterized in that: The step of projecting the two-dimensional feature map of the scene image onto a preset view to obtain the feature map under the preset view includes: For each of the views, the two-dimensional feature map of the scene image is input into the projection sub-model corresponding to the view to obtain a feature map under the view output by the projection sub-model.

3. The detection method according to claim 2, characterized in that: The projection sub-model is an attention mechanism model, and each projection sub-model generates a feature map under the corresponding view of the projection sub-model in the following manner: For each feature position of the feature map under the view to be generated, select n position points in the vertical direction of the position, and determine the projection point coordinates of each position point on the scene image; According to the projection point coordinates of each position point on the scene image, the query vector corresponding to the feature position and the two-dimensional feature map of the scene image, based on the attention mechanism, the feature value at the feature position is determined to obtain the feature map under the view, where n is an integer greater than 1.

4. The detection method according to claim 3, characterized in that: The determining, based on the attention mechanism, the feature value at the feature position according to the projection point coordinates of each position point on the scene image, the query vector corresponding to the feature position, and the two-dimensional feature map of the scene image, comprises: The features at the feature positions are determined by the following formula: Wherein, Q represents the feature value at the feature position, N represents the number of the scene images, c represents the cth frame scene image, q represents the query vector corresponding to the feature position, and p ic Represents the projection point of the i-th position point on the c-th frame scene image, F c A two-dimensional feature map representing the c-th frame scene image, and DeformAttn() represents the attention mechanism algorithm.

5. The detection method according to claim 1, characterized in that: The detection method further comprises: The scene image is input into an image processing model to obtain a two-dimensional feature map of the scene image output by the image processing model.

6. The detection method according to claim 5, characterized in that: The image processing model includes a coding module and a feature extraction module; The step of inputting the scene image into an image processing model to obtain a two-dimensional feature map of the scene image output by the image processing model comprises: Inputting the scene image into the encoding module to obtain a plurality of feature maps of different scales output by the encoding module; The multiple feature maps of different scales are input into a feature extraction module, so that the feature extraction module performs feature fusion on the multiple feature maps of different scales to generate a two-dimensional feature map of the scene image.

7. The detection method according to any one of claims 1 to 6, characterized in that: The scene image is a scene image captured by a camera arranged on a vehicle, and / or the scene image includes multiple frame images.

8. The detection method according to any one of claims 1 to 6, characterized in that: The preset views include three views that are perpendicular to each other.

9. A detection device, characterized in that: include: An acquisition module, configured to acquire a scene image; A projection module is configured to project the two-dimensional feature map of the scene image to a preset view to obtain a feature map under the preset view, wherein the preset view includes multiple views; A determination module is configured to determine a three-dimensional feature map corresponding to the scene image according to the feature map under each view; The detection module is configured to perform detection according to the three-dimensional feature map.

10. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the steps of the detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the detection method described in any one of claims 1 to 8 are implemented.

Images