Target detection method and device, controller and automatic driving equipment

Abstract

The embodiment of the invention provides a target detection method and device, a controller and automatic driving equipment, and the method comprises the steps: obtaining a multi-view image of a target region where the automatic driving equipment is located, and corresponding camera parameters, and the target region comprises a to-be-detected target object; performing feature extraction on the multi-view image, fusing camera parameters to obtain a three-dimensional feature image of the target area, performing rasterized semantic segmentation and offset prediction on the three-dimensional feature image to respectively obtain a grid mask image and offset information, and performing image segmentation on the grid mask image and the offset information; the grid mask image is used for indicating information of a target grid occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used for indicating position offset information of the target object in the target grid; and determining the position information of the target object according to the offset information and the information of the target grid in the grid mask image. By adopting the scheme, the obtained position information error of the target object is smaller, and the position detection result of the target object is more accurate.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of autonomous driving technology, and in particular to a target detection method, device, controller, and autonomous driving equipment. Background Art

[0002] At present, in the field of autonomous driving technology, BEV (Bird's Eye View) perception technology is usually applied to target detection technology. The target detection method based on BEV perception mainly includes the rasterization semantic segmentation method. Rasterization semantic segmentation predicts the position information of road elements in the rasterized BEV image by performing semantic segmentation on the BEV image. Road elements include passable areas, lane lines, etc. In the rasterized BEV image, if the road element occupies multiple grids, the position of the occupied grids is used as the position prediction result of the road element. When more grids are occupied or the size of the grids is large, the error of the position prediction result of the road element will be large. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a target detection method, device, controller, and autonomous driving equipment to at least partially solve the above-mentioned problems.

[0004] According to a first aspect of an embodiment of the present application, a target detection method is provided, including: obtaining multi-view images of a target area where an autonomous driving device is located and corresponding camera parameters, the target area including a target object to be detected; performing feature extraction on the multi-view images, and fusing the camera parameters to obtain a three-dimensional feature image of the target area, performing rasterized semantic segmentation and offset prediction on the three-dimensional feature image, and obtaining a grid mask image and offset information, respectively, wherein the grid mask image is used to indicate information about a target grid occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid; determining the position information of the target object based on the offset information and the information about the target grid in the grid mask image.

[0005] According to the second aspect of the embodiment of the present application, a target detection device is provided, comprising an image acquisition module, a prediction module, and a position information determination module. The image acquisition module is used to acquire multi-view images of a target area where the autonomous driving device is located and corresponding camera parameters, wherein the target area includes a target object to be detected; the prediction module is used to extract features from the multi-view images and fuse the camera parameters to obtain a three-dimensional feature image of the target area, perform gridded semantic segmentation and offset prediction on the three-dimensional feature image, and obtain a grid mask image and offset information, respectively, wherein the grid mask image is used to indicate information about the target grid occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid; and the position information determination module is used to determine the position information of the target object based on the offset information and the information about the target grid in the grid mask image.

[0006] According to a third aspect of an embodiment of the present application, an autonomous driving device is provided, comprising: a sensor, a processor, a memory, a communication interface and a communication bus, wherein the sensor, the processor, the memory and the communication interface communicate with each other through the communication bus; the sensor is used to collect environmental data of the environment in which the autonomous driving device is located and generate perception data; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method of the first aspect based on the perception data.

[0007] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of the first aspect is implemented.

[0008] According to the fifth aspect of the embodiment of the present application, a controller is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the memory is used to store computer programs; the processor is used to execute the method steps of the first aspect by running the computer program stored in the memory.

[0009] According to the target detection method, device, controller and autonomous driving equipment provided by the embodiments of the present application, the target detection method includes: obtaining multi-view images of the target area where the autonomous driving equipment is located and corresponding camera parameters, the target area includes the target object to be detected; performing feature extraction on the multi-view images, and fusing the camera parameters to obtain a three-dimensional feature image of the target area, performing rasterization semantic segmentation and offset prediction on the three-dimensional feature image, and obtaining a grid mask image and offset information respectively, wherein the grid mask image is used to indicate information about the target grid occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid; determining the position information of the target object based on the offset information and the information of the target grid in the grid mask image. In this scheme, after determining the target grid occupied by the target object, the offset information of the target object in the target grid is further added, and the information of the target grid is corrected using the offset information to obtain the position information of the target object. Therefore, compared with directly using the position information of the target grid as the position information of the target object, this scheme will not cause the accuracy of the obtained target object position information to be low due to the number or size of the target grid. It can make the error of the detected target object position information smaller and the target object position detection result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0011] Figure 1 is a flowchart of a target detection method according to an exemplary embodiment of the present application;

[0012] Figure 2 is a flowchart of a target detection method according to another exemplary embodiment of the present application;

[0013] Figure 3 is a structural diagram of a neural network model according to an exemplary embodiment of the present application;

[0014] Figure 4 Schematic diagram of a process of rasterized semantic segmentation according to an exemplary embodiment of the present application;

[0015] Figure 5 is a schematic diagram of a grid mask image according to an exemplary embodiment of the present application;

[0016] Figure 6 is a schematic diagram of offset information according to an exemplary embodiment of the present application;

[0017] Figure 7 is a structural block diagram of a target detection device according to an exemplary embodiment of the present application;

[0018] Figure 8 Schematic diagram of the structure of an automatic driving device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0020] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0021] Reference Figure 1 , shows a flowchart of the steps of a target detection method according to an exemplary embodiment of the present application. The target detection method of this embodiment includes the following steps:

[0022] S102: Obtain multi-view images of the target area where the autonomous driving device is located and corresponding camera parameters.

[0023] The target area includes a target object to be detected.

[0024] Exemplarily, the target object may be a road element, which may be divided into static elements and dynamic elements, wherein static elements may include lane lines, passable areas, zebra crossing areas, etc., and dynamic elements may include pedestrians, vehicles, etc., which are not limited in this embodiment. Multi-perspective images may be captured by a camera configured on an autonomous driving device, and the multi-perspective images may be a set of images captured with a 360-degree panoramic view centered on the autonomous driving device. Each captured image corresponds to a set of camera parameters, which include camera extrinsics and camera intrinsics. Camera intrinsics are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.; camera extrinsics are parameters in the world coordinate system (i.e., the coordinate system where the autonomous driving device is located), such as the camera's position, rotation direction, etc. The camera intrinsics and camera extrinsics can be used to map points in the world coordinate system to points in the image coordinate system, so that the actual position of the point in the world coordinate system can be determined based on the position of the point in the image coordinate system.

[0025] S104: Extract features from the multi-view images and fuse camera parameters to obtain a three-dimensional feature image of the target area. Perform rasterized semantic segmentation and offset prediction on the three-dimensional feature image to obtain a raster mask image and offset information, respectively.

[0026] The grid mask image is used to indicate information about target grids occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid.

[0027] For example, this step can be implemented by a neural network model, referring to Figure 3 The neural network model may include an encoder and a decoder. The multi-view image and the corresponding camera parameters are input into the neural network model into the encoder. The encoder extracts features from the multi-view image and fuses the camera parameters to output a three-dimensional feature image of the target area. The three-dimensional feature image may be a bird's-eye view image. The three-dimensional feature image includes feature information of the target object, such as the endpoints of the lane lines and the boundary lines of the passable area. The three-dimensional feature image is then input into the decoder of the neural network model. The decoder performs rasterized semantic segmentation and offset prediction on the three-dimensional feature image, and outputs a raster mask image and offset information.

[0028] Among them, rasterization semantic segmentation of the 3D feature image involves rasterizing the 3D feature image to obtain a 3D feature image consisting of multiple grids. The grid occupied by the target object is determined as the target grid. The target grid can be one or more. The target grid is used as the foreground and the other grids as the background to generate a grid mask image. Offset prediction is to establish an independent coordinate system within the area of ​​each target grid to determine the position offset of the target object in the target grid and obtain offset information.

[0029] S106 : Determine the position information of the target object according to the offset information and the information of the target grid in the grid mask image.

[0030] For example, the position information of the target grid in the target area can be determined based on the information of the target grid to serve as the initial position information of the target object. The initial position information is then corrected based on the offset information of the target object in each target grid to obtain the position information of the target object.

[0031] The target detection method of this embodiment obtains multi-view images of a target area where the autonomous driving device is located and corresponding camera parameters, wherein the target area includes a target object to be detected; performs feature extraction on the multi-view images and fuses the camera parameters to obtain a three-dimensional feature image of the target area; performs grid semantic segmentation and offset prediction on the three-dimensional feature image to obtain a grid mask image and offset information, respectively, wherein the grid mask image is used to indicate the target grid occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate the position offset information of the target object in the target grid; and determines the position information of the target object based on the offset information and the target grid information in the grid mask image. In this solution, after determining the target grid occupied by the target object, the offset information of the target object in the target grid is further added, and the target grid information is corrected using the offset information to obtain the position information of the target object. Therefore, compared with directly using the target grid position information as the target object position information, this solution does not reduce the accuracy of the target object position information due to the number or size of the target grid, and can reduce the error of the detected target object position information, thereby making the target object position detection result more accurate.

[0032] Reference Figure 2 , shows a flowchart of the steps of a target detection method according to another exemplary embodiment of the present application. As shown in the figure, this embodiment mainly shows a specific implementation scheme of step S104 of the above embodiment, and this embodiment mainly includes the following steps:

[0033] S202: Obtain multi-view images of the target area where the autonomous driving device is located and corresponding camera parameters.

[0034] The target area includes a target object to be detected.

[0035] It should be noted that step S202 can be implemented with reference to the specific implementation of step S102 above, which will not be described in detail here.

[0036] S204: Input the multi-view images and the corresponding camera parameters into a pre-trained neural network model, extract features from the multi-view images through the encoder of the neural network model, fuse the camera parameters, and output a three-dimensional feature image of the target area.

[0037] For example, referring to Figure 3The neural network model may include an encoder and a decoder, wherein the encoder may extract image features of a multi-view image through a backbone network and a feature pyramid network, and may generally perform feature extraction by downsampling, and the downsampling multiple may be 8 times, 16 times, or 32 times, etc., which is not limited in this embodiment; and then, based on the image feature resolution and camera parameters, a 3D spatial coordinate information feature may be constructed; and a three-dimensional feature image may be output by fusing the image features and the 3D spatial coordinate information feature.

[0038] S206. Input the three-dimensional feature image into the decoder of the neural network model, perform rasterization semantic segmentation on the three-dimensional feature image through the decoder, output a raster mask image, and perform offset prediction on the three-dimensional feature image through the decoder, and output offset information.

[0039] The grid mask image is used to indicate information about target grids occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid.

[0040] For example, referring to Figure 3 The neural network model's decoder can consist of a multi-head attention mechanism, a decoding layer, and a Head layer. After decoding the 3D feature image through the multi-head attention mechanism and decoding layer, the target object is input into the Head layer. The Head layer performs rasterized semantic segmentation and offset prediction, and outputs a raster mask image and offset information. Based on the target object category, such as lane lines, drivable areas, and zebra crossing areas, a Head layer can be assigned to each category to output the corresponding raster mask image and offset information.

[0041] In an optional implementation, a three-dimensional feature image is subjected to rasterization semantic segmentation by a decoder to output a raster mask image, including: rasterizing the three-dimensional feature image to obtain a raster image of the target area, wherein the raster image includes information of each grid and feature information of the target object, the information of each grid is used to indicate the position and size of the grid, and the feature information is used to indicate the target object in the raster image; based on the information of each grid and the feature information of the target object, multiple boundary intersections between the target object and the grid are determined; based on the multiple boundary intersections, information of the target grid occupied by the target object is determined from each grid to generate a raster mask image.

[0042] For example, taking the target object as a lane line, the three-dimensional feature image is rasterized to obtain a raster image of the target area, for example Figure 4As shown in FIG, each grid in the grid image is represented by a dotted line, a solid line represents a target object, a black dot represents a boundary intersection, and the grid marked with an "*" is the target grid. The target object can be a lane boundary, a lane endpoint, or the like. The characteristic information of the target object can be information that can be used to indicate the target object in the grid image, such as lane boundary information or lane endpoint information. This embodiment does not limit this. The grid resolution can be flexibly set by those skilled in the art based on actual conditions. For example, it can be set to 200*200. The width and height of the resolution can be equal or unequal. This embodiment does not limit this. The 3D feature image obtained by the encoder can be used to determine the size of the target area, that is, the size of the grid area, and thus the size information of each grid. Assuming the target area size is 100m*100m, the grid size information is 0.5m*0.5m. The information of each grid indicates the position and size of the grid. For example, the information of each grid can include the number of rows and columns in which each grid is located, and the size information. This embodiment does not limit this. Based on the information of each grid and the characteristic information of the target object, the intersection of the target object and the boundary line of each grid is determined, and multiple boundary intersections can be obtained. Based on the multiple boundary intersections, the grid where the line segment or point is located can be determined as the target grid occupied by the target object according to the line segment or point between adjacent boundary intersections. The target grid is used as the foreground, for example, it can be assigned a value of 1, and the other grids are used as the background, for example, they can be assigned a value of 0, and then a grid mask image can be generated, such as Figure 5 An example raster mask image.

[0043] In this implementation, a grid image of the target area is generated by rasterizing the 3D feature image. Based on the information of each grid in the raster image and the target object's feature information, multiple boundary intersection points between the target object and the grid are determined. Based on these boundary intersection points, the information of the target grid occupied by the target object is determined from each grid, thereby generating a grid mask image. This allows the target grid's position to be initially calculated to determine the target object's location range, narrowing the calculation range for the target object's location and facilitating subsequent, more accurate calculation of the target object's location.

[0044] In an optional implementation, information about the target grid occupied by the target object is determined from each grid based on multiple boundary intersections, including: traversing multiple boundary intersections and determining points between two adjacent boundary intersections as interpolation points; and determining information about the target grid occupied by the target object based on the grid where the interpolation point is located.

[0045] For example, all boundary intersections are traversed to determine each set of two adjacent boundary intersections ( Figure 4Any point on the line connecting the two adjacent boundary intersections (black dots shown in ) is an interpolation point. For example, the interpolation point can be the midpoint of the line connecting the two adjacent boundary intersections, such as Figure 5 The grid where each interpolation point is located is then determined as the target grid occupied by the target object, that is, Figure 4 The grid where the “*” is shown in the figure is the target grid, so that information of multiple target grids can be obtained.

[0046] In this implementation, by determining the interpolation point between two adjacent boundary intersections, and then determining the information of the grid where the interpolation point is located as the information of the target grid occupied by the target object, the target grid occupied by the target object can be accurately determined, and the subsequent calculation of the offset information is convenient.

[0047] In an optional implementation, a decoder performs offset prediction on a three-dimensional feature image and outputs offset information, including: determining reference position information in the target grid based on information of the target grid; determining the position offset of the interpolation point in each target grid relative to the reference position information based on the reference position information and the corresponding interpolation point in the target grid, so as to generate offset information of the target object in the target grid.

[0048] Exemplarily, based on the size information in the information of each target grid, for example, the size information of the target grid is 0.5m*0.5m, a reference position can be determined in each target grid to obtain the reference position information. For example, the reference position can be the center point of the target grid or any point in the target grid. This is not limited in this embodiment. A two-dimensional coordinate system can be established in the target grid with the reference position as the coordinate origin, and the reference position information is the coordinate (0,0). Based on the reference position information and the corresponding interpolation point in the target grid, the position offset information of the interpolation point in each target grid relative to the reference position information can be determined. The position offset information can be composed of first direction offset information and second direction offset information. For example, the first direction offset information can be a lateral offset value determined based on the lateral coordinate axis of the two-dimensional coordinate system, and the second direction offset information can be a longitudinal offset value determined based on the longitudinal coordinate axis of the two-dimensional coordinate system. Taking the reference position as the center point of the target grid as an example, the range of the lateral offset value and the longitudinal offset value are both [-0.25, 0.25]. The first direction offset information finally obtained is the lateral offset information (such as Figure 6 The second direction offset information is the longitudinal offset information (as shown in the left side of the figure) Figure 6 shown on the right side of the diagram).

[0049] In this implementation, by determining the reference position information in the target grid, the position offset of the interpolation point in each target grid relative to the reference position information can be determined based on the reference position information and the corresponding interpolation point in the target grid to generate the offset information of the target object in the target grid. The calculation method is simple, and the offset information can be used to make corrections based on the position information of the target grid to obtain more accurate information of the target object.

[0050] In an optional implementation, the target detection method of this embodiment also includes: inputting the multi-view sample images and the corresponding sample camera parameters into the neural network model to be trained, and training the neural network model according to the first loss function and the second loss function, wherein the first loss function is used to calculate the loss of the grid mask image during the training process, and the second loss function is used to calculate the loss of the offset information during the training process.

[0051] For example, the structure of the neural network model is as follows Figure 3 As shown, the first loss function is used to calculate the loss of the grid mask image during the training process. The loss calculation method of semantic segmentation can be used, such as FocalLoss, etc., which is not limited in this embodiment. The second loss function is used to calculate the loss of the offset information during the training process. The loss calculation method of center point offset prediction regression can be used, such as SmoothL1 Loss, etc., which is not limited in this embodiment. The epoch of training can be flexibly set by those skilled in the art according to actual conditions. For example, the epoch can be 50, which is not limited in this embodiment.

[0052] In this implementation, multi-view sample images and corresponding sample camera parameters are input into the neural network model to be trained, and the neural network model is trained according to a first loss function and a second loss function, wherein the first loss function is used to perform loss calculation on the grid mask image during the training process, and the second loss function is used to perform loss calculation on the offset information during the training process. Through the first loss function and the second loss function, the losses of the prediction of the grid mask image and the prediction of the offset information converge to smaller values, so that when the trained neural network model is used for inference, the output grid mask image and offset information can be made more accurate, thereby improving the accuracy of target object detection.

[0053] S208 : Determine the position information of the target object according to the offset information and the information of the target grid in the grid mask image.

[0054] It should be noted that step S202 can be implemented with reference to the specific implementation of step S102 above, which will not be described in detail here.

[0055] In an optional implementation, the position information of the target object is determined based on the offset information and the information of the target grid in the grid mask image, including: determining the position information of the target grid in the target area based on the information of the target grid and the information of each grid in the grid mask image, as the position information to be corrected of the target object; and performing position correction on the position information to be corrected based on the offset information to obtain the position information of the target object.

[0056] For example, this embodiment uses the lane line as the target object for illustration. Assume that the grid resolution for rasterized semantic segmentation of the 3D feature image is 200*200, and the coordinates of the target area centered on the autonomous driving device determined based on the 3D feature image are [-50,-50,50,50], that is, 50 meters in front, behind, left, and right of the autonomous driving device. The grid mask image and offset information can be obtained, for example, Figure 5 is an exemplary partial grid mask image, Figure 6 is the offset information corresponding to the grid mask image, where the grid numbered 1 in the grid mask image is the target grid. The offset information here includes the lateral offset information ( Figure 6 Center left) and longitudinal offset information ( Figure 6 The target object's position information is calculated as follows:

[0057] 1) According to the information of each grid in the grid mask image and the information of the target grid, determine the row and column number of the target grid:

[0058] col=[col1,col2,...,col N ]

[0059] row=[row1,row2,...,row N ]

[0060] 2) Get the offset information of each target grid, which are:

[0061] offset_x=[offset_x1,offset_x2,...,offset_x N ]

[0062] offset_y=[offset_y1,offset_y2,...,offset_y N ]

[0063] 3) Calculate the position information of the target grid in the target area, i.e., the position information of the target object to be corrected, by using the row and column numbers of the target grid and the grid size information; calculate the position information of the target object (center_x, center_y):

[0064]

[0065]

[0066] Among them, scale_w and scale_h are calculated as follows:

[0067] scale_w=200 / (50-(-50))=2, scale_h=200 / (50-(-50))=2

[0068] Here, the rasterization resolution for performing rasterization semantic segmentation on the three-dimensional feature image can be flexibly set according to the actual scenario. For example, the width and height of the rasterization resolution can be set to be equal or unequal. If the width and height of the rasterization resolution are set to be unequal, the calculated scale_w and scale_h will be unequal. This embodiment does not limit this. The grid size information can be calculated based on scale_w and scale_h.

[0069] Where trans_x and trans_y are 50 meters respectively. If the multi-view image uses a 360-degree panoramic image, the origin of the world coordinate system is located at the center of the raster image. trans_x and trans_y are the horizontal and vertical translations required to translate the origin of the world coordinate system to [-50, -50]. trans_x and trans_y here can be flexibly adjusted based on the size of the target area calculated in the actual scene and the origin of the world coordinate system, and this embodiment does not limit this.

[0070] In addition, when calculating center_x and center_y, since the actual size of the grid is 0.5m (the target area is 100m*100m, and the grid resolution is 200*200), and the reference position of the offset information is the center point of the target grid, it is necessary to add 0.25m to the reference position of the offset information before calculating the offset information. The 0.25 here is an example of this embodiment and can be flexibly adjusted according to the reference position of the offset information and the size of the grid determined in the actual scenario. This embodiment does not limit this.

[0071] In this implementation, the position information of the target grid in the target area is determined according to the information of the target grid and the information of each grid in the grid mask image, which is used as the position information to be corrected of the target object; the offset information is then used to correct the position information to obtain the position information of the target object, thereby improving the accuracy of target object position detection and simplifying the calculation process.

[0072] In this embodiment, multi-view images and corresponding camera parameters are input into a pre-trained neural network model. The neural network model's encoder extracts features from the multi-view images and fuses the camera parameters to output a three-dimensional feature image of the target area. The three-dimensional feature image is input into the neural network model's decoder. The decoder performs rasterized semantic segmentation on the three-dimensional feature image, outputting a raster mask image. The decoder also performs offset prediction on the three-dimensional feature image, outputting offset information. This embodiment utilizes a pre-trained neural network model to predict the raster mask image and offset information, thereby improving data processing efficiency.

[0073] Reference Figure 7 , shows a structural block diagram of a target detection device according to an exemplary embodiment of the present application.

[0074] The target detection device of this embodiment includes an image acquisition module 702 , a prediction module 704 and a position information determination module 706 .

[0075] Among them, the image acquisition module 702 is used to obtain multi-view images of the target area where the autonomous driving equipment is located and the corresponding camera parameters, and the target area includes the target object to be detected; the prediction module 704 is used to extract features from the multi-view images and fuse the camera parameters to obtain a three-dimensional feature image of the target area, and perform rasterized semantic segmentation and offset prediction on the three-dimensional feature image to obtain a grid mask image and offset information respectively, wherein the grid mask image is used to indicate the information of the target grid occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate the position offset information of the target object in the target grid; the position information determination module 706 is used to determine the position information of the target object based on the offset information and the information of the target grid in the grid mask image.

[0076] In an optional implementation, the prediction module 704 is further used to: input the multi-view image and the corresponding camera parameters into a pre-trained neural network model, perform feature extraction on the multi-view image through the encoder of the neural network model, fuse the camera parameters, and output a three-dimensional feature image of the target area; input the three-dimensional feature image into the decoder of the neural network model, perform rasterized semantic segmentation on the three-dimensional feature image through the decoder, output a raster mask image, and perform offset prediction on the three-dimensional feature image through the decoder, and output offset information.

[0077] In an optional implementation, the prediction module 704 is further used to: perform rasterization processing on the three-dimensional feature image to obtain a raster image of the target area, wherein the raster image includes information of each grid and feature information of the target object, the information of each grid is used to indicate the position and size of the grid, and the feature information is used to indicate the target object in the raster image; based on the information of each grid and the feature information of the target object, determine multiple boundary intersections between the target object and the grid; based on the multiple boundary intersections, determine the information of the target grid occupied by the target object from each grid to generate a raster mask image.

[0078] In an optional implementation, the prediction module 704 is further configured to: traverse multiple boundary intersections, and determine points between two adjacent boundary intersections as interpolation points; and determine information about the target grid occupied by the target object based on the grid where the interpolation point is located.

[0079] In an optional implementation, the prediction module 704 is further used to: determine the reference position information in the target grid based on the information of the target grid; determine the position offset of the interpolation point in each target grid relative to the reference position information based on the reference position information and the corresponding interpolation point in the target grid, so as to generate the offset information of the target object in the target grid.

[0080] In an optional implementation, the prediction module 704 is further used to: determine the first direction offset information and the second direction offset information of the interpolation point in each target grid relative to the reference position information based on the reference position information and the corresponding interpolation point in the target grid; determine the position offset of the interpolation point in each target grid relative to the reference position information based on the first direction offset information and the second direction offset information, so as to output the offset information of the target object in the target grid.

[0081] In an optional implementation, the position information determination module 706 is further used to: determine the position information of the target grid in the target area based on the information of the target grid and the information of each grid in the grid mask image, as the position information to be corrected of the target object; based on the offset information, perform position correction on the position information to be corrected to obtain the position information of the target object.

[0082] In an optional implementation, the target detection device of this embodiment also includes a training module, which is used to: input multi-view sample images and corresponding sample camera parameters into the neural network model to be trained, and train the neural network model according to a first loss function and a second loss function, wherein the first loss function is used to calculate the loss of the grid mask image during the training process, and the second loss function is used to calculate the loss of the offset information during the training process.

[0083] The target detection device of this embodiment is used to implement the corresponding target detection methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the target detection device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0084] Reference Figure 8 , showing a structural schematic diagram of an automatic driving device according to an exemplary embodiment of the present application.

[0085] The autonomous driving device of this embodiment includes: a sensor 802, a processor 804, a memory 806, a communication interface, and a communication bus. The sensor 802, the processor 804, the memory 806, and the communication interface communicate with each other via the communication bus.

[0086] Among them, sensor 802 is used to collect environmental data of the environment where the autonomous driving device is located and generate perception data.

[0087] The memory 806 is used to store at least one executable instruction, which enables the processor 804 to perform the operation corresponding to the target detection method described in any of the above multiple method embodiments based on the perception data.

[0088] Optionally, the processor 804 may also make an autonomous driving decision based on the target detection result; and, based on the decision result, drive the autonomous driving device to perform driving task processing.

[0089] Generally, the autonomous driving device also includes a drive device 808 and a control device 810. After the processor 804 determines the corresponding autonomous driving decision, it sends a drive signal to the drive device 808. The drive device 808 uses the drive signal to control the control device 810 (such as the brake, accelerator, steering wheel, etc.) in the autonomous driving device to perform corresponding control operations on the autonomous driving settings, thereby achieving autonomous driving of the autonomous driving device.

[0090] For the convenience and brevity of description, the specific working process of the autonomous driving equipment described above can refer to the corresponding process description in the aforementioned method embodiment and will not be repeated here.

[0091] It should be noted that in the embodiments of the present application, the autonomous driving device may be a vehicle device or a robotic device having the following functions:

[0092] (1) Passenger-carrying function, such as family cars and buses;

[0093] (2) Cargo carrying function, such as ordinary trucks, box trucks, trailer trucks, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures, etc.;

[0094] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol cars, cranes, hoists, excavators, bulldozers, forklifts, rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.;

[0095] (4) Entertainment functions, such as entertainment vehicles, amusement park self-driving devices, balance vehicles, etc.;

[0096] (5) Special rescue functions, such as fire trucks, ambulances, power repair trucks, engineering rescue trucks, etc.

[0097] The target detection solution of the embodiments of the present application can be applied to the above-mentioned autonomous driving equipment.

[0098] An embodiment of the present application also provides a controller including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the memory is used to store computer programs; the processor is used to execute the target detection method described in any of the above-mentioned multiple method embodiments by running the computer program stored in the memory.

[0099] An embodiment of the present application further provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection method as described in any one of the above-mentioned method embodiments.

[0100] An embodiment of the present application further provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to the target detection method described in any one of the above-mentioned method embodiments.

[0101] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0102] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0103] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0104] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A target detection method, characterized in that: include: Acquire multi-view images of a target area where the autonomous driving device is located and corresponding camera parameters, wherein the target area includes a target object to be detected; Performing feature extraction on the multi-view images and fusing the camera parameters to obtain a three-dimensional feature image of the target area, performing grid semantic segmentation and offset prediction on the three-dimensional feature image to obtain a grid mask image and offset information, respectively, wherein the grid mask image is used to indicate information about target grids occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid; The position information of the target object is determined according to the offset information and the information of the target grid in the grid mask image.

2. The method according to claim 1, characterized in that The step of extracting features from the multi-view images and fusing the camera parameters to obtain a three-dimensional feature image of the target area, and performing rasterization semantic segmentation and offset prediction on the three-dimensional feature image to obtain a raster mask image and offset information includes: Inputting the multi-view images and corresponding camera parameters into a pre-trained neural network model, performing feature extraction on the multi-view images through an encoder of the neural network model, and fusing the camera parameters to output a three-dimensional feature image of the target area; The three-dimensional feature image is input into a decoder of a neural network model, the three-dimensional feature image is subjected to rasterization semantic segmentation by the decoder, a raster mask image is output, and the three-dimensional feature image is subjected to offset prediction by the decoder, and offset information is output.

3. The method according to claim 2, characterized in that The step of performing rasterization semantic segmentation on the three-dimensional feature image by the decoder and outputting a raster mask image comprises: performing rasterization processing on the three-dimensional feature image to obtain a raster image of the target area, wherein the raster image includes information of each grid and feature information of the target object, wherein the information of each grid is used to indicate the position and size of the grid, and the feature information is used to indicate the target object in the raster image; Determining a plurality of boundary intersections between the target object and the grids based on information of each grid and characteristic information of the target object; According to the plurality of boundary intersection points, information of the target grid occupied by the target object is determined from each of the grids to generate the grid mask image.

4. The method according to claim 3, characterized in that The step of determining information of a target grid occupied by the target object from each grid according to the plurality of boundary intersection points includes: Traversing the plurality of boundary intersection points, and determining the points between two adjacent boundary intersection points as interpolation points; According to the grid where the interpolation point is located, information about the target grid occupied by the target object is determined.

5. The method according to claim 4, characterized in that The performing offset prediction on the three-dimensional feature image by the decoder and outputting offset information includes: Determining reference position information in the target grid according to information of the target grid; According to the reference position information and the corresponding interpolation points in the target grid, a position offset of the interpolation point in each target grid relative to the reference position information is determined to generate offset information of the target object in the target grid.

6. The method according to claim 1, characterized in that The determining the position information of the target object according to the offset information and the information of the target grid in the grid mask image includes: determining, based on the information of the target grid and the information of each grid in the grid mask image, the position information of the target grid in the target area as the position information to be corrected of the target object; Based on the offset information, the position information to be corrected is corrected to obtain the position information of the target object.

7. The method according to claim 2, characterized in that The method further comprises: The multi-view sample images and the corresponding sample camera parameters are input into the neural network model to be trained, and the neural network model is trained according to a first loss function and a second loss function, wherein the first loss function is used to calculate the loss of the grid mask image during the training process, and the second loss function is used to calculate the loss of the offset information during the training process.

8. A target detection device, characterized in that: include: An image acquisition module, configured to acquire multi-view images of a target area where the autonomous driving device is located and corresponding camera parameters, wherein the target area includes a target object to be detected; a prediction module, configured to extract features from the multi-view images and fuse the camera parameters to obtain a three-dimensional feature image of the target area, perform gridded semantic segmentation and offset prediction on the three-dimensional feature image, and obtain a grid mask image and offset information, respectively, wherein the grid mask image is used to indicate information about target grids occupied by the target object in each grid of the three-dimensional feature image, and the offset information is used to indicate position offset information of the target object in the target grid; A position information determining module is used to determine the position information of the target object according to the offset information and the information of the target grid in the grid mask image.

9. A controller comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 7 by running the computer program stored in the memory.

10. An automatic driving device, characterized in that: include: A sensor, a processor, a memory, a communication interface and a communication bus, wherein the sensor, the processor, the memory and the communication interface communicate with each other via the communication bus; The sensor is used to collect environmental data of the environment in which the autonomous driving device is located and generate perception data; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 7 based on the perception data.

Images