Target detection method and device, vehicle and storage medium
By predicting the offset between the preset position points and the target object in the three-dimensional space on the BEV view, the problem of detecting target objects in the complex topological shapes in the prior art is solved, and efficient and accurate detection of target objects such as lane lines is achieved.
Patent Information
- Application Number
- CN202311071271.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-23
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art is difficult to effectively detect and identify static target objects of complex topological shapes, such as lane lines, in autonomous driving scenarios, and traditional methods lack adaptability to curved or U-shaped lane lines.
By predicting the offset between the preset position point and the target object in the three-dimensional space on the BEV view, the offset between the first foreground point and the second foreground point is determined using the preset space grid and the target detection model, sub-pixel regression of the target object is achieved.
This method can accurately predict target objects of any topological shape, provides a lightweight modeling method, has good platform adaptability, and improves detection efficiency.
Smart Images
Figure CN119516507A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of autonomous driving of vehicles, and in particular to a target detection method, device, vehicle and storage medium. Background Art
[0002] Target detection is to accurately locate and identify the object to be detected, and determine the location and specific category of the target. In the scenario of autonomous driving, it is necessary to perceive the static elements (such as lane lines) in the vehicle's driving environment.
[0003] In the process of target detection, in order to simplify the complexity of the overall detection system and improve the robustness of the detection results, target perception can be performed based on the BEV (Birds Eye View) view. Compared with the heavy post-processing operations required for traditional two-dimensional image perception solutions, BEV perception can directly output detection results that meet downstream needs. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides a target detection method, device, vehicle and storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a target detection method, including:
[0006] Acquire a target image of a target environment where the vehicle is located;
[0007] Determine a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image through a target detection model, wherein the first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in a target space, and the second foreground point is a real pixel position point of the target object in the target space, and the target space is a preset three-dimensional space corresponding to the target environment;
[0008] For each of the first foreground points, the target object is detected according to the first foreground point and the offset.
[0009] Optionally, the preset three-dimensional space includes a preset spatial grid, the preset position point includes a vertex of each grid in the preset spatial grid, and the determining a plurality of first foreground points through a target detection model according to the target image includes:
[0010] The first foreground point is determined from the plurality of vertices according to the target image by using a target detection model.
[0011] Optionally, determining the offset between each first foreground point and a second foreground point by using a target detection model according to the target image includes:
[0012] For each of the first foreground points, outputting the offsets of the first foreground point and the corresponding second foreground point in a plurality of preset spatial directions respectively according to the target image through the target detection model;
[0013] The detecting the target object according to the first foreground point and the offset comprises:
[0014] A predicted foreground point corresponding to the target object is determined according to the position of the first foreground point and the corresponding offsets in the plurality of preset spatial directions.
[0015] Optionally, the target detection model is pre-trained in the following manner:
[0016] Acquire a sample image corresponding to the target object;
[0017] After performing BEV feature extraction on the sample image, the sample image is input into a preset detection model to be trained, and a first prediction result and a second prediction result are output through the preset detection model, wherein the first prediction result is a foreground position point of the target object determined from a plurality of preset position points in the target space after performing image segmentation on the sample image through the preset detection model, and the second prediction result is a predicted position point of the target object in the target space predicted by the preset detection model;
[0018] Determining a foreground grid through which the target object passes from a preset spatial grid corresponding to the target space;
[0019] For each foreground grid, the vertices corresponding to the foreground grid are used as first label data of the image segmentation result, and the actual position point of the target object in the foreground grid is used as second label data of the model prediction result;
[0020] Calculating a loss value of the preset detection model by using a preset loss function according to the first prediction result, the second prediction result, the first label data, and the second label data;
[0021] After the preset detection model is trained according to the loss value, the target detection model is obtained.
[0022] Optionally, the method further comprises:
[0023] Performing feature extraction on the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image;
[0024] For each scale, determining a semantic enhancement feature of the scale according to the two-dimensional feature map of the scale;
[0025] Determine the bird's-eye view BEV fusion feature of the target object according to the semantic enhancement feature at each scale;
[0026] The step of determining a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image by using a target detection model comprises:
[0027] After the BEV fusion feature is input into the target detection model, a plurality of the first foreground points and an offset between each of the first foreground points and a second foreground point are determined.
[0028] Optionally, the extracting features of the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image includes:
[0029] After the target image is input into the feature pyramid network FPN, the multi-scale two-dimensional feature map is obtained.
[0030] Optionally, determining the semantic enhancement feature of the scale according to the two-dimensional feature map of the scale includes:
[0031] After inputting the target feature map of this scale into the semantic segmentation network, the semantic segmentation features of this scale are obtained;
[0032] According to the target feature map of this scale and the speech segmentation features of this scale, the semantic enhancement features of this scale are determined through a multi-head attention mechanism model.
[0033] Optionally, determining the bird's-eye view BEV fusion feature of the target object according to the semantic enhancement feature at each scale includes:
[0034] For the semantic enhancement features of each scale, converting the semantic enhancement features of the scale into BEV features through a preset view transformation method;
[0035] After the BEV features at multiple scales are fused, the BEV fused features are obtained.
[0036] According to a second aspect of an embodiment of the present disclosure, there is provided a target detection device, including:
[0037] An acquisition module is configured to acquire a target image of a target environment where the vehicle is located;
[0038] a determination module configured to determine a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image through a target detection model, wherein the first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in a target space, and the second foreground point is a real pixel position point of the target object in the target space, and the target space is a preset three-dimensional space corresponding to the target environment;
[0039] The detection module is configured to detect the target object according to the first foreground point and the offset for each of the first foreground points.
[0040] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising:
[0041] processor;
[0042] a memory for storing processor-executable instructions;
[0043] Wherein, the processor is configured to execute the steps of the method described in the first aspect of the present disclosure.
[0044] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the target detection method provided in the first aspect of the present disclosure are implemented.
[0045] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: accurate prediction of the target object is achieved by predicting the offset between a preset position point in the three-dimensional space and the target object (such as a lane line) on the BEV view, thereby providing a sub-pixel regression solution for the target object, and this method can predict target objects of any topological shape, and the modeling method is also lightweight enough and has good platform adaptability.
[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0048] Figure 1 The figure is a flow chart of a target detection method according to an exemplary embodiment.
[0049] Figure 2 is based on Figure 1 The illustrated embodiment shows a flow chart of a target detection method.
[0050] Figure 3 The figure is a schematic diagram of a process of image feature enhancement according to an exemplary embodiment.
[0051] Figure 4 It is a schematic diagram of a multi-scale BEV feature fusion process according to an exemplary embodiment.
[0052] Figure 5 The figure is a schematic diagram showing modeling of a lane line in a BEV grid according to an exemplary embodiment.
[0053] Figure 6 The figure is a flowchart of a method for training a target detection model according to an exemplary embodiment.
[0054] Figure 7 The figure is a block diagram of a target detection device according to an exemplary embodiment.
[0055] Figure 8 is based on Figure 7 The illustrated embodiment shows a block diagram of a target detection device.
[0056] Fig. 9 It is a schematic diagram of a functional block diagram of a vehicle showing an exemplary embodiment. DETAILED DESCRIPTION
[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0058] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.
[0059] The present disclosure is mainly applied to the scenario of automatic perception of static elements on the road in the autonomous driving scenario. The static elements may be lane lines, for example, and the spatial coordinate values of the lane lines in 3D space may be output based on the image perception of the BEV view.
[0060] Taking lane line detection as an example, the process of detecting lane lines based on BEV views involves modules such as efficient BEV feature extraction, 3D lane line modeling, and lane line regression supervision. The following lane line modeling methods are provided in related technologies:
[0061] Method 1: anchor-based: Borrowing the concept of anchor in target detection, according to the distribution of lane lines in the scene, the network is set with an initial prior for the lane line anchor distribution (learnable attributes can be added to the anchor), and the subsequent network implements lane line regression based on the anchor prior. This method is direct in modeling and can converge quickly, but it is also more dependent on the lane line anchor, and is not very adaptable to lane lines of arbitrary shapes such as curved and U-shaped.
[0062] Method 2, segment-based: Segmentation technology is naturally capable of adapting to lane lines of any shape. After adding offset prediction and lane line instance differentiation, it can adapt to lane lines of any shape. However, it is more dependent on segmentation quality, and lane line sub-pixel regression cannot adapt to lane lines of any shape.
[0063] Method 3, vector-based: The lane line is constructed as a set of curve vector points. The lane line instance is constructed through a sequence of key points. However, its regressed GT (true value) requires more complex modeling steps. For example, in MapTR, the topological structure of the lane line is first regressed, and then the specific and accurate position of each node is predicted.
[0064] In summary, among the lane line modeling methods provided in the relevant technologies, the anchor-based and segment-based solutions divide the lane lines along the Y-axis direction, and then regress the X-axis and Z-axis offsets of the lane lines. However, this prediction method requires that the lane lines are distributed in a single direction along the Y-axis, that is, there is no situation where one Y value corresponds to multiple X and Z values, which limits its adaptability in complex lane line scenarios. The vector-based solution can directly convert the lane line prediction problem into the lane line key point prediction problem, so that the road topology structure of the lane line can be directly output without complex post-processing operations, but the problem is that this type is mostly built with a transformer structure and requires support from a larger computing power platform.
[0065] Therefore, in order to solve the above problems, the present disclosure provides a target detection method, device, vehicle and storage medium. The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.
[0066] Figure 1 is a flow chart of a target detection method according to an exemplary embodiment, which can be applied to vehicles, such as Figure 1 As shown, the method comprises the following steps:
[0067] In step S101, a target image of a target environment in which a vehicle is located is acquired.
[0068] The target image may be environment images at different locations around the vehicle captured by multiple surround view cameras disposed on the vehicle.
[0069] In step S102, a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point are determined according to the target image through a target detection model, wherein the first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in a target space, and the second foreground point is a real pixel position point of the target object in the target space, and the target space is a preset three-dimensional space corresponding to the target environment.
[0070] The preset three-dimensional space may include preset space grids, each of which may be a rectangular grid, and the preset position point may include a vertex of each grid.
[0071] For example, the preset spatial grid can be understood as a plurality of grids obtained by gridding the bird's-eye view corresponding to the target environment in which the vehicle is located. Therefore, the plurality of preset spatial grids can also be referred to as BEV grids. Taking the target object to be detected as a lane line as an example, the lane line can be drawn in the BEV grid (here the height effect in the perception space can be temporarily ignored, and the lane line is only observed from the perspective of the bird's-eye view), and the lane line will fall within certain grids of the BEV grid. In this way, the grid that falls within the lane line can be defined as the foreground grid, and the grid that the lane line does not fall within can be defined as the background, where one of the vertices corresponding to the foreground grid is the first foreground point output by the target detection model. The specific position point where the lane line falls in the foreground grid (the position point may be located within the grid, on the edge of the grid, or at a vertex) is the second foreground point.
[0072] In addition, the offset between the first foreground point and the second foreground point may include offsets corresponding to a plurality of preset spatial directions, and the preset spatial directions may be, for example, the X-axis direction, the Y-axis direction, and the Z-axis direction on a preset three-dimensional coordinate system.
[0073] In one possible implementation, feature extraction may be performed on the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image, and then for each scale, the semantic enhancement feature of the scale may be determined based on the two-dimensional feature map of the scale; the bird's-eye view BEV fusion feature of the target object may be determined based on the semantic enhancement feature of each scale. In this way, in the process of executing this step, the BEV fusion feature may be input into the target detection model, and then the target detection model may output a plurality of the first foreground points and the offset of each of the first foreground points from the second foreground point.
[0074] In step S103, for each of the first foreground points, the target object is detected according to the first foreground point and the offset.
[0075] After detecting the first foreground point and the offset between the first foreground point and the second foreground point, the target detection model can determine the second foreground point based on the position of the first foreground point and the offset. The second foreground point is the actual position of the lane line in the BEV grid. In this way, each position point of the lane line in the BEV grid can be determined, thereby realizing the detection of the lane line.
[0076] The above method is used to accurately predict the target object by predicting the offset between the preset position point and the target object (such as the lane line) in the three-dimensional space on the BEV view, thereby providing a sub-pixel regression scheme for the target object. This method can predict target objects of any topological shape, and the modeling method is lightweight enough and has good platform adaptability. In addition, the spatial position of the target object is determined based on the offset between the preset position point and the real pixel position point of the target object, which can also improve the detection efficiency compared with the complex target detection algorithm in the related technology.
[0077] Figure 2 is based on Figure 1 The embodiment shown is a flow chart of a target detection method, such as Figure 2 As shown, the method comprises the following steps:
[0078] In step S201, a target image of a target environment in which a vehicle is located is acquired.
[0079] In this step, multiple surround view cameras arranged around the vehicle body may respectively collect environmental images in different directions around the vehicle, and the target image is the environmental image.
[0080] In step S202, feature extraction is performed on the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image.
[0081] In this step, the target image may be input into FPN (Feature Pyramid Networks) to obtain the multi-scale two-dimensional feature map.
[0082] Among them, the model structure of the FPN model includes a backbone network and an upsampling network connected to the backbone network, the upsampling network includes multiple layers, and each upsampling layer can output the two-dimensional feature map of different scales. In this way, after the target image is input into the FPN model, different upsampling layers of the FPN model can be used to output two-dimensional feature maps of different scales. The specific implementation method of the present disclosure for multi-scale feature extraction of the target image based on the FPN model can refer to the description in the relevant literature and is not specifically limited here.
[0083] In step S203, for each scale, a semantic enhancement feature of the scale is determined according to the two-dimensional feature map of the scale.
[0084] In this step, the target feature map of the scale can be input into the semantic segmentation network to obtain the semantic segmentation features of the scale; the semantic enhancement features of the scale are determined through a multi-head attention mechanism model based on the target feature map of the scale and the speech segmentation features of the scale.
[0085] The semantic segmentation network may be in the form of an auxiliary segmentation head (aux-head), for example.
[0086] For example, taking the target object to be detected as a lane line, Figure 3 is a schematic diagram of a process of image feature enhancement according to an exemplary embodiment. Figure 3 As shown, for the two-dimensional feature map of each scale, the two-dimensional feature map can be input into the semantic segmentation network and the multi-head attention mechanism model respectively, wherein the semantic segmentation network can perform semantic segmentation on the two-dimensional feature map to obtain the semantic segmentation features of the lane line, and the semantic segmentation features of the lane line can be input into the multi-head attention mechanism model as the key (Key) and value (Value) corresponding to the multi-head attention mechanism model, and the two-dimensional feature map can be used as the query (Query) corresponding to the multi-head attention mechanism model. In this way, the multi-head attention mechanism model can perform feature enhancement on the two-dimensional feature map according to the Key and Value input by the semantic segmentation network, and use the two-dimensional feature map as the query, to obtain the semantic enhancement features of the scale. The above examples are only illustrative, and the present disclosure is not limited to this.
[0087] In this way, by adding aux-head in the 2D feature space, the corresponding spatial features are enhanced, thereby improving the accuracy of subsequent target detection.
[0088] In step S204, the bird's-eye view BEV fusion features of the target object are determined according to the semantic enhancement features at each scale.
[0089] In a possible implementation, the semantic enhancement features at each scale may be converted into BEV features by a preset view transformation method; and the BEV features at multiple scales may be fused to obtain the BEV fused features.
[0090] The view transform here refers to the process of transforming the target object to be identified from the world coordinate system to the camera coordinate system. The view transform can be achieved by translating, rotating, and scaling the coordinates of the target object.
[0091] In one implementation of the present disclosure, the conversion of two-dimensional semantic enhancement features to three-dimensional BEV features can be achieved through the preset view transformation method shown in the following formula (1).
[0092] In another possible implementation, according to actual business needs, the semantic features of the two-dimensional feature map may not be enhanced. In this case, the two-dimensional feature map of each scale can be directly converted into a three-dimensional BEV feature through a preset view transformation method, and the multi-scale BEV features can be further fused to obtain the BEV fused feature.
[0093] Taking the example of fusing multi-scale semantic enhancement features to obtain the BEV fusion feature, the following describes the conversion of two-dimensional features to three-dimensional features and the specific implementation method of feature fusion.
[0094] The multi-scale features of images can better adapt to the different sizes and semantic expressions of the target objects to be detected. Therefore, extracting BEV features in multiple scale spaces can greatly utilize the expressive power of multi-scale features.
[0095] For example, suppose the BEV feature expression of a certain scale is expressed as The two-dimensional semantic enhancement features of each scale can be mapped to three-dimensional BEV features by the following formula:
[0096]
[0097] in, represents the BEV feature of the i-th scale, represents the semantic enhancement feature of scale i, proj(...) represents the spatial projection process from 3D to 2D, K represents the camera intrinsic parameter, [R|t] represents the camera extrinsic parameter, P represents the 3D spatial projection point, and bilinear_sample(...) represents the bilinear sampling process.
[0098] In this way, based on the above formula (1), the BEV features of each scale can be obtained, and then the BEV features of multiple scales can be fused to obtain the BEV fused features.
[0099] For example, Figure 4 FIG. 1 is a schematic diagram of a multi-scale BEV feature fusion process according to an exemplary embodiment. Figure 4 As shown, the BEV features of each scale (i.e. Figure 4 After the BEV features 1, BEV features 2, and BEV features 3 in the image are matched in feature dimensions (or optimized and isolated in features, for example, feature dimension matching can be achieved through the Convblock module, where Conv Block is a basic module in ResNet50 and its function is to change the dimension of features), the BEV features of multiple scales are concatenated (for example, feature concatenation can be performed in the channel dimension through the concat model. In deep learning, concat is an operation that concatenates two or more tensors in a certain dimension), and the concatenated features are input into the Convolutional Block Attention Module (CBAM) that combines space and channels, and the BEV fusion features are output through CBAM. The CBAM module can be described as the following formula (2):
[0100] F bev '=F bev *σ([AvgPool(F bev ),MaxPool(F bev )]) (2)
[0101] Among them, F bev ' represents the BEV fusion feature, F bev represents the features after concatenating BEV features at multiple scales, σ(...) is the activation function, AvgPool(F bev ) indicates the bev Perform mean sampling, MaxPool(F bev ) indicates the bev Perform maximum value sampling.
[0102] The present invention constructs multi-scale BEV features by projecting and sampling from three-dimensional space to multi-scale two-dimensional space, and then generates a BEV feature map containing rich semantics through a spatial information optimization network, which greatly improves the efficiency of feature conversion in different spaces. In this way, when realizing target detection based on the BEV feature map, while improving detection accuracy, the target detection efficiency can also be improved.
[0103] In step S205, a first foreground point is determined from the plurality of vertices according to the target image by using a target detection model.
[0104] The first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in the target space. The target space is a preset three-dimensional space corresponding to the target environment, and the preset three-dimensional space may include, for example, a preset spatial grid corresponding to the BEV fusion feature, and the preset position point includes the vertex of each grid in the preset spatial grid.
[0105] In a possible implementation, the preset spatial grid may be pre-generated in the following manner:
[0106] First, determine the vehicle's perception range, for example, 120 degrees forward, 30 degrees backward, and 30 degrees left and right. Then set the length, width, and height of each grid according to the prediction accuracy requirements, for example, 0.5m per grid in the longitudinal direction, 0.2m per grid in the horizontal direction, and 0.1m in height, so that the preset spatial grid can be obtained.
[0107] In this way, the point P in formula (1) is the vertex of each grid in the preset spatial grid, and the pixel point corresponding to each vertex on the BEV feature map can be determined according to formula (1). Taking the target object as a lane line as an example, the first foreground point is the target vertex corresponding to the pixel point belonging to the lane line among the multiple vertices predicted by the target detection model.
[0108] In a possible implementation of this step, after the BEV fusion feature is input into the target detection model, the target detection model can output multiple first foreground points. For example, the target detection model can determine the first foreground point from multiple vertices in a preset spatial grid through image segmentation.
[0109] For example, assuming that the target object is still the lane line, the BEV fusion feature can be expressed as a feature map of N*C*H*W (which can be referred to as the BEV feature map for short. For example, the VGG16 model can be used to extract features to obtain a feature map of size N*C*H*W), wherein N, C, H, and W refer to BatchSize (the number of samples selected for one training), the number of feature map channels, the feature map height, and the feature map width, respectively. After binary segmentation of the BEV feature map, a binary map is obtained (for example, the pixel value in the binary map is 0 or 1), wherein 0 represents the background and 1 represents the foreground (i.e., the pixel point of the lane line). In this way, the pixel points of the lane line on the BEV feature map can be determined based on the binary segmentation, and further based on the correspondence between each vertex in the preset spatial grid and the pixel point on the BEV feature map, it can be determined which vertices belong to the pixel points of the lane line (i.e., the first foreground point). This is only an example and the present disclosure is not limited to this.
[0110] In step S206, for each of the first foreground points, the target detection model outputs the offsets of the first foreground point and the corresponding second foreground point in a plurality of preset spatial directions according to the target image.
[0111] The preset spatial direction may be, for example, an X-axis direction, a Y-axis direction, and a Z-axis direction on a preset three-dimensional coordinate system.
[0112] The second foreground point is a real pixel position point of the target object in the target space. For example, if the target object is a lane line, the second foreground point is the actual pixel position point of the lane line in the preset space grid.
[0113] For example, Figure 5 FIG. 1 is a schematic diagram showing a modeling of a lane line in a BEV grid according to an exemplary embodiment. Figure 5 As shown in the figure, a lane line is drawn in the set BEV grid (here we temporarily ignore the height effect in the perception space and only observe the lane line on the top view), and the lane line will fall within some grids of the BEV grid. These grids are defined as the foreground, and the corresponding grids without lane lines are defined as the background, so we can get a binary mask that distinguishes the foreground and background. Figure 5 Point G i That is, it is the vertex of one of the foreground grids (equivalent to the first foreground point predicted by the target detection model in the lane line detection scene), and the first foreground point G i Draw a straight line l2 that intersects the lane line at point k i , so point k i That is, the second foreground point (i.e., the actual pixel position point where the lane line falls within the BEV grid), point k i The corresponding tangent is l1. It is necessary to ensure that l2 is perpendicular to l1 and G i ,k i The distance is the shortest, and the first foreground point G is determined based on this condition. i The offset regression object is the second foreground point k i . Then the offset of these two points in the X and Y directions must satisfy the following relationship:
[0114]
[0115]
[0116] in, Indicates point k i The corresponding coordinate value on the X-axis is Indicates point G i The corresponding coordinate value on the X-axis is Indicates point G i With point k i The offset in the X direction, Indicates point k i The corresponding coordinate value on the Y axis is Indicates point G i The corresponding coordinate value on the Y axis is Indicates point G i With point k i The offset in the Y direction.
[0117] The above completes the expression of lane lines in the X and Y directions. Similarly, for the prediction of the Z axis, the offset in the Z axis direction can be added to the reference height:
[0118]
[0119] in, Indicates point k i The corresponding coordinate value on the Z axis is Indicates point G i With point k i The offset in the Z direction. 0.1 is the base height of the current BEV grid.
[0120] In this way, by executing this step to determine each first foreground point and the corresponding second foreground point, and the corresponding offsets in the X, Y, and Z axis directions respectively, the three-dimensional spatial coordinates of the predicted foreground point can be determined according to the position of the first foreground point and the offset. The preset foreground point is the pixel point of the lane line predicted by the target detection model.
[0121] In step S207, a predicted foreground point corresponding to the target object is determined according to the position of the first foreground point and the corresponding offsets in the plurality of preset spatial directions.
[0122] By adopting the above method, accurate prediction of the target object is achieved by predicting the offset between the preset position point in the preset three-dimensional space and the target object (such as the lane line) on the BEV view. In addition, this method can predict the target object of any topological shape, and the modeling method is lightweight enough and has good platform adaptability.
[0123] The training process of the target detection model is described below.
[0124] Figure 6 is a flow chart of a method for training a target detection model according to an exemplary embodiment. Figure 6 As shown, the method comprises the following steps:
[0125] In step S601, a sample image corresponding to the target object is obtained.
[0126] For example, when the target object is a lane line, the sample image may include a pre-collected scene image containing the lane line. In addition, in order to improve the generalization ability of the target detection model to be trained, the sample image may include scene images of lane lines of various shapes.
[0127] In step S602, after the sample image is subjected to BEV fusion feature extraction, it is input into a preset detection model to be trained, and a first prediction result and a second prediction result are output by the preset detection model. The first prediction result is the foreground position point of the target object determined from multiple preset position points in the target space after the sample image is segmented by the preset detection model, and the second prediction result is the predicted position point of the target object in the target space predicted by the preset detection model.
[0128] In one implementation method, the multi-scale two-dimensional features of the sample image can first be extracted through the FPN network model, and then the two-dimensional features of each scale can be semantically enhanced and converted into three-dimensional BEV features. After that, the multi-scale BEV features can be fused to obtain the BEV fused features.
[0129] In addition, the first prediction result here can be analogous to the first foreground point mentioned above, and the second prediction result can be analogous to the predicted foreground point mentioned above.
[0130] In step S603, a foreground grid through which the target object passes is determined from preset spatial grids corresponding to the target space.
[0131] Here, the actual pixel position point of the target object on the sample image can be used as a training label, and the grid that the target object actually passes through in the preset spatial grid can be used as the foreground grid.
[0132] In step S604, for each foreground grid, the vertices corresponding to the foreground grid are used as first label data of the image segmentation result, and the actual position point of the target object in the foreground grid is used as second label data of the model prediction result.
[0133] Here, any vertex of the foreground grid can be used as the first label data of the image segmentation result, but usually for each foreground grid, the vertex at the same position is selected as the first label data corresponding to each grid. For example, assuming that any vertex is the vertex in the upper left corner of the four vertices of the rectangular grid, then for each foreground grid, the vertex in the upper left corner is selected as the first label data. This is only an example, and the present disclosure does not limit this.
[0134] In step S605, a loss value of the preset detection model is calculated using a preset loss function according to the first prediction result, the second prediction result, the first label data, and the second label data.
[0135] In step S606, the preset detection model is trained according to the loss value to obtain the target detection model.
[0136] For example, the preset loss function may be, for example, the following formula (3):
[0137]
[0138] Among them, L bev_lane Represents the loss value of the preset detection model, represents the i-th foreground grid in the BEV grid, k i represents the second label data (i.e., the real pixel points of the target object, or called GT true value), k i ′ represents the second prediction result, G, G' represent the first label data and the first prediction result on the BEV grid respectively, α, β, γ are all preset coefficients, SmoothL1(...) and L1(...) and BCE(...) are all loss functions; among them, the L1 loss function, also known as the mean absolute error (MAE), refers to the average value of the absolute difference between the model prediction value and the true value, the Smooth L1 loss function is a function that combines the L1 loss and the L2 loss, the L2 loss function is also called the least square error (LSE), and the BCE (Binary Cross Entropy) loss function is also a commonly used loss function, which measures the difference between the model prediction result and the true label. T(k i ) = k i -k i-1 is the first-order smoothing amount of the lane line. The first term in the loss function shown in formula (3) is used to constrain the offset of the lane line when the foreground is known, the second term is used to constrain the relative offset of the points on the lane line, so that the lane line has the ability to perceive the overall 3D shape, and the third term is a binary classification constraint for distinguishing the foreground and background of the lane line.
[0139] In this way, based on the above formula (3), the loss value of the preset detection model can be calculated, and then the preset detection model can be trained through the back propagation algorithm to obtain the target detection model.
[0140] It should be noted that, as shown in formula (3), in the training process of the target detection model in the present disclosure, in order to make the network training stable and convergent, the constraints related to the offset are enabled only in the foreground area. At the same time, in order to associate the various points on the lane line, the constraints on T(ki ) smoothing loss makes the perception of lane lines subject to the constraints of the overall 3D shape, so that the trained object detection model can adapt to the detection of lane lines of arbitrary shapes. For 3D lane line detection scenarios, a 3D lane line sub-pixel (sub-pixel refers to the pixel points of the target object except the pixel points corresponding to the grid vertices) regression scheme is provided, which has the advantages of unique label directionality and fast convergence speed.
[0141] Figure 7 is a block diagram of a target detection device according to an exemplary embodiment. Figure 7 As shown, the device comprises:
[0142] An acquisition module 701 is configured to acquire a target image of a target environment in which a vehicle is located;
[0143] The determination module 702 is configured to determine a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image through a target detection model, wherein the first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in a target space, and the second foreground point is a real pixel position point of the target object in the target space, and the target space is a preset three-dimensional space corresponding to the target environment;
[0144] The detection module 703 is configured to detect the target object according to the first foreground point and the offset for each of the first foreground points.
[0145] Optionally, the preset three-dimensional space includes a preset spatial grid, the preset position points include vertices of each grid in the preset spatial grid, and the determination module 702 is configured to determine the first foreground point from a plurality of the vertices according to the target image through a target detection model.
[0146] Optionally, the determination module 702 is configured to output, for each of the first foreground points, offsets of the first foreground points and the corresponding second foreground points in a plurality of preset spatial directions respectively according to the target image through the target detection model;
[0147] The detection module 703 is configured to determine a predicted foreground point corresponding to the target object according to the position of the first foreground point and the corresponding offsets in the plurality of preset spatial directions.
[0148] Optionally, the target detection model is pre-trained in the following manner:
[0149] Acquire a sample image corresponding to the target object;
[0150] After performing BEV feature extraction on the sample image, the sample image is input into a preset detection model to be trained, and a first prediction result and a second prediction result are output through the preset detection model, wherein the first prediction result is a foreground position point of the target object determined from a plurality of preset position points in the target space after performing image segmentation on the sample image through the preset detection model, and the second prediction result is a predicted position point of the target object in the target space predicted by the preset detection model;
[0151] Determining a foreground grid through which the target object passes from a preset spatial grid corresponding to the target space;
[0152] For each foreground grid, the vertices corresponding to the foreground grid are used as first label data of the image segmentation result, and the actual position point of the target object in the foreground grid is used as second label data of the model prediction result;
[0153] Calculating a loss value of the preset detection model by using a preset loss function according to the first prediction result, the second prediction result, the first label data, and the second label data;
[0154] After the preset detection model is trained according to the loss value, the target detection model is obtained.
[0155] Optionally, Figure 8 is based on Figure 7 The block diagram of a target detection device shown in the embodiment shown in FIG. Figure 8 As shown, the device comprises:
[0156] The BEV feature extraction module 704 is configured to extract features from the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image; for each scale, determine the semantic enhancement feature of the scale according to the two-dimensional feature map of the scale; and determine the bird's-eye view BEV fusion feature of the target object according to the semantic enhancement feature of each scale;
[0157] The determination module 702 is configured to determine a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point after inputting the BEV fusion feature into the target detection model.
[0158] Optionally, the BEV feature extraction module 704 is configured to input the target image into a feature pyramid network FPN to obtain the multi-scale two-dimensional feature map.
[0159] Optionally, the BEV feature extraction module 704 is configured to input the target feature map of the scale into the semantic segmentation network to obtain the semantic segmentation features of the scale; and determine the semantic enhancement features of the scale through a multi-head attention mechanism model based on the target feature map of the scale and the speech segmentation features of the scale.
[0160] Optionally, the BEV feature extraction module 704 is configured to convert the semantic enhancement features of each scale into BEV features by a preset view transformation method; and obtain the BEV fused features after performing feature fusion on the BEV features of multiple scales.
[0161] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0162] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the target detection method provided by the present disclosure are implemented.
[0163] Fig. 9 is a block diagram of a vehicle 900 according to an exemplary embodiment. For example, the vehicle 900 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 900 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.
[0164] Reference Fig. 9 , the vehicle 900 may include various subsystems, for example, an infotainment system 910, a perception system 920, a decision control system 930, a drive system 940, and a computing platform 950. The vehicle 900 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 900 may be interconnected by wire or wireless means.
[0165] In some embodiments, the infotainment system 910 may include a communication system, an entertainment system, and a navigation system, etc.
[0166] The perception system 920 may include several sensors for sensing information about the environment around the vehicle 900. For example, the perception system 920 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.
[0167] The decision control system 930 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.
[0168] The drive system 940 may include components that provide powered motion for the vehicle 900. In one embodiment, the drive system 940 may include an engine, an energy source, a transmission system, and wheels. The engine may be a combination of one or more of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.
[0169] Some or all functions of the vehicle 900 are controlled by a computing platform 950. The computing platform 950 may include at least one processor 951 and a memory 952, and the processor 951 may execute instructions 953 stored in the memory 952.
[0170] The processor 951 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.
[0171] The memory 952 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0172] In addition to instructions 953 , memory 952 may also store data, such as road maps, route information, vehicle location, direction, speed, etc. The data stored in memory 952 may be used by computing platform 950 .
[0173] In the embodiment of the present disclosure, the processor 951 may execute instruction 953 to complete all or part of the steps of the above-mentioned target detection method.
[0174] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above target detection method when executed by the programmable device.
[0175] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0176] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A target detection method, characterized in that: include: Acquire a target image of a target environment where the vehicle is located; Determine a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image through a target detection model, wherein the first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in a target space, and the second foreground point is a real pixel position point of the target object in the target space, and the target space is a preset three-dimensional space corresponding to the target environment; For each of the first foreground points, the target object is detected according to the first foreground point and the offset.
2. The method according to claim 1, characterized in that The preset three-dimensional space includes a preset spatial grid, the preset position point includes a vertex of each grid in the preset spatial grid, and the determining of a plurality of first foreground points according to the target image by a target detection model includes: The first foreground point is determined from the plurality of vertices according to the target image by using a target detection model.
3. The method according to claim 1, characterized in that Determining the offset between each first foreground point and a second foreground point by using a target detection model according to the target image includes: For each of the first foreground points, outputting the offsets of the first foreground point and the corresponding second foreground point in a plurality of preset spatial directions respectively according to the target image through the target detection model; The detecting the target object according to the first foreground point and the offset comprises: A predicted foreground point corresponding to the target object is determined according to the position of the first foreground point and the corresponding offsets in the plurality of preset spatial directions.
4. The method according to claim 1, characterized in that The target detection model is pre-trained in the following way: Acquire a sample image corresponding to the target object; After performing BEV feature extraction on the sample image, the sample image is input into a preset detection model to be trained, and a first prediction result and a second prediction result are output through the preset detection model, wherein the first prediction result is a foreground position point of the target object determined from a plurality of preset position points in the target space after performing image segmentation on the sample image through the preset detection model, and the second prediction result is a predicted position point of the target object in the target space predicted by the preset detection model; Determining a foreground grid through which the target object passes from a preset spatial grid corresponding to the target space; For each foreground grid, the vertices corresponding to the foreground grid are used as first label data of the image segmentation result, and the actual position point of the target object in the foreground grid is used as second label data of the model prediction result; Calculating a loss value of the preset detection model by using a preset loss function according to the first prediction result, the second prediction result, the first label data, and the second label data; After the preset detection model is trained according to the loss value, the target detection model is obtained.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Performing feature extraction on the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image; For each scale, determining a semantic enhancement feature of the scale according to the two-dimensional feature map of the scale; Determine the bird's-eye view BEV fusion feature of the target object according to the semantic enhancement feature at each scale; The step of determining a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image by using a target detection model comprises: After the BEV fusion feature is input into the target detection model, a plurality of the first foreground points and an offset between each of the first foreground points and a second foreground point are determined.
6. The method according to claim 5, characterized in that The step of extracting features from the target image to obtain a multi-scale two-dimensional feature map corresponding to the target image includes: After the target image is input into the feature pyramid network FPN, the multi-scale two-dimensional feature map is obtained.
7. The method according to claim 5, characterized in that Determining the semantic enhancement feature of the scale according to the two-dimensional feature map of the scale includes: After inputting the target feature map of this scale into the semantic segmentation network, the semantic segmentation features of this scale are obtained; According to the target feature map of this scale and the speech segmentation features of this scale, the semantic enhancement features of this scale are determined through a multi-head attention mechanism model.
8. The method according to claim 5, characterized in that Determining the bird's-eye view BEV fusion features of the target object according to the semantic enhancement features at each scale includes: For the semantic enhancement features of each scale, converting the semantic enhancement features of the scale into BEV features through a preset view transformation method; After the BEV features at multiple scales are fused, the BEV fused features are obtained.
9. A target detection device, characterized in that: include: An acquisition module is configured to acquire a target image of a target environment where the vehicle is located; a determination module configured to determine a plurality of first foreground points and an offset between each of the first foreground points and a second foreground point according to the target image through a target detection model, wherein the first foreground point is a predicted pixel position point corresponding to a target object to be detected among a plurality of preset position points in a target space, and the second foreground point is a real pixel position point of the target object in the target space, and the target space is a preset three-dimensional space corresponding to the target environment; The detection module is configured to detect the target object according to the first foreground point and the offset for each of the first foreground points.
10. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.