BEV-based accurate calculation method for road disease area ratio

Abstract

The invention discloses a BEV-based accurate calculation method for a road disease area ratio, and the method comprises the steps: configuring a vehicle-mounted data collection system, and carrying out the marking and preprocessing of data; constructing a BEV-based disease detection network and training the BEV-based disease detection network; a BEV-based disease detection network is used for generating disease detection results, the results are spliced in a fixed calculation unit, and the disease area ratio is calculated, the top view of the whole road is reconstructed, so that seamless fusion of multi-view information is realized, the problem of dislocation possibly caused by a traditional splicing technology is effectively solved, and the method is suitable for large-scale popularization and application. According to the method, the comprehensiveness and the accuracy of area measurement are ensured, measurement errors caused by view angle differences and repeated calculation of overlapped area disease areas are avoided, the accuracy and the integrity of disease area statistics are ensured, high geometric consistency and accurate representation of disease areas can be kept even under the conditions of complex road conditions and changeable environments, and the method is suitable for large-scale popularization and application. And the efficiency and the credibility of road condition evaluation are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for accurately calculating a road damage area ratio based on a BEV. Background Art

[0002] In the construction and maintenance of modern transportation infrastructure, the quality and safety of roads are of vital importance. As time goes by and traffic volume increases, roads will suffer from natural environmental erosion and mechanical wear, and defects such as cracks, potholes, and peeling will occur. This not only affects the performance and life of the road, but may also pose a threat to driving safety. Therefore, accurately assessing the degree of road damage is of great significance for formulating effective maintenance plans and optimizing resource allocation. Traditional road damage detection methods often rely on manual inspections and local sampling, which are inefficient and easily affected by human factors, resulting in insufficient assessment accuracy and reliability. In recent years, with the advancement of computer vision and image processing technology, especially the application of deep learning technology, target detection and pixel-level segmentation algorithms have become key tools for road damage detection. Target detection algorithms can accurately locate the diseased area, while pixel segmentation algorithms can carefully distinguish between the diseased and healthy road surfaces, providing accurate disease information.

[0003] However, although existing detection methods have made significant progress in disease detection, they still face challenges in the accurate calculation of disease area ratio, mainly including the difficulty of disease edges crossing image boundaries, image resolution limitations, the impact of changing lighting conditions, and the difficulty of accurately outlining complex disease morphology, as well as the efficiency and robustness of algorithms in large-scale road detection, which limit the effectiveness and scope of applicability of existing methods in practical applications. Summary of the invention

[0004] The present invention provides a method for accurately calculating the road defect area ratio based on BEV, which can effectively solve the problem proposed in the above background technology that although the existing detection methods have made significant progress in disease detection, they still face challenges in the accurate calculation of the disease area ratio, mainly including the problem that the disease edge crosses the image boundary, the image resolution is limited, the influence of changes in lighting conditions, the problem of accurately outlining the complex disease morphology, and the efficiency and robustness of the algorithm in large-scale road detection, which limits the effectiveness and scope of application of the existing methods in practical applications.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for accurately calculating the road damage area ratio based on BEV, comprising the following steps:

[0006] S1. Configure the vehicle-mounted data acquisition system to label and pre-process the data;

[0007] S2, build a disease detection network based on BEV and train it;

[0008] S3, using the BEV-based disease detection network to generate disease detection results, splicing the results within a fixed calculation unit and calculating the disease area ratio;

[0009] In S1, configuring the vehicle-mounted data acquisition system is to annotate and preprocess the data, specifically including: installing and configuring the vehicle-mounted laser radar and camera equipment to collect data, obtain multi-view and surround images, capture information on road diseases from different angles, and perform registration, correction, annotation and data enhancement processing on the collected data;

[0010] In S2, a BEV-based disease detection network is constructed. The network adopts a standard encoder-decoder structure, and its components mainly include a BEV feature encoder, a BEV mask-aware query generator, a BEV mask position query generator, a BEV mask-aware hybrid query generator, and a BEV feature decoder;

[0011] The multi-scale BEV feature map is obtained through the BEV feature encoder. The BEV mask-aware query generator generates a mask-aware query vector and a BEV segmentation mask from the multi-scale BEV features through a multi-layer update query Transformer decoder. The BEV mask position generator generates a query vector containing position information and semantic information through position embedding and pooling operations. The BEV mask-aware hybrid query generator combines the position query with the BEV segmentation mask through the sampling point feature MLP and cross attention to form a mask-aware hybrid query. Using the mask-aware hybrid query and multi-scale BEV features, the segmentation instance category and precise point coordinates are generated through the BEV feature decoder.

[0012] In S3, a disease detection network based on BEV is used to generate disease detection results, the results are spliced ​​in a fixed calculation unit and the disease area ratio is calculated. The specific process includes:

[0013] Determine the global coordinate system, transform the coordinates of the BEV result map output by each network into the same global coordinate system, select an anchor point, and splice the BEV result maps within the set calculation unit based on the anchor point to avoid area overlap. According to the defined disease type, calculate the area of ​​the spliced ​​BEV image, and finally convert it to the actual physical size.

[0014] According to the above technical solution, in S1, specifically, the data acquisition system uses a 128-line high-precision laser radar to obtain three-dimensional point cloud information of the environment, and 6 on-board cameras cover the panoramic view around the vehicle from the front, rear, left front, left rear, right front and right rear, and use a GPS sensor to synchronize data from different sensors, save timestamp information, and ensure that all sensor data are accurately aligned;

[0015] Data collection is required to be carried out at different times and lighting conditions in rural road areas prone to road damage. Data annotation is set to several common road damage categories such as cracks, potholes, broken plates and lane edge lines.

[0016] According to the above technical solution, in S2, the loss function of the BEV-based disease detection network is defined as L road , where L mask represents the BEV segmentation task loss output in the BEV mask-aware query generator. The generated queries are matched with the true labels using the bilateral matching algorithm and the Hungarian algorithm, and the loss is calculated based on the matching results.

[0017] L bev Represents the loss of the vectorized graph construction task, uses L1 loss to calculate the regression task, and FocalLoss to calculate the category loss. The expression is as follows:

[0018]

[0019] L road =L mask +L bev ;

[0020] Among them, matching is to find the match with the minimum loss, C(q i ,g σ(i) ) is the query q i With the real instance g i The matching cost between i With g i The matched index;

[0021] Among them, g i is the true label of the i-th pixel, p i is the predicted value of the i-th pixel, CE represents the cross entropy loss, λ represents the loss coefficient, L1 represents the sum of the absolute errors between the predicted value and the true value, and Focal represents the focal loss, which is an improved cross entropy loss.

[0022] According to the above technical solution, the BEV feature encoder specifically includes: extracting features from the multi-view images taken by the vehicle-mounted camera by using the ImageNet pre-trained network to obtain the 2D features of the input image, projecting the 2D image features to the BEV feature space through the perspective conversion module to obtain the feature representation of the BEV;

[0023] The above BEV features are obtained by fusing high-dimensional and low-dimensional features through the FPN feature pyramid network, and the multi-scale BEV features are jointly encoded using the deformable Transformer encoder to generate multi-scale BEV features, which capture the contextual information between features of different scales and make the generated features contain global vision and local details;

[0024] The BEV mask-aware query generator specifically includes: inputting an initialized learnable query vector and the above-mentioned multi-scale BEV features, the multi-scale BEV features are cyclically decoded through multiple layers of Transformer decoders, and each layer updates the BEV mask-aware query;

[0025] In the above decoding process, the BEV segmentation mask of the previous layer will be thresholded, pixels with high probability will be retained, and irrelevant background areas will be suppressed, so that the perceptual query can better capture the key features of the instance. The query vector, key and value vectors are generated through linear transformation, and the BEV mask perceptual query vector is updated using the attention mechanism to better capture the semantic information of each instance. On the BEV feature map of the largest scale, the BEV segmentation mask is generated through dot product operation and activation function;

[0026] The BEV mask position query generator specifically includes: determining the portion of the BEV segmentation mask used to generate the position query by setting a threshold, generating a sparse BEV mask, performing pixel-level multiplication and pooling operations on the position embedding and the sparse BEV mask, fusing the position information into the query, and generating a query vector with the position information;

[0027] The position query vector is concatenated with the above-mentioned BEV perception query vector, and further enhanced by the learnable query to generate an enhanced query vector containing rich semantic features and spatial information. The enhanced query vector is copied to the initialization point query and added to generate a BEV mask position query vector, adding position and sequence information to the query vector.

[0028] The BEV mask-aware hybrid query generator specifically includes: determining the portion of the BEV segmentation mask used to generate the position query by setting a threshold, generating a sparse BEV mask, the purpose of which is to reduce redundant information, extract key points and thus highlight the geometric structure of the instance, pooling a number of features from the BEV multi-scale features based on the selected key point positions, thereby extracting the most significant features from the selected area to ensure that only the most meaningful information for the detection task is retained, using a multi-layer perceptron MLP to encode the coordinates of the pooled features, and concatenating the MLP output feature vectors of each scale to form a final geometric feature representation;

[0029] The BEV mask geometric features are used as keys and values ​​together with the enhanced BEV mask position query vector, input into the cross attention module, and the BEV mask-aware hybrid query vector is output, which contains the semantic features, position information and geometric structure information of the instance;

[0030] The BEV feature decoder specifically includes: inputting the above-mentioned multi-scale BEV features and the BEV mask-aware hybrid query vector, outputting the interactive and aggregated feature representation through the deformable attention layer, and finally predicting and outputting the category and coordinates of each instance of the road disease.

[0031] According to the above technical solution, the BEV feature encoder, specifically, extracts image features from 2D multi-view images through a pre-trained ResNeXt network. The LSS method estimates the depth information of the image and projects the 2D image features into the BEV feature space in combination with the camera's transformation matrix to generate BEV features. The BEV features are passed through the FPN feature pyramid network to obtain multi-scale BEV features. The multi-scale BEV feature map uses a 3-layer deformable attention encoder to encode the relationship between BEV features.

[0032] According to the above technical solution, the BEV mask-aware query generator specifically initializes a learnable query vector, which will be continuously updated during the entire Transformer decoding process. In each layer of the Transformer decoder, the segmentation mask generated by the previous layer is first thresholded and updated, and then the query vector Q is calculated. l , key K l Sum value V l , the expression is as follows:

[0033]

[0034] in, represents the perceptual query when decoding the (l-1)th layer, W Q , W K , W V is a learnable weight matrix, is the BEV feature of the lth layer;

[0035] The above threshold is set to 0.5 in this embodiment. If it is greater than the threshold, it is set to 0, otherwise it is set to negative infinity, so as to screen out the valid mask area in the BEV segmentation process. Through the 6-layer Transformer decoder, the query vector is decoded and updated layer by layer. Each layer of decoder updates the BEV mask-aware query vector through the interaction between the mask feature, the query vector and the BEV feature. The expression is as follows:

[0036]

[0037] in, represents the perceptual query when decoding at layer l, M l-1 represents the mask when decoding the (l-1)th layer, represents the perceptual query during decoding of the (l-1)th layer, and the softmax function represents the normalization operation;

[0038] After the last layer of decoder outputs the BEV mask query vector, it is dot-producted with the BEV feature of the largest scale in the channel dimension, and the BEV segmentation mask is generated through the sigmoid function, as shown below:

[0039]

[0040] Among them, M l represents the BEV segmentation mask output by this module, Represents the BEV feature of the largest scale, and the sigmoid function is a normalization operation. Indicates the last layer decoder output BEV mask query.

[0041] According to the above technical solution, the BEV mask position query generator specifically extracts the part greater than a threshold of 0.3 from the BEV segmentation mask, retains the pixel value, otherwise sets it to 0, thereby generating a sparse BEV mask, determining the pixel points to be retained during the position query, generating a 2D position embedding vector for each pixel position, and multiplying the position embedding with the sparse BEV mask at the pixel level;

[0042] Through the average pooling operation, the location query vector of each instance is calculated, and its expression is as follows:

[0043]

[0044] Among them, N nz represents the number of non-zero pixels, represents the sparse BEV mask, represents pixel-level multiplication, P E Represents positional embedding;

[0045] The location query vector is concatenated with the above BEV mask-aware query vector to generate a combined location query, which is expressed as follows:

[0046]

[0047] Among them, concate represents the concatenation operation. and represent the location query vector and the BEV mask-aware query vector respectively;

[0048] Copy the combined location query to the number of learnable point queries and add it to the learnable point queries to generate a query vector containing semantic information and location information.

[0049] According to the above technical solution, the BEV mask-aware hybrid query generator specifically extracts the part greater than a threshold of 0.8 from the BEV segmentation mask, retains the pixels above the threshold, otherwise it is set to 0, thereby generating a sparse BEV mask, determining the pixel points as candidates for geometric feature screening, and performing maximum pooling on BEV feature maps of different scales based on the positions of the screened key points, effectively capturing the core geometric information in the feature map, and the expression is as follows:

[0050]

[0051] in, is the BVE feature map of scale S, represents the sparse BEV mask, Represent the maximum pooled features of the BVE feature map of scale S, use the multi-layer perceptron MLP to encode the coordinates of several features after pooling, concatenate the feature vectors of all scales, and obtain the BEV mask geometric features of the instance, which can retain the collective information of each scale to the greatest extent;

[0052] The BEV masked geometric features are used as keys and values ​​together with the BEV position query, input into the cross attention module, and the BEV mask-aware hybrid query vector is output, which contains semantic features, position information and geometric structure information.

[0053] According to the above technical solution, the BEV feature decoder, specifically, uses multi-scale BEV features as the value of the BEV decoder and the BEV-aware mixed query as the query vector. The decoder aggregates feature information of different scales through an attention mechanism. Finally, the classification prediction head outputs the category of the query feature, and the regression prediction head outputs the normalized coordinates of the query feature.

[0054] According to the above technical solution, in S3, specifically, the disease detection results are generated using the BEV-based disease detection network, the results are spliced ​​within 100m and the disease area ratio is calculated, specifically including:

[0055] Input multi-view camera images within the detection distance, use the BEV-based disease detection network to generate a BEV result map, take the driving vehicle as the origin, determine the position of the vehicle in the BEV result map, and calculate the coordinate position and offset angle of the driving vehicle in multiple BEV maps;

[0056] The coordinate system of each BEV image is converted to a unified coordinate system, and the BEV image is translated or rotated according to the direction and distance of the vehicle's movement. The translated BEV images are spliced, and the weighted average method is used to smooth the spliced ​​area and fuse the overlapping area.

[0057] Calculate the total number of pixels in the diseased area, convert it into actual physical distance, and calculate the area ratio.

[0058] Compared with the prior art, the present invention has the following beneficial effects: the present invention has a scientific and reasonable structure and is safe and convenient to use:

[0059] Based on BEV technology, the present invention uses high-precision sensor data to project three-dimensional spatial information onto a two-dimensional plane view, and displays the road and its surrounding environment from a bird's-eye view. This can provide a global overview of the distribution of road damage, and enable a more intuitive and comprehensive understanding and analysis of the damage status. At the same time, objects can be represented at a unified scale, avoiding the situation where objects that are near appear larger and objects that are far away appear smaller. Through intelligent processing and analysis of BEV images, more detailed damage details can be captured, providing a solid data foundation for the accurate calculation of the road damage area ratio, thereby promoting the scientificity and effectiveness of road maintenance decisions.

[0060] In addition, by reconstructing the bird's-eye view of the entire road, seamless integration of multi-perspective information is achieved, effectively solving the misalignment problem that may be caused by traditional splicing technology, ensuring the comprehensiveness and accuracy of area measurement, avoiding measurement errors caused by perspective differences and repeated calculation of diseased areas in overlapping areas, and ensuring the accuracy and completeness of diseased area statistics. Even in complex road conditions and changeable environments, a high degree of geometric consistency and accurate representation of diseased areas can be maintained, greatly improving the efficiency and credibility of road condition assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0062] In the attached picture:

[0063] Figure 1 is a flow chart of the calculation method of the present invention;

[0064] Figure 2 It is a structural schematic diagram of a disease detection network based on BEV of the present invention;

[0065] Figure 3 It is a schematic diagram of the structure of the BEV feature encoder of the present invention;

[0066] Figure 4 It is a structural diagram of the BEV mask-aware query generator of the present invention;

[0067] Figure 5 It is a structural schematic diagram of a BEV mask position query generator of the present invention;

[0068] Figure 6 It is a structural schematic diagram of the BEV mask-aware hybrid query generator and the BEV feature decoder of the present invention;

[0069] Figure 7 It is an accurate calculation method of the road damage area ratio based on BEV in the present invention. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0071] Example: Figure 1 As shown, the present invention provides a technical solution, a method for accurately calculating the road damage area ratio based on BEV, comprising the following steps:

[0072] S1. Configure the vehicle-mounted data acquisition system to label and pre-process the data;

[0073] S2, build a disease detection network based on BEV and train it;

[0074] S3, using the BEV-based disease detection network to generate disease detection results, splicing the results within a fixed calculation unit and calculating the disease area ratio;

[0075] In S1, the vehicle-mounted data collection system is configured to annotate and preprocess the data, including: installing and configuring the vehicle-mounted laser radar and camera equipment to collect data, obtain multi-view and surround images, capture road disease information from different angles, and perform registration, correction, annotation and data enhancement processing on the collected data;

[0076] In S2, a BEV-based disease detection network is constructed. The network adopts a standard encoder-decoder structure. The components mainly include BEV feature encoder, BEV mask-aware query generator, BEV mask position query generator, BEV mask-aware hybrid query generator and BEV feature decoder.

[0077] The multi-scale BEV feature map is obtained through the BEV feature encoder. The BEV mask-aware query generator generates a mask-aware query vector and a BEV segmentation mask from the multi-scale BEV features through a multi-layer update query Transformer decoder. The BEV mask position generator generates a query vector containing position information and semantic information through position embedding and pooling operations. The BEV mask-aware hybrid query generator combines the position query with the BEV segmentation mask through the sampling point feature MLP and cross attention to form a mask-aware hybrid query. Using the mask-aware hybrid query and multi-scale BEV features, the segmentation instance category and precise point coordinates are generated through the BEV feature decoder.

[0078] In S3, the disease detection network based on BEV is used to generate disease detection results, the results are spliced ​​in fixed calculation units and the disease area ratio is calculated. The specific process includes:

[0079] Determine the global coordinate system, transform the coordinates of the BEV result map output by each network into the same global coordinate system, select an anchor point, and splice the BEV result maps within the set calculation unit based on the anchor point to avoid area overlap. According to the defined disease type, calculate the area of ​​the spliced ​​BEV image, and finally convert it to the actual physical size.

[0080] Based on the above technical solution, in S1, specifically, the data acquisition system uses a 128-line high-precision laser radar to obtain three-dimensional point cloud information of the environment, and 6 on-board cameras cover the panoramic view around the vehicle from the front, rear, left front, left rear, right front and right rear. The GPS sensor is used to synchronize data from different sensors, save timestamp information, and ensure that all sensor data are accurately aligned;

[0081] Data collection is required to be carried out at different times and lighting conditions in rural road areas prone to road damage. Data annotation is set to several common road damage categories such as cracks, potholes, broken plates and lane edge lines.

[0082] like Figure 2 As shown, based on the above technical solution, in S2, the loss function of the BEV-based disease detection network is defined as L road , where L mask represents the BEV segmentation task loss output in the BEV mask-aware query generator. The generated queries are matched with the true labels using the bilateral matching algorithm and the Hungarian algorithm, and the loss is calculated based on the matching results.

[0083] L bev Represents the loss of the vectorized graph construction task, uses L1 loss to calculate the regression task, and FocalLoss to calculate the category loss. The expression is as follows:

[0084]

[0085] L road =L mask +L bev ;

[0086] Among them, matching is to find the match with the minimum loss, C(q i ,g σ(i) ) is the query q i With the real instance g i The matching cost between i With g i The matched index;

[0087] Among them, g i is the true label of the i-th pixel, p i is the predicted value of the i-th pixel, CE represents the cross entropy loss, λ represents the loss coefficient, L1 represents the sum of the absolute errors between the predicted value and the true value, and Focal represents the focal loss, which is an improved cross entropy loss.

[0088] like Figure 2 As shown, based on the above technical solution, the BEV feature encoder specifically includes: extracting features from the multi-view images taken by the vehicle-mounted camera by using the ImageNet pre-trained network to obtain the 2D features of the input image, and projecting the 2D image features to the BEV feature space through the perspective conversion module to obtain the feature representation of BEV;

[0089] The above BEV features are obtained by fusing high-dimensional and low-dimensional features through the FPN feature pyramid network, and the multi-scale BEV features are jointly encoded using the deformable Transformer encoder to generate multi-scale BEV features, which capture the contextual information between features of different scales and make the generated features contain global vision and local details;

[0090] The BEV mask-aware query generator includes the following steps: inputting the initialized learnable query vector and the multi-scale BEV features, which are decoded by multiple layers of Transformer decoders, and each layer updates the BEV mask-aware query.

[0091] In the above decoding process, the BEV segmentation mask of the previous layer will be thresholded, pixels with high probability will be retained, and irrelevant background areas will be suppressed, so that the perceptual query can better capture the key features of the instance. The query vector, key and value vectors are generated through linear transformation, and the BEV mask perceptual query vector is updated using the attention mechanism to better capture the semantic information of each instance. On the BEV feature map of the largest scale, the BEV segmentation mask is generated through dot product operation and activation function;

[0092] The BEV mask location query generator includes: determining the portion of the BEV segmentation mask used to generate the location query by setting a threshold, generating a sparse BEV mask, performing pixel-level multiplication and pooling operations on the location embedding and the sparse BEV mask, fusing the location information into the query, and generating a query vector with location information;

[0093] The position query vector is concatenated with the above-mentioned BEV perception query vector, and further enhanced by the learnable query to generate an enhanced query vector containing rich semantic features and spatial information. The enhanced query vector is copied to the initialization point query and added to generate a BEV mask position query vector, adding position and sequence information to the query vector.

[0094] BEV mask-aware hybrid query generator, the specific process includes: determining the part of the BEV segmentation mask used to generate the position query through threshold setting, generating a sparse BEV mask, the purpose is to reduce redundant information, extract key points to highlight the geometric structure of the instance, based on the selected key point positions, pooling several features from the BEV multi-scale features, so as to extract the most significant features from the filtered area, ensuring that only the most meaningful information for the detection task is retained, using a multi-layer perceptron MLP to encode the coordinates of several pooled features, and splicing the MLP output feature vectors of each scale to form the final geometric feature representation;

[0095] The BEV mask geometric features are used as keys and values ​​together with the enhanced BEV mask position query vector, input into the cross attention module, and the BEV mask-aware hybrid query vector is output, which contains the semantic features, position information and geometric structure information of the instance;

[0096] The BEV feature decoder includes the following steps: inputting the multi-scale BEV features and the BEV mask-aware hybrid query vector, outputting the interacted and aggregated feature representations through the deformable attention layer, and finally predicting and outputting the category and coordinates of each instance of road damage.

[0097] like Figure 3As shown, based on the above technical solution, the BEV feature encoder, specifically, extracts image features from 2D multi-view images through a pre-trained ResNeXt network, the Lift-Splat-Shoot method estimates the depth information of the image, and combines the camera's transformation matrix to project the 2D image features into the BEV feature space to generate BEV features, the BEV features are obtained by the FPN feature pyramid network to obtain multi-scale BEV features, and the multi-scale BEV feature map uses a 3-layer deformable attention encoder to encode the relationship between BEV features.

[0098] like Figure 4 As shown in the figure, based on the above technical solution, the BEV mask-aware query generator specifically initializes a learnable query vector, which will be continuously updated during the entire Transformer decoding process. In each layer of the Transformer decoder, the segmentation mask generated by the previous layer is first thresholded and updated, and then the query vector Q is calculated. l , key K l Sum value V l , the expression is as follows:

[0099]

[0100] in, represents the perceptual query when decoding the (l-1)th layer, W Q , W K , W V is a learnable weight matrix, is the BEV feature of the lth layer;

[0101] The above threshold is set to 0.5 in this embodiment. If it is greater than the threshold, it is set to 0, otherwise it is set to negative infinity, so as to screen out the valid mask area in the BEV segmentation process. Through the 6-layer Transformer decoder, the query vector is decoded and updated layer by layer. Each layer of decoder updates the BEV mask-aware query vector through the interaction between the mask feature, the query vector and the BEV feature. The expression is as follows:

[0102]

[0103] in, represents the perceptual query when decoding at layer l, M l-1 represents the mask when decoding the (l-1)th layer, represents the perceptual query during decoding of the (l-1)th layer, and the softmax function represents the normalization operation;

[0104] After the last layer of decoder outputs the BEV mask query vector, it is dot-producted with the BEV feature of the largest scale in the channel dimension, and the BEV segmentation mask is generated through the sigmoid function, as shown below:

[0105]

[0106] Among them, M l represents the BEV segmentation mask output by this module, Represents the BEV feature of the largest scale, and the sigmoid function is a normalization operation. Indicates the last layer decoder output BEV mask query.

[0107] like Figure 5 As shown, based on the above technical solution, the BEV mask position query generator, specifically, extracts the part greater than the threshold value of 0.3 from the BEV segmentation mask, retains the pixel value, otherwise sets it to 0, thereby generating a sparse BEV mask, determining the pixel points to be retained during the position query, generating a 2D position embedding vector for each pixel position, and multiplying the position embedding with the sparse BEV mask at the pixel level;

[0108] Through the average pooling operation, the location query vector of each instance is calculated, and its expression is as follows:

[0109]

[0110] Among them, N nz represents the number of non-zero pixels, represents the sparse BEV mask, represents pixel-level multiplication, P E Represents positional embedding;

[0111] The location query vector is concatenated with the above BEV mask-aware query vector to generate a combined location query, which is expressed as follows:

[0112]

[0113] Among them, concate represents the concatenation operation. and represent the location query vector and the BEV mask-aware query vector respectively;

[0114] The combined location query is copied to the number of learnable point queries, which is set to 20 in this embodiment, and added to the learnable point queries to generate a query vector containing semantic information and location information.

[0115] like Figure 6As shown in the figure, based on the above technical solution, the BEV mask-aware hybrid query generator, specifically, extracts the part greater than the threshold of 0.8 from the BEV segmentation mask, retains the pixels above the threshold, otherwise it is set to 0, thereby generating a sparse BEV mask, determining the pixel points as candidates for geometric feature screening, and performing maximum pooling on BEV feature maps of different scales based on the positions of the screened key points, effectively capturing the core geometric information in the feature map. The expression is as follows:

[0116]

[0117] in, is the BVE feature map of scale S, represents the sparse BEV mask, Represent the maximum pooled features of the BVE feature map of scale S, use the multi-layer perceptron MLP to encode the coordinates of several features after pooling, concatenate the feature vectors of all scales, and obtain the BEV mask geometric features of the instance, which can retain the collective information of each scale to the greatest extent;

[0118] The BEV masked geometric features are used as keys and values ​​together with the BEV position query, input into the cross attention module, and the BEV mask-aware hybrid query vector is output, which contains semantic features, position information and geometric structure information.

[0119] like Figure 6 As shown, based on the above technical solution, the BEV feature decoder, specifically, uses multi-scale BEV features as the value of the BEV decoder, and the BEV-aware mixed query as the query vector. The decoder aggregates feature information of different scales through the attention mechanism. Finally, the classification prediction head outputs the category of the query feature, and the regression prediction head outputs the normalized coordinates of the query feature.

[0120] like Figure 7 As shown, based on the above technical solution, in S3, specifically, a disease detection result is generated using a BEV-based disease detection network, the results are spliced ​​within 100m and the disease area ratio is calculated, specifically including:

[0121] Input multi-view camera images within the detection distance, use the BEV-based disease detection network to generate a BEV result map, take the driving vehicle as the origin, determine the position of the vehicle in the BEV result map, and calculate the coordinate position and offset angle of the driving vehicle in multiple BEV maps;

[0122] The coordinate system of each BEV image is converted to a unified coordinate system, and the BEV image is translated or rotated according to the direction and distance of the vehicle's movement. The translated BEV images are spliced, and the weighted average method is used to smooth the spliced ​​area and fuse the overlapping area.

[0123] Calculate the total number of pixels in the diseased area, convert it into actual physical distance, and calculate the area ratio.

[0124] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for accurately calculating the road damage area ratio based on BEV, characterized by: The steps include: S1. Configure the vehicle-mounted data acquisition system to label and pre-process the data; S2, build a disease detection network based on BEV and train it; S3, using the BEV-based disease detection network to generate disease detection results, splicing the results within a fixed calculation unit and calculating the disease area ratio; In S1, configuring the vehicle-mounted data acquisition system is to annotate and preprocess the data, specifically including: installing and configuring the vehicle-mounted laser radar and camera equipment to collect data, obtain multi-view and surround images, capture information on road diseases from different angles, and perform registration, correction, annotation and data enhancement processing on the collected data; In S2, a BEV-based disease detection network is constructed. The network adopts a standard encoder-decoder structure, and its components mainly include a BEV feature encoder, a BEV mask-aware query generator, a BEV mask position query generator, a BEV mask-aware hybrid query generator, and a BEV feature decoder; The multi-scale BEV feature map is obtained through the BEV feature encoder. The BEV mask-aware query generator generates a mask-aware query vector and a BEV segmentation mask from the multi-scale BEV features through a multi-layer update query Transformer decoder. The BEV mask position generator generates a query vector containing position information and semantic information through position embedding and pooling operations. The BEV mask-aware hybrid query generator combines the position query with the BEV segmentation mask through the sampling point feature MLP and cross attention to form a mask-aware hybrid query. Using the mask-aware hybrid query and multi-scale BEV features, the segmentation instance category and precise point coordinates are generated through the BEV feature decoder. In S3, a disease detection network based on BEV is used to generate disease detection results, the results are spliced ​​in a fixed calculation unit and the disease area ratio is calculated. The specific process includes: Determine the global coordinate system, transform the coordinates of the BEV result map output by each network into the same global coordinate system, select an anchor point, and splice the BEV result maps within the set calculation unit based on the anchor point to avoid area overlap. According to the defined disease type, calculate the area of ​​the spliced ​​BEV image, and finally convert it to the actual physical size.

2. The accurate calculation method of road damage area ratio based on BEV according to claim 1 is characterized by: In S1, specifically, the data acquisition system uses a 128-line high-precision laser radar to obtain three-dimensional point cloud information of the environment, and 6 on-board cameras cover the panoramic view around the vehicle from the front, rear, left front, left rear, right front and right rear. The GPS sensor is used to synchronize data from different sensors, save timestamp information, and ensure that all sensor data are accurately aligned; Data collection is required to be carried out at different times and lighting conditions in rural road areas prone to road damage. Data annotation is set to several common road damage categories such as cracks, potholes, broken plates and lane edge lines.

3. The accurate calculation method of road damage area ratio based on BEV according to claim 1 is characterized by: In S2, the loss function of the BEV-based disease detection network is defined as L road , where L mask represents the BEV segmentation task loss output in the BEV mask-aware query generator. The generated queries are matched with the true labels using the bilateral matching algorithm and the Hungarian algorithm, and the loss is calculated based on the matching results. L bev Represents the loss of the vectorized graph construction task, uses L1 loss to calculate the regression task, and FocalLoss to calculate the category loss. The expression is as follows: L road =L mask +L bev ; Among them, matching is to find the match with the minimum loss, C(q i ,g σ(i) ) is the query q i With the real instance g i The matching cost between i With g i The matched index; Among them, g i is the true label of the i-th pixel, p i is the predicted value of the i-th pixel, CE represents the cross entropy loss, λ represents the loss coefficient, L1 represents the sum of the absolute errors between the predicted value and the true value, and Focal represents the focal loss, which is an improved cross entropy loss.

4. The method for accurately calculating the road damage area ratio based on BEV according to claim 1, characterized in that: The BEV feature encoder specifically includes: extracting features from a multi-view image taken by a vehicle-mounted camera by using an ImageNet pre-trained network to obtain 2D features of the input image, projecting the 2D image features into the BEV feature space through a view conversion module to obtain a feature representation of the BEV; The above BEV features are obtained by fusing high-dimensional and low-dimensional features through the FPN feature pyramid network, and the multi-scale BEV features are jointly encoded using the deformable Transformer encoder to generate multi-scale BEV features, which capture the contextual information between features of different scales and make the generated features contain global vision and local details; The BEV mask-aware query generator specifically includes: inputting an initialized learnable query vector and the above-mentioned multi-scale BEV features, the multi-scale BEV features are cyclically decoded through multiple layers of Transformer decoders, and each layer updates the BEV mask-aware query; In the above decoding process, the BEV segmentation mask of the previous layer will be thresholded, pixels with high probability will be retained, and irrelevant background areas will be suppressed, so that the perceptual query can better capture the key features of the instance. The query vector, key and value vectors are generated through linear transformation, and the BEV mask perceptual query vector is updated using the attention mechanism to better capture the semantic information of each instance. On the BEV feature map of the largest scale, the BEV segmentation mask is generated through dot product operation and activation function; The BEV mask position query generator specifically includes: determining the portion of the BEV segmentation mask used to generate the position query by setting a threshold, generating a sparse BEV mask, performing pixel-level multiplication and pooling operations on the position embedding and the sparse BEV mask, fusing the position information into the query, and generating a query vector with the position information; The position query vector is concatenated with the above-mentioned BEV perception query vector, and further enhanced by the learnable query to generate an enhanced query vector containing rich semantic features and spatial information. The enhanced query vector is copied to the initialization point query and added to generate a BEV mask position query vector, adding position and sequence information to the query vector. The BEV mask-aware hybrid query generator specifically includes: determining the portion of the BEV segmentation mask used to generate the position query by setting a threshold, generating a sparse BEV mask, the purpose of which is to reduce redundant information, extract key points and thus highlight the geometric structure of the instance, pooling a number of features from the BEV multi-scale features based on the selected key point positions, thereby extracting the most significant features from the selected area to ensure that only the most meaningful information for the detection task is retained, using a multi-layer perceptron MLP to encode the coordinates of the pooled features, and concatenating the MLP output feature vectors of each scale to form a final geometric feature representation; The BEV mask geometric features are used as keys and values ​​together with the enhanced BEV mask position query vector, input into the cross attention module, and the BEV mask-aware hybrid query vector is output, which contains the semantic features, position information and geometric structure information of the instance; The BEV feature decoder specifically includes: inputting the above-mentioned multi-scale BEV features and the BEV mask-aware hybrid query vector, outputting the interactive and aggregated feature representation through the deformable attention layer, and finally predicting and outputting the category and coordinates of each instance of the road disease.

5. The method for accurately calculating the road damage area ratio based on BEV according to claim 4, characterized in that: Specifically, the BEV feature encoder extracts image features from 2D multi-view images through a pre-trained ResNeXt network. The LSS method estimates the depth information of the image and projects the 2D image features into the BEV feature space in combination with the camera's transformation matrix to generate BEV features. The BEV features are passed through an FPN feature pyramid network to obtain multi-scale BEV features. The multi-scale BEV feature map uses a three-layer deformable attention encoder to encode the relationship between BEV features.

6. The method for accurately calculating the road damage area ratio based on BEV according to claim 4, characterized in that: The BEV mask-aware query generator specifically initializes a learnable query vector, which will be continuously updated during the entire Transformer decoding process. In each layer of the Transformer decoder, the segmentation mask generated by the previous layer is first thresholded and updated, and then the query vector Q is calculated. l , key K l Sum value V l , the expression is as follows: in, represents the perceptual query when decoding the (l-1)th layer, W Q , W K , W V is a learnable weight matrix, is the BEV feature of the lth layer; The above threshold is set to 0.5 in this embodiment. If it is greater than the threshold, it is set to 0, otherwise it is set to negative infinity, so as to screen out the valid mask area in the BEV segmentation process. Through the 6-layer Transformer decoder, the query vector is decoded and updated layer by layer. Each layer of decoder updates the BEV mask-aware query vector through the interaction between the mask feature, the query vector and the BEV feature. The expression is as follows: in, represents the perceptual query when decoding at layer l, M l-1 represents the mask when decoding the (l-1)th layer, represents the perceptual query during decoding of the (l-1)th layer, and the softmax function represents the normalization operation; After the last layer of decoder outputs the BEV mask query vector, it is dot-producted with the BEV feature of the largest scale in the channel dimension, and the BEV segmentation mask is generated through the sigmoid function, which is expressed as follows: Among them, M l represents the BEV segmentation mask output by this module, Represents the BEV feature of the largest scale, and the sigmoid function is a normalization operation. Indicates the last layer decoder output BEV mask query.

7. The method for accurately calculating the road damage area ratio based on BEV according to claim 4, characterized in that: The BEV mask position query generator, specifically, extracts the part greater than a threshold of 0.3 from the BEV segmentation mask, retains the pixel value, otherwise sets it to 0, thereby generating a sparse BEV mask, determines the pixel points to be retained during the position query, generates a 2D position embedding vector for each pixel position, and multiplies the position embedding with the sparse BEV mask at the pixel level; Through the average pooling operation, the location query vector of each instance is calculated, and its expression is as follows: Among them, N nz represents the number of non-zero pixels, represents the sparse BEV mask, represents pixel-level multiplication, P E Represents positional embedding; The location query vector is concatenated with the above BEV mask-aware query vector to generate a combined location query, which is expressed as follows: Among them, concate represents the concatenation operation. and represent the location query vector and the BEV mask-aware query vector respectively; Copy the combined location query to the number of learnable point queries and add it to the learnable point queries to generate a query vector containing semantic information and location information.

8. The method for accurately calculating the road damage area ratio based on BEV according to claim 4 is characterized by: The BEV mask-aware hybrid query generator specifically extracts the part greater than a threshold of 0.8 from the BEV segmentation mask, retains the pixels above the threshold, and otherwise sets them to 0, thereby generating a sparse BEV mask, determining the pixel points that are candidates for geometric feature screening, and performing maximum pooling on BEV feature maps of different scales based on the positions of the screened key points, effectively capturing the core geometric information in the feature map. The expression is as follows: in, is the BVE feature map of scale S, represents the sparse BEV mask, Represent the maximum pooled features of the BVE feature map of scale S, use the multi-layer perceptron MLP to encode the coordinates of several features after pooling, concatenate the feature vectors of all scales, and obtain the BEV mask geometric features of the instance, which can retain the collective information of each scale to the greatest extent; The BEV masked geometric features are used as keys and values ​​together with the BEV position query, input into the cross attention module, and the BEV mask-aware hybrid query vector is output, which contains semantic features, position information and geometric structure information.

9. The method for accurately calculating the road damage area ratio based on BEV according to claim 4, characterized in that: The BEV feature decoder, specifically, uses multi-scale BEV features as the value of the BEV decoder and the BEV-aware mixed query as the query vector. The decoder aggregates feature information of different scales through an attention mechanism. Finally, the classification prediction head outputs the category of the query feature, and the regression prediction head outputs the normalized coordinates of the query feature.

10. The method for accurately calculating the road damage area ratio based on BEV according to claim 1, characterized in that: In S3, specifically, the disease detection results are generated using the BEV-based disease detection network, the results are spliced ​​within 100m and the disease area ratio is calculated, including: Input multi-view camera images within the detection distance, use the BEV-based disease detection network to generate a BEV result map, take the driving vehicle as the origin, determine the position of the vehicle in the BEV result map, and calculate the coordinate position and offset angle of the driving vehicle in multiple BEV maps; The coordinate system of each BEV image is converted to a unified coordinate system, and the BEV image is translated or rotated according to the direction and distance of the vehicle's movement. The translated BEV images are spliced, and the weighted average method is used to smooth the spliced ​​area and fuse the overlapping area. Calculate the total number of pixels in the diseased area, convert it into actual physical distance, and calculate the area ratio.

Images