Road rolling method, device and electronic equipment based on situation awareness

Abstract

The embodiments of the present disclosure disclose a road compaction method, device, and electronic device based on situational awareness. A specific implementation of the method includes: collecting point information corresponding to each monitoring point in a set of monitoring points when the roller moves along the initial compaction route; for each point in the point information set, determining the road surface state information based on a pre-trained road surface state recognition model, the first image, the first echo signal, the second image, and the second echo signal included in the point information; screening out monitoring points whose corresponding road surface state information meets the screening conditions from the monitoring point set as points to be compacted; generating a secondary compaction route based on the set of points to be compacted; and controlling the roller to move along the secondary compaction route to perform secondary compaction on the points to be compacted. This implementation can locate defects in real time and repeat compaction during the road compaction stage to ensure road quality.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the fields of engineering construction and computer technology, and particularly to a road compaction method, device, and electronic equipment based on situational awareness. Background Art

[0002] A road roller is an engineering machine that compacts the road surface by using its own weight or vibration to generate additional downforce. In practice, the quality of a road roller's compaction significantly impacts the subsequent lifespan of the road surface. Currently, visual inspections are the most common method of checking compaction quality.

[0003] However, when the above method is adopted, the following technical problems often occur: when the construction area is large, the inspection efficiency of visual inspection is low, and it is difficult to effectively identify defects such as pavement cavities.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a road compaction method, device, and electronic device based on situational awareness to solve the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a road rolling method based on situational awareness, the method comprising: collecting point information corresponding to each monitoring point in a monitoring point set when a road roller travels along an initial rolling route, and obtaining a point information set, wherein the point information comprises: point position, a first image, a first echo signal, a second image and a second echo signal, the first image and the first echo signal being respectively a road surface image and a road surface echo signal of the point corresponding to the point information when the road roller has not rolled, and the second image and the second echo signal being respectively a road surface image and a road surface echo signal of the point corresponding to the point information after the road roller has rolled; for each point information in the above point information set, according to a pre-trained road surface state recognition The identification model, the first image, the first echo signal, the second image and the second echo signal included in the above-mentioned point information are used to determine the road surface state information, wherein the road surface state information includes: road surface compaction and defect information set, and the defect information includes: defect location and defect type; the monitoring points whose corresponding road surface state information meets the screening conditions are screened out from the above-mentioned monitoring point set as the points to be rolled, and the point set to be rolled is obtained, wherein the screening conditions are: the road surface compaction included in the road surface state information is less than or equal to the preset road surface compaction or the defect information set included in the road surface state information is not empty; according to the above-mentioned point set to be rolled, a secondary rolling route is generated; the above-mentioned roller is controlled to move along the above-mentioned secondary rolling route to perform secondary rolling on the points to be rolled.

[0008] In a second aspect, some embodiments of the present disclosure provide a road rolling device based on situational awareness, the device comprising: an acquisition unit, configured to acquire point information corresponding to each monitoring point in a monitoring point set when the roller moves along an initial rolling route, to obtain a point information set, wherein the point information comprises: a point position, a first image, a first echo signal, a second image and a second echo signal, the first image and the first echo signal being respectively a road surface image and a road surface echo signal of the point corresponding to the point information when the roller has not rolled, and the second image and the second echo signal being respectively a road surface image and a road surface echo signal of the point corresponding to the point information after the roller has rolled; a determination unit being configured to, for each point information in the above-mentioned point information set, determine the road surface state according to a pre-trained road surface state recognition method. The model, the first image, the first echo signal, the second image and the second echo signal included in the above-mentioned point information, determine the road surface condition information, wherein the road surface condition information includes: road surface compaction and a defect information set, and the defect information includes: defect location and defect type; the screening unit is configured to screen out the monitoring points whose corresponding road surface condition information meets the screening conditions from the above-mentioned monitoring point set as the points to be rolled, and obtain the set of points to be rolled, wherein the screening conditions are: the road surface compaction included in the road surface condition information is less than or equal to the preset road surface compaction or the defect information set included in the road surface condition information is not empty; according to the above-mentioned set of points to be rolled, generate a secondary rolling route; the control unit is configured to control the above-mentioned roller to move along the above-mentioned secondary rolling route to perform secondary rolling on the points to be rolled.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the situational awareness-based pavement compaction method of some embodiments of the present disclosure, the defect position can be efficiently and accurately located during the pavement compaction stage and repeated compaction can be performed to ensure the quality of the pavement. Specifically, first, the point information corresponding to each monitoring point in the monitoring point set is collected when the roller moves along the initial compaction route to obtain a point information set, wherein the point information includes: point position, first image, first echo signal, second image, and second echo signal. The first image and first echo signal are respectively the pavement image and pavement echo signal of the point corresponding to the point information before the roller compacts, and the second image and second echo signal are respectively the pavement image and pavement echo signal of the point corresponding to the point information after the roller compacts. By planning the initial compaction route, the roller can compact the compaction area as much as possible without blind spots. At the same time, the images and echo signals before and after compaction are collected respectively as the data basis for subsequent defect location. Next, for each point in the aforementioned point information set, the road surface condition information is determined based on a pre-trained road surface condition recognition model and the first image, first echo signal, second image, and second echo signal included in the aforementioned point information. The road surface condition information includes road surface compaction and a defect information set, and the defect information includes defect location and defect type. Combining the image and echo signal, the defect location and compaction quality (a representation of road surface compaction) of the compacted road surface are determined. Next, monitoring points whose corresponding road surface condition information meets a screening condition are selected from the aforementioned monitoring point set as the points to be compacted, thereby obtaining a set of points to be compacted. The screening condition is that the road surface compaction included in the road surface condition information is less than or equal to a preset road surface compaction or the defect information set included in the road surface condition information is not empty. A secondary compaction route is generated based on the aforementioned set of points to be compacted. Finally, the roller is controlled to travel along the secondary compaction route to perform secondary compaction on the points to be compacted. By screening out monitoring points that meet the screening criteria and replanning the secondary compaction route, repeated compaction is performed at those points requiring secondary compaction to eliminate road surface defects and improve road surface compaction, thereby ensuring road surface quality. This method allows for real-time defect location and repeated compaction during the road surface compaction phase, ensuring road surface quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flow chart of some embodiments of a road surface compaction method based on situational awareness according to the present disclosure;

[0014] Figure 2 It is a schematic diagram of the positional relationship between the initial rolling route and the monitoring points;

[0015] Figure 3 It is a schematic diagram of the process of collecting point information;

[0016] Figure 4 It is a schematic diagram of the scenarios corresponding to different defect types;

[0017] Figure 5 It is a schematic diagram of the process of generating road surface state information;

[0018] Figure 6 It is a position relationship diagram between the points to be rolled and the secondary rolling route;

[0019] Figure 7 Schematic diagram of the graph structure of an undirected graph;

[0020] Figure 8 is a schematic structural diagram of some embodiments of a road surface compaction device based on situational awareness according to the present disclosure;

[0021] Figure 9 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] refer to Figure 1 , shows a process 100 of some embodiments of the road surface compaction method based on situational awareness according to the present disclosure. The road surface compaction method based on situational awareness includes the following steps:

[0029] Step 101 : collecting point information corresponding to each monitoring point in a monitoring point set when the roller travels along an initial rolling route to obtain a point information set.

[0030] In some embodiments, the execution entity (e.g., a computing device) of the situational awareness-based road compaction method can collect point information corresponding to each monitoring point in a set of monitoring points as the roller travels along an initial compaction route, thereby obtaining a set of point information. The point information includes: point location, a first image, a first echo signal, a second image, and a second echo signal. The first image and the first echo signal are, respectively, an image of the road surface and an echo signal of the road surface at the point corresponding to the point information before the roller has compacted the road. The second image and the second echo signal are, respectively, an image of the road surface and an echo signal of the road surface at the point corresponding to the point information after the roller has compacted the road. The initial compaction route is the roller's initial route for the compaction area. Specifically, the first image and the second image can be collected by a camera. The first echo signal and the second echo signal can be collected by a lidar.

[0031] For example, see Figure 2 The schematic diagram of the positional relationship between the initial rolling route and the monitoring points is shown in FIG. Figure 2 The initial rolling routes for two different road types are shown. Specifically, the initial rolling routes are set within the road boundary to maximize coverage of the rolling area. Monitoring points are evenly distributed throughout the rolling area. The density of monitoring points can be adjusted to meet different monitoring needs.

[0032] It should be noted that the computing device described above can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules, for example, to provide distributed services, or as a single software or software module. No specific limitations are given here.

[0033] In some optional implementations of some embodiments, the execution entity collects point information corresponding to each monitoring point in the monitoring point set when the roller travels along the initial rolling route, including:

[0034] In the first step, in response to the distance value between the current position of the above-mentioned roller and the above-mentioned monitoring point being a preset distance value, and the moving direction of the above-mentioned roller being toward the above-mentioned monitoring point, the first image acquisition module and the first echo signal acquisition module arranged on the first side of the above-mentioned roller are controlled to collect the first image and the first echo signal including the point information corresponding to the monitoring point.

[0035] In practice, the first side may be the front side of the roller. The first image acquisition module and the first echo signal acquisition module may both be disposed on the front side of the roller. Specifically, the first image acquisition module and the first echo signal acquisition module may be disposed on the front side of a rolling wheel included in the roller. The first image acquisition module may include at least one camera. The first echo signal acquisition module may include a laser radar.

[0036] For example, see Figure 3 The schematic diagram of the point information acquisition process shown in the figure, wherein the first image acquisition module and the first echo signal acquisition module are arranged on the front side of the rolling wheel. The first image acquisition module may include: two cameras. The first echo signal acquisition module may include: a laser radar. Specifically, when the roller moves along the initial rolling route, when the front side of the roller is at a preset distance value from the monitoring point, the first echo signal is acquired by a laser radar included in the first echo signal acquisition module. And the first image is acquired by the two cameras included in the first image acquisition module. In particular, when the first image acquisition module includes at least two cameras, it is necessary to perform image stitching on the images acquired by the at least two cameras as the first image.

[0037] In the second step, in response to the distance value between the current position of the above-mentioned roller and the above-mentioned monitoring point being a preset distance value, and the moving direction of the above-mentioned roller being away from the above-mentioned monitoring point, the second image acquisition module and the second echo signal acquisition module arranged on the second side of the above-mentioned roller are controlled to acquire the second image and the second echo signal including the point information corresponding to the monitoring point.

[0038] In practice, the second side may be the rear side of the roller. The second image acquisition module and the second echo signal acquisition module may both be disposed on the rear side of the roller. The first image acquisition module may include at least one camera. The first echo signal acquisition module may include: a laser radar.

[0039] Specifically, by setting distance measuring sensors on the front and rear sides of the roller, it is possible to determine whether the front and rear sides of the roller are at a preset distance from the monitoring point. In addition, in order to reduce implementation costs and monitoring intensity, the monitoring points can be set non-repeatedly and continuously. Therefore, it is only necessary to control the time interval for collecting point information. Taking the first image and the first echo signal acquisition process as an example, when the monitoring points are continuous, the acquisition of the first image and the first echo signal corresponding to the point information of different monitoring points is periodic acquisition, and the acquisition period of the periodic acquisition is determined by the length of the monitoring point and the travel speed of the roller. For example, acquisition period = length of the monitoring point / travel speed.

[0040] For example, see further Figure 3 , wherein the second image acquisition module may include: two cameras. The second echo signal acquisition module may include: a laser radar. Specifically, when the roller continues to move along the initial rolling route, when the rear side of the roller is at a preset distance value from the monitoring point, the second echo signal is collected by a laser radar included in the second echo signal acquisition module. And the second image is collected by the two cameras included in the second image acquisition module. In particular, when the second image acquisition module includes at least two cameras, it is necessary to perform image stitching on the images collected by the at least two cameras as the second image.

[0041] Step 102 : For each point information in the point information set, determine the road surface condition information based on a pre-trained road surface condition recognition model, the first image, the first echo signal, the second image, and the second echo signal included in the point information.

[0042] In some embodiments, the execution entity may determine the pavement condition information for each point information in the point information set based on a pre-trained pavement condition recognition model, the first image, the first echo signal, the second image, and the second echo signal included in the point information. The pavement condition recognition model is a machine learning model used to identify the pavement compactness and pavement defects of the monitoring point corresponding to the point information. The pavement condition information characterizes the pavement condition of the pavement corresponding to the monitoring point. The pavement condition information includes: pavement compactness and a defect information set. Specifically, the compactness of the pavement changes after being rolled by a roller. Therefore, the compactness of the pavement after rolling is characterized by the pavement compactness. The defect information includes: defect location and defect type. In practice, the defect types include: crack type, cavity type, and foreign matter type. The crack type characterizes the presence of cracks in the pavement. The cavity type characterizes the presence of cavities inside the pavement. The foreign matter type characterizes the presence of foreign matter in the pavement. For example, the pavement condition recognition model may be a VG-W3D model.

[0043] For example, see Figure 4Schematic diagram of scenarios corresponding to different defect types shown, where: Figure 4 The road surface shows a crack type from a top view, a foreign matter type from a top view, and a cavity type from a side view.

[0044] Optionally, the road surface condition recognition model includes: a first image feature extraction network, a road surface defect position locator, a signal feature extraction network, a region of interest positioning network, a second image feature extraction network and a road surface compactness classifier.

[0045] In practice, the signal feature extraction network uses PointNet as its backbone. Specifically, the MaxPooling and MLP layers included in the PointNet network are removed. This allows the network to output a point cloud feature map corresponding to the echo signal. Specifically, since the first and second echo signals are acquired at a time difference, they are sequentially input into the signal feature extraction network to produce point cloud feature map A corresponding to the first echo signal and point cloud feature map B corresponding to the second echo signal. Considering that radar wave divergence is relatively small on a compacted road surface, the signal strength of the second echo signal is theoretically greater than that of the first echo signal. Therefore, point cloud feature map B is subtracted from point cloud feature map A to produce a signal heatmap. This filter eliminates clutter (feature) interference and highlights abnormal points. Secondly, the signal feature extraction network converts the echo signal (point cloud data) into a feature map, facilitating subsequent region-of-interest localization through image feature processing.

[0046] In practice, considering positioning speed, the ROI localization network adopts a one-stage positioning model. Specifically, the ROI localization network uses YOLOv3-Tiny as the backbone network. Considering that the signal feature extraction network has already performed feature extraction once, in order to avoid repeated feature extraction and further reduce the data processing volume of the feature extraction process, the first four convolutional layers (conv) and the corresponding pooling layers (max) in the YOLOv3-Tiny network are removed. In particular, considering that the input size of the fifth convolutional layer in the YOLOv3-Tiny network is different from the size of the signal heat map, the signal heat map cannot be input into the ROI localization network. Therefore, a local segmentation method is used to divide the signal heat map into K signal heat map blocks with the same input size as the ROI localization network to perform region-by-region ROI localization.

[0047] In practice, both the first and second image feature extraction networks employ conventional convolutional neural network models. For example, ResNet-50 can be used as the backbone network structure. The road surface compactness classifier can be implemented using a fully connected layer to obtain the road surface compactness. Specifically, K road surface compactness levels can be set, so that the road surface compactness classifier outputs a 1×K one-dimensional vector.

[0048] In practice, the road surface defect location locator uses the RPN (Region Proposal Network) as the backbone network to regress the defect location. It also includes a classifier to output the defect type.

[0049] The above-mentioned road surface state recognition model, as an inventive point of the present disclosure, realizes real-time recognition of road surface defects and compactness by combining images and echo signals, thereby improving the quality of road surface construction.

[0050] In some optional implementations of some embodiments, the execution entity determines the road surface condition information based on a pre-trained road surface condition recognition model and the first image, the first echo signal, the second image, and the second echo signal included in the point information, including:

[0051] In the first step, the first image is flipped with the second image as the image reference to obtain a flipped first image.

[0052] In practice, see further Figure 3 The acquisition of the first image and the second image is symmetrical with the detection point as the center. From the imaging principle, the positions corresponding to the same pixel coordinates actually correspond to different image contents, so the image needs to be flipped.

[0053] The second step is to extract the image feature points included in the second image as the first image feature points.

[0054] In practice, the first and second image acquisition modules can be pre-calibrated. During roller operation, the complex construction environment can lead to loosening or misalignment of the first and second image acquisition modules due to mechanical vibration. Therefore, image alignment requires feature point matching to improve robustness. Specifically, the SUFT (Speeded Up Robust Features) algorithm can be used to extract feature points.

[0055] The third step is to extract the image feature points included in the inverted first image as the second image feature points.

[0056] In practice, feature points can be extracted using the SUFT (Speeded Up Robust Features) algorithm.

[0057] In the fourth step, the second image and the inverted first image are aligned based on the feature points of the first image and the feature points of the second image to obtain an aligned first image and an aligned second image.

[0058] In practice, the second image and the inverted first image can be aligned by feature matching to obtain the aligned first image and the aligned second image.

[0059] The fifth step is to determine the image difference between the aligned second image and the aligned first image as a difference image.

[0060] In practice, after alignment, the (reference) feature points in the aligned first image and the (reference) feature points in the aligned second image are used as alignment positions, and the image difference between the aligned second image and the aligned first image is determined as a difference image.

[0061] In the sixth step, a signal heat map is generated through the signal feature extraction network, the first echo signal and the second echo signal.

[0062] For example, see Figure 5 Figure 1 shows a schematic diagram of the road surface state information generation process. Due to the time difference between the acquisition of first echo signal 501 and second echo signal 502, first echo signal 501 can be first input into the signal feature extraction network to obtain a point cloud feature map A corresponding to first echo signal 501. Then, second echo signal 502 can be first input into the signal feature extraction network to obtain a point cloud feature map B corresponding to second echo signal 502. The difference between point cloud feature map B and point cloud feature map A is used as a signal heat map 503.

[0063] Step 7: Determine the region of interest based on the signal heat map and the region of interest positioning network.

[0064] For example, see further Figure 5 , wherein the execution subject uses the signal heat map as input to the region of interest positioning network to obtain a region of interest 504 for possible defects within the monitoring point. In particular, before the signal heat map is input to the region of interest positioning network, a local segmentation method is used to divide the signal heat map into K signal heat map blocks with the same input size as the region of interest positioning network, so as to perform region-by-region region of interest positioning.

[0065] In the eighth step, the difference image is preprocessed according to the above-mentioned region of interest to obtain a preprocessed image.

[0066] In practice, while directly setting pixels outside the ROI to 0 can reduce the amount of data processing, it will miss boundary features at the ROI boundary, potentially affecting positioning accuracy. Therefore, the aforementioned execution entity can maintain the pixel values unchanged at the locations framed by the ROI within the difference image, and decrease the corresponding pixel values outside the ROI in descending order based on the distance from the pixel to the ROI. This approach can make the corresponding pixel values of pixels farther from the ROI approach 0, while also further highlighting the image features within the ROI.

[0067] In the ninth step, a first image feature map is generated by using the first image feature extraction network and the preprocessed image.

[0068] For example, see further Figure 5 , wherein the first image feature extraction network takes the preprocessed image 505 as input to perform image feature extraction and obtain a first image feature map.

[0069] The tenth step is to generate a defect information set included in the road surface status information based on the first image feature map and the road surface defect position locator.

[0070] For example, see further Figure 5 , where the defect information set 506 may include: {[defect A; P: (x1, y1); T: 01], [defect B; P: (x2, y2); T: 10]}. "P: (x1, y1)" represents the pixel coordinates corresponding to the defect location of defect A. "P: (x2, y2)" represents the pixel coordinates corresponding to the defect location of defect B. "T: 01" represents the defect type corresponding to defect A. "T: 10" represents the defect type corresponding to defect B. In practice, "01" may correspond to a crack type. "10" may correspond to a foreign matter type. "11" may correspond to a void type.

[0071] In the eleventh step, a second image feature map is generated based on the second image feature extraction network and the aligned second image.

[0072] For example, see further Figure 5 , wherein the above-mentioned execution entity can extract image features from the aligned image 507 through a second image feature extraction network to obtain a second image feature map.

[0073] In the twelfth step, the road surface compactness included in the road surface state information is generated based on the second image feature map, the signal heat map and the road surface compactness classifier.

[0074] For example, see further Figure 5, wherein the above-mentioned execution subject can extract image features from the aligned image 507 through a second image feature extraction network to obtain a second image feature map. In particular, before the second image feature map and the above-mentioned signal heat map are input into the road surface compactness classifier, it is necessary to unify the feature sizes of the second image feature map and the above-mentioned signal heat map, and perform feature superposition after unification as the input of the road surface compactness classifier. The road surface compactness classifier can be implemented using a fully connected layer to obtain the road surface compactness. Specifically, K road surface compactness can be set, so that the road surface compactness classifier outputs a one-dimensional vector of 1×K. For example, Figure 5 The road surface firmness in 508 is shown as S.

[0075] Step 103 : Filter out the monitoring points whose corresponding road surface status information meets the filtering conditions from the monitoring point set as the points to be rolled, thereby obtaining a set of points to be rolled.

[0076] In some embodiments, the execution entity can filter out monitoring points whose corresponding road surface condition information meets the filtering conditions from the set of monitoring points as the points to be compacted, thereby obtaining a set of points to be compacted. The filtering conditions are: the road surface compaction included in the road surface condition information is less than or equal to the preset road surface compaction or the defect information set included in the road surface condition information is not empty; and a secondary compaction route is generated based on the set of points to be compacted. In practice, a single compaction cannot effectively guarantee that all points in the compaction area are compacted. Repeating the compaction directly will increase construction time. Therefore, it is necessary to determine a new compaction route (secondary compaction route) based on the road surface condition information to accurately compact abnormal points.

[0077] Step 104: Generate a secondary rolling route based on the set of points to be rolled.

[0078] In some embodiments, the aforementioned execution entity can generate a secondary rolling route based on the set of points to be rolled. In practice, a directed graph containing the set of points to be rolled can be constructed based on the locations of the points to be rolled. Path planning can then be performed using, for example, the Dijkstra algorithm to obtain the secondary rolling route.

[0079] In some optional implementations of some embodiments, the execution entity generates a secondary rolling route based on the set of points to be rolled, including:

[0080] The first step is to generate a basic mesh based on the vehicle information of the above-mentioned road roller.

[0081] The vehicle information includes: vehicle width and vehicle length. In practice, the grid length of the basic grid can be the vehicle length. The grid width of the basic grid is the vehicle width.

[0082] The second step is to project the points to be rolled in the above set of points to be rolled onto the rolling area to obtain the projected rolling area.

[0083] For example, see Figure 6 The position relationship diagram between the points to be rolled and the secondary rolling route is shown, where the points to be rolled correspond to the point positions, so they can be projected to the rolling area to obtain the projected area.

[0084] The third step is to use the basic grid as the grid unit to divide the projected compaction area into grids to obtain the area to be compacted after grid division.

[0085] As an example, Figure 6 The figure shows the area to be compacted after grid division. Specifically, the projected compaction area is divided into 4×10 grid units, totaling 40 grid units, using the base grid as the grid unit.

[0086] The fourth step is to generate a local rolling route for each layer in the area to be rolled after the above grid division along the road direction corresponding to the rolling area, and obtain a local rolling route set.

[0087] Each local rolling route in the local rolling route set is parallel to the road, contains at least one rolling point, and is separated by at least K base grids from any two local rolling routes in the same layer, where K is 2. For example, any two local rolling routes in the same horizontal layer are separated by at least K base grids. K is set so that the roller has room to change direction between two local rolling routes.

[0088] For example, see further Figure 6 For any point to be rolled, a horizontal search is conducted for points to be rolled that are less than 2 basic grid lengths apart, and a horizontal line is formed between at least one point to be rolled that is less than 2 basic grid lengths apart and located in the same layer as a local rolling route.

[0089] The fifth step is to use the current position of the above-mentioned roller as the starting position and generate a candidate rolling route set based on the above-mentioned local rolling route set.

[0090] Among them, the candidate rolling routes in the candidate rolling route set all cover the above-mentioned local rolling route set.

[0091] For example, see Figure 7 The diagram of the graph structure of the undirected graph shown in Figure 6 Simplified to Figure 7The undirected graph shown in Figure 1. Each graph node in the undirected graph corresponds to a local rolling route. There is a graph edge between any two graph nodes corresponding to local rolling routes. Since the candidate rolling routes all cover the above set of local rolling routes, the candidate rolling routes are Hamiltonian paths. The number of Hamiltonian paths (candidate rolling routes) contained in the undirected graph is equal to the factorial of the number of nodes in the undirected graph. Figure 7 The undirected graph shown in the figure includes 6 nodes, so the candidate rolling route set includes 720 candidate rolling routes. Therefore, the above execution entity can determine the local rolling route set by Hamiltonian path traversal. It can be found that as the number of graph nodes increases, the number of Hamiltonian paths increases exponentially. Considering the candidate rolling route planning process, the existence of leapfrogging routes should be avoided, for example Figure 7 Therefore, when there is a dotted graph edge in the Hamiltonian path, the Hamiltonian path is directly discarded and no longer used as a candidate rolling route, thereby reducing the number of traversals.

[0092] Step 6: For each candidate rolling route in the candidate rolling route set, determine the percentage of blank grids corresponding to the candidate rolling route.

[0093] Among them, the blank grid ratio represents the ratio of the basic grid that the candidate rolling route passes through and does not contain the points to be rolled to the basic grid that the candidate rolling route passes through.

[0094] As an example, Figure 6 Take the secondary rolling route (candidate rolling route) shown in the figure as an example. The secondary rolling route passes through 27 basic grids. Among them, 9 are basic grids that do not contain the points to be rolled. Therefore, Figure 6 The blank grids corresponding to the candidate rolling routes shown account for 1 / 3.

[0095] In the sixth step, based on the proportion of blank grids corresponding to the candidate rolling routes, a route recommendation value corresponding to each candidate rolling route in the candidate rolling route set is generated.

[0096] In practice, the route recommendation value = 1 / (A1 × blank grid percentage + A2 × route length). A1 and A2 are weights. Route length is the length of the candidate rolling route, which can be represented by the number of basic grids traversed. Specifically, the lower the blank grid percentage and the shorter the route length, the smaller the value of A1 × blank grid percentage + A2 × route length. Therefore, the inverse of this value is used as the route recommendation value. In other words, the lower the blank grid percentage and the shorter the route length, the higher the corresponding route recommendation value.

[0097] In the seventh step, a candidate rolling route whose corresponding route recommendation value meets the route screening condition is selected from the above candidate rolling route set as the above secondary rolling route.

[0098] Among them, the route screening condition is: the route recommendation value is the maximum value.

[0099] Step 105: Control the roller to move along the secondary rolling route to perform secondary rolling on the points to be rolled.

[0100] In some embodiments, the aforementioned execution entity may control the roller to travel along the secondary compaction route to perform secondary compaction on the points to be compacted. In practice, during the secondary compaction process, steps 101 to 105 may be repeatedly executed until the number of points to be compacted in the selected set of points to be compacted is less than a preset number of points to be compacted (the minimum number of abnormal points that meets engineering quality requirements).

[0101] Optionally, the above method further includes:

[0102] The first step is to determine the number of rolling times corresponding to each basic grid in the projected rolling area according to the initial rolling route and the secondary rolling route.

[0103] In practice, since both the initial rolling route and the above-mentioned secondary rolling route pass through the basic grid, the number of times each basic grid is rolled can be counted by means of statistics.

[0104] The second step is to generate a heat map of the rolling area corresponding to the above-mentioned projected rolling area according to the rolling times corresponding to the basic grid.

[0105] In practice, the higher the number of rolling times, the higher the thermal value of the corresponding base grid in the rolling thermal map.

[0106] The third step is to visualize the heat map of the rolling area on the remote visualization terminal.

[0107] In practice, the remote visualization terminal can be a visualization terminal for remotely monitoring construction progress. Specifically, the rolling area heat map can be synchronized to the remote visualization terminal in real time through a wired or wireless connection.

[0108] Optionally, the above method further includes:

[0109] The first step is to load a local map of the location of the above-mentioned roller on the remote visualization terminal.

[0110] In practice, you can call Amap's map interface to load a local map of the roller's location.

[0111] The second step is to determine the real-time positioning position corresponding to the above-mentioned roller.

[0112] In practice, when the position of the roller changes, the real-time positioning position of the roller can be determined by GPS (Global Positioning System).

[0113] The third step is to mark the real-time positioning position in the local map in real time.

[0114] In practice, the real-time positioning position is marked to visually display the changes in the roller's working route.

[0115] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the situational awareness-based pavement compaction method of some embodiments of the present disclosure, the defect position can be efficiently and accurately located during the pavement compaction stage and repeated compaction can be performed to ensure the quality of the pavement. Specifically, first, the point information corresponding to each monitoring point in the monitoring point set is collected when the roller moves along the initial compaction route to obtain a point information set, wherein the point information includes: point position, first image, first echo signal, second image, and second echo signal. The first image and first echo signal are respectively the pavement image and pavement echo signal of the point corresponding to the point information before the roller compacts, and the second image and second echo signal are respectively the pavement image and pavement echo signal of the point corresponding to the point information after the roller compacts. By planning the initial compaction route, the roller can compact the compaction area as much as possible without blind spots. At the same time, the images and echo signals before and after compaction are collected respectively as the data basis for subsequent defect location. Next, for each point in the aforementioned point information set, the road surface condition information is determined based on a pre-trained road surface condition recognition model and the first image, first echo signal, second image, and second echo signal included in the aforementioned point information. The road surface condition information includes road surface compaction and a defect information set, and the defect information includes defect location and defect type. Combining the image and echo signal, the defect location and compaction quality (a representation of road surface compaction) of the compacted road surface are determined. Next, monitoring points whose corresponding road surface condition information meets a screening condition are selected from the aforementioned monitoring point set as the points to be compacted, thereby obtaining a set of points to be compacted. The screening condition is that the road surface compaction included in the road surface condition information is less than or equal to a preset road surface compaction or the defect information set included in the road surface condition information is not empty. A secondary compaction route is generated based on the aforementioned set of points to be compacted. Finally, the roller is controlled to travel along the secondary compaction route to perform secondary compaction on the points to be compacted. By screening out monitoring points that meet the screening criteria and replanning the secondary compaction route, repeated compaction is performed at those points requiring secondary compaction to eliminate road surface defects and improve road surface compaction, thereby ensuring road surface quality. This method allows for real-time defect location and repeated compaction during the road surface compaction phase, ensuring road surface quality.

[0116] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a road rolling device based on situational awareness. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the road surface compaction device based on situational awareness can be specifically applied to various electronic devices.

[0117] like Figure 8 As shown, some embodiments of the road rolling device 800 based on situational awareness include: an acquisition unit 801, a determination unit 802, a screening unit 803, a generation unit 804 and a control unit 805. The acquisition unit 801 is configured to acquire point information corresponding to each monitoring point in the monitoring point set when the roller moves along the initial rolling route, and obtain a point information set, wherein the point information includes: point position, a first image, a first echo signal, a second image and a second echo signal, the first image and the first echo signal are respectively the road surface image and the road surface echo signal of the point corresponding to the point information when the roller has not rolled, and the second image and the second echo signal are respectively the road surface image and the road surface echo signal of the point corresponding to the point information after the roller has rolled; the determination unit 802 is configured to, for each point information in the above point information set, determine the road surface state recognition model according to the pre-trained road surface state recognition model, the first image, the first echo signal included in the above point information, and the road surface echo signal included in the above point information. , the second image and the second echo signal, to determine the road surface state information, wherein the road surface state information includes: road surface compaction and defect information set, and the defect information includes: defect location and defect type; the screening unit 803 is configured to screen out the monitoring points whose corresponding road surface state information meets the screening conditions from the above-mentioned monitoring point set as the points to be rolled, and obtain the set of points to be rolled, wherein the screening conditions are: the road surface compaction included in the road surface state information is less than or equal to the preset road surface compaction or the defect information set included in the road surface state information is not empty; the generating unit 804 is configured to generate a secondary rolling route according to the above-mentioned set of points to be rolled; the control unit 805 is configured to control the above-mentioned roller to move along the above-mentioned secondary rolling route to perform secondary rolling on the points to be rolled. It can be understood that the units recorded in the situation-awareness-based road rolling device 800 are the same as those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the situation awareness-based road compaction device 800 and the units included therein, and will not be repeated here.

[0118] Reference below Figure 9 , which shows a structural schematic diagram of an electronic device (eg, a computing device) suitable for implementing some embodiments of the present disclosure. Figure 9The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure. Figure 9 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system and a computer program. The computer program includes program instructions, which, when executed, may enable the processor to execute any of the above methods. The processor is used to provide computing and control capabilities to support the operation of the entire computer device. The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium, which, when executed by the processor, may enable the processor to execute any of the above methods. The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present disclosure, and does not constitute a limitation on the computer device to which the solution of the present disclosure is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0119] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0120] In one embodiment, the processor is used to run a computer program stored in a memory to implement the following steps: collecting point information corresponding to each monitoring point in a monitoring point set when the roller moves along an initial rolling route to obtain a point information set, wherein the point information includes: point position, a first image, a first echo signal, a second image, and a second echo signal, the first image and the first echo signal are respectively a road surface image and a road surface echo signal of the point corresponding to the point information when the roller has not rolled, and the second image and the second echo signal are respectively a road surface image and a road surface echo signal of the point corresponding to the point information after the roller has rolled; for each point information in the above point information set, according to the pre-trained road state Identify the model, the first image, the first echo signal, the second image and the second echo signal included in the above-mentioned point information, and determine the road surface condition information, wherein the road surface condition information includes: road surface compaction and a defect information set, and the defect information includes: defect location and defect type; filter out the monitoring points whose corresponding road surface condition information meets the filtering conditions from the above-mentioned monitoring point set, as the points to be rolled, and obtain the set of points to be rolled, wherein the filtering conditions are: the road surface compaction included in the road surface condition information is less than or equal to the preset road surface compaction or the defect information set included in the road surface condition information is not empty; generate a secondary rolling route based on the above-mentioned set of points to be rolled; control the above-mentioned roller to move along the above-mentioned secondary rolling route to perform secondary rolling on the points to be rolled.

[0121] An embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the road compaction method based on situational awareness disclosed in the present disclosure.

[0122] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., provided on the computer device.

[0123] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0124] The above descriptions are merely some preferred embodiments of the present disclosure and illustrate the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A road compaction method based on situational awareness, characterized in that: include: Collecting point information corresponding to each monitoring point in the monitoring point set when the roller travels along the initial rolling route to obtain a point information set, wherein the point information includes: a point position, a first image, a first echo signal, a second image, and a second echo signal, wherein the first image and the first echo signal are, respectively, a road surface image and a road surface echo signal at the point corresponding to the point information before the roller has rolled the road surface, and the second image and the second echo signal are, respectively, an image and a road surface echo signal at the point corresponding to the point information after the roller has rolled the road surface; For each point information in the point information set, the road surface state information is determined according to the pre-trained road surface state recognition model, the first image, the first echo signal, the second image and the second echo signal included in the point information, wherein the road surface state information includes: road surface compactness and defect information set, the defect information includes: defect position and defect type, wherein the defect type includes: crack type, cavity type, foreign body type, the road surface state recognition model includes: a first image feature extraction network, a road surface defect position locator, a signal feature extraction network, an area of interest positioning network, a second image feature extraction network and a road surface compactness classifier, wherein the signal feature extraction network uses PointNet as the backbone network, and the signal feature extraction network removes Poi The ntNet network includes a MaxPooling layer and an MLP layer, so that the signal feature extraction network outputs a point cloud feature map corresponding to the echo signal. The region of interest positioning network uses YOLOv3-Tiny as the backbone network. The region of interest positioning network removes the first four convolutional layers and corresponding pooling layers in the YOLOv3-Tiny network. The signal heat map is divided into K signal heat map blocks with the same input size as the region of interest positioning network by local segmentation to perform region-by-region region of interest positioning. The first image feature extraction network and the second image feature extraction network both use ResNet-50 as the backbone network structure. The road surface compactness classifier uses a fully connected layer. The road surface defect location locator uses the RPN (Region Proposal Network) network as the backbone network to regress the defect location. The road surface defect location locator also includes a classifier for outputting the defect type. Selecting monitoring points whose corresponding road surface condition information satisfies a screening condition from the set of monitoring points as points to be compacted, thereby obtaining a set of points to be compacted, wherein the screening condition is that a road surface compaction included in the road surface condition information is less than or equal to a preset road surface compaction or a defect information set included in the road surface condition information is not empty; Generating a secondary rolling route according to the set of points to be rolled; The roller is controlled to move along the secondary rolling route to perform secondary rolling on the points to be rolled, wherein: The determining of the road surface condition information based on the pre-trained road surface condition recognition model and the first image, the first echo signal, the second image, and the second echo signal included in the point information includes: Taking the second image as an image reference, performing image flipping on the first image to obtain a flipped first image; extracting image feature points included in the second image as first image feature points; extracting image feature points included in the inverted first image as second image feature points; performing image alignment on the second image and the inverted first image according to the first image feature points and the second image feature points to obtain an aligned first image and an aligned second image; determining an image difference between the aligned second image and the aligned first image as a difference image; generating a signal heat map through the signal feature extraction network, the first echo signal, and the second echo signal; Determining a region of interest based on the signal heat map and the region of interest positioning network; performing image preprocessing on the difference image according to the region of interest to obtain a preprocessed image, wherein the image preprocessing includes maintaining pixel values at positions within the difference image framed by the region of interest unchanged, and decreasing pixel values outside the region of interest in a decreasing manner according to the distance of the pixels from the region of interest; generating a first image feature map through the first image feature extraction network and the preprocessed image; generating a defect information set included in the road surface state information according to the first image feature map and the road surface defect position locator; generating a second image feature map according to the second image feature extraction network and the aligned second image; The road surface compactness included in the road surface state information is generated according to the second image feature map, the signal heat map and the road surface compactness classifier.

2. The method according to claim 1, characterized in that The collecting of point information corresponding to each monitoring point in the monitoring point set when the roller travels along the initial rolling route includes: In response to a preset distance between the current position of the roller and the monitoring point, and the roller moving in a direction toward the monitoring point, controlling a first image acquisition module and a first echo signal acquisition module disposed on a first side of the roller to acquire a first image and a first echo signal including point information corresponding to the monitoring point; In response to the distance value between the current position of the roller and the monitoring point being a preset distance value, and the moving direction of the roller being away from the monitoring point, the second image acquisition module and the second echo signal acquisition module arranged on the second side of the roller are controlled to collect the second image and second echo signal including the point information corresponding to the monitoring point.

3. The method according to claim 2, characterized in that Generating a secondary rolling route according to the set of points to be rolled includes: generating a basic mesh according to the vehicle information of the road roller; Projecting the points to be rolled in the set of points to be rolled onto the rolling area to obtain the projected rolling area; Using the basic grid as a grid unit, the projected rolling area is grid-divided to obtain a grid-divided area to be rolled; Following the road orientation corresponding to the rolling area, a local rolling route is generated for each layer in the area to be rolled after the grid division, to obtain a local rolling route set, wherein each local rolling route in the local rolling route set is parallel to the road orientation, contains at least one point to be rolled, and any two local rolling routes in the same layer are at least K basic grids apart, where K is 2; Taking the current position of the roller as the starting position, generating a candidate rolling route set according to the local rolling route set, wherein the candidate rolling routes in the candidate rolling route set all cover the local rolling route set; For each candidate rolling route in the candidate rolling route set, determining a blank grid ratio corresponding to the candidate rolling route, wherein the blank grid ratio represents a ratio of basic grids that the candidate rolling route passes through and does not contain points to be rolled to basic grids that the candidate rolling route passes through; Generate a route recommendation value corresponding to each candidate rolling route in the candidate rolling route set according to the proportion of blank grids corresponding to the candidate rolling routes; A candidate rolling route whose corresponding route recommendation value meets the route screening condition is screened out from the candidate rolling route set as the secondary rolling route.

4. The method according to claim 3, characterized in that The method further comprises: Determining the number of rolling times corresponding to each basic grid in the projected rolling area according to the initial rolling route and the secondary rolling route; Generating a thermal map of the rolling area corresponding to the projected rolling area according to the rolling times corresponding to the basic grid; The thermal map of the rolling area is visualized on a remote visualization terminal.

5. The method according to claim 4, characterized in that The method further comprises: Loading a local map of the location of the road roller on a remote visualization terminal; Determine the real-time positioning position corresponding to the road roller; The real-time positioning position is marked in the local map in real time.

6. A road rolling device based on situational awareness, characterized in that: include: a collection unit configured to collect point information corresponding to each monitoring point in the monitoring point set when the roller travels along the initial rolling route, to obtain a point information set, wherein the point information includes: a point position, a first image, a first echo signal, a second image, and a second echo signal, wherein the first image and the first echo signal are, respectively, a road surface image and a road surface echo signal at the point corresponding to the point information before the roller has rolled the road surface, and the second image and the second echo signal are, respectively, an image and a road surface echo signal at the point corresponding to the point information after the roller has rolled the road surface; The determination unit is configured to determine the pavement state information for each point information in the point information set according to a pre-trained pavement state recognition model, the first image, the first echo signal, the second image and the second echo signal included in the point information, wherein the pavement state information includes: pavement compactness and defect information set, the defect information includes: defect location and defect type, wherein the defect type includes: crack type, cavity type, foreign body type, the pavement state recognition model includes: a first image feature extraction network, a pavement defect location locator, a signal feature extraction network, an area of interest positioning network, a second image feature extraction network and a pavement compactness classifier, wherein the signal feature extraction network uses PointNet as the backbone network, and the signal feature extraction network is removed The PointNet network includes a MaxPooling layer and an MLP layer, so that the signal feature extraction network outputs a point cloud feature map corresponding to the echo signal. The region of interest positioning network uses YOLOv3-Tiny as the backbone network. The region of interest positioning network removes the first four convolutional layers and the corresponding pooling layers in the YOLOv3-Tiny network. The signal heat map is divided into K signal heat map blocks with the same input size as the region of interest positioning network by local segmentation, so as to perform region of interest positioning region by region. The first image feature extraction network and the second image feature extraction network both use ResNet-50 as the backbone network structure. The road surface compactness classifier uses a fully connected layer. The road surface defect location locator uses the RPN (Region Proposal Network) network as the backbone network to regress the defect location. The road surface defect location locator also includes a classifier for outputting the defect type. a screening unit configured to screen out, from the set of monitoring points, monitoring points whose corresponding road surface condition information satisfies a screening condition as points to be compacted, thereby obtaining a set of points to be compacted, wherein the screening condition is that a road surface compaction included in the road surface condition information is less than or equal to a preset road surface compaction or a defect information set included in the road surface condition information is not empty; A generating unit is configured to generate a secondary rolling route according to the set of points to be rolled; A control unit is configured to control the roller to move along the secondary rolling route to perform secondary rolling on the points to be rolled, wherein: The determining of the road surface condition information based on the pre-trained road surface condition recognition model and the first image, the first echo signal, the second image, and the second echo signal included in the point information includes: Taking the second image as an image reference, performing image flipping on the first image to obtain a flipped first image; extracting image feature points included in the second image as first image feature points; extracting image feature points included in the inverted first image as second image feature points; performing image alignment on the second image and the inverted first image according to the first image feature points and the second image feature points to obtain an aligned first image and an aligned second image; determining an image difference between the aligned second image and the aligned first image as a difference image; generating a signal heat map through the signal feature extraction network, the first echo signal, and the second echo signal; Determining a region of interest based on the signal heat map and the region of interest positioning network; performing image preprocessing on the difference image according to the region of interest to obtain a preprocessed image, wherein the image preprocessing includes maintaining pixel values at positions within the difference image framed by the region of interest unchanged, and decreasing pixel values outside the region of interest in a decreasing manner according to the distance of the pixels from the region of interest; generating a first image feature map through the first image feature extraction network and the preprocessed image; generating a defect information set included in the road surface state information according to the first image feature map and the road surface defect position locator; generating a second image feature map according to the second image feature extraction network and the aligned second image; The road surface compactness included in the road surface state information is generated according to the second image feature map, the signal heat map and the road surface compactness classifier.

7. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable medium, characterized in that A computer program is stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images