Road safety detection method, system and device based on image recognition

Abstract

The present invention provides a road safety detection method, system and device based on image recognition, the method comprising the following steps: constructing a shared feature extraction network and multiple road detection branch networks; completing the training of the shared feature extraction network and all road detection branch networks; receiving a road detection task list and obtaining the detection task identification currently required to be performed based on the road detection task list; in the process of executing the road detection task, collecting real-time image information of the surrounding road environment through a high-definition camera module; transmitting the detection task identification and the pre-processed real-time image information to the shared feature extraction network in the edge computing module, and outputting the road detection result; if there is a road abnormality result in the road detection result, obtaining the real-time positioning information of the road safety detection device through the Beidou positioning module; combining the road abnormality result and the real-time positioning information to generate road defect feedback information. The present invention has the effect of reducing the use of computing power during the detection process.

Description

Technical Field

[0001] The present invention belongs to the technical field of road inspection, and specifically relates to a road safety detection method, system and device based on image recognition. Background Art

[0002] In intelligent transportation systems, road safety detection is a key link, which aims to monitor road conditions in real time and promptly detect and deal with potential safety hazards through advanced sensing technology and data processing methods. With the acceleration of urbanization and the increase in the number of motor vehicles, road traffic conditions have become more complex and changeable, and traditional detection methods can no longer meet actual needs. At the same time, with the rapid development of artificial intelligence, the Internet of Things and big data technologies, more and more intelligent devices and systems are being used in traffic management, providing new technical means and solutions to improve road safety. In this context, developing a technical solution that can efficiently and accurately detect road safety conditions has become an important direction for the development of intelligent transportation systems.

[0003] In the prior art, road safety detection is achieved through unmanned detection devices and image recognition technology. However, in large-scale road safety detection, traditional image processing and detection algorithms need to consume a lot of computing resources when processing a large amount of real-time image information. This not only increases the hardware cost of the system, but also reduces the real-time and efficiency of the detection. In addition, traditional detection systems usually perform data processing on a central server. This method has a certain delay in data transmission and processing, which may not meet the requirements for road safety detection tasks that require a quick response. Therefore, how to reduce computing power requirements and improve system detection efficiency while ensuring detection accuracy has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a road safety detection method, system and device based on image recognition to solve the problem of consuming a large amount of computing resources in large-scale road safety detection.

[0005] In a first aspect, the present invention provides a road safety detection method based on image recognition, which is applied to a road safety detection device, wherein the road safety detection device is equipped with a Beidou positioning module, a high-definition camera module, and an edge computing module, and the method comprises the following steps:

[0006] Constructing a shared feature extraction network and a plurality of road detection branch networks in the edge computing module, wherein the plurality of road detection branch networks are all connected to the last gating mechanism layer in the shared feature extraction network, and all the road detection branch networks include a region of interest pooling layer;

[0007] Acquire a road detection image training set, and use the road detection image training set to complete the training of the shared feature extraction network and all the road detection branch networks, wherein all training images in the road detection image training set have pre-labeled feature category identifiers and task category identifiers;

[0008] Receiving a road detection task list and obtaining a detection task identifier currently required to be executed based on the road detection task list;

[0009] In the process of the road safety detection device performing the road detection task according to the road detection task list, collecting real-time image information of the surrounding road environment through the high-definition camera module;

[0010] Preprocessing the real-time image information, transmitting the detection task identifier and the preprocessed real-time image information to the shared feature extraction network in the edge computing module, and outputting the road detection result through one or more road detection branch networks;

[0011] If there is an abnormal road result in the road detection result, obtaining the real-time positioning information of the road safety detection device through the Beidou positioning module;

[0012] The road defect feedback information is generated by combining the road anomaly result and the real-time positioning information.

[0013] Optionally, the shared feature extraction network further includes a shared feature extraction layer, the shared feature extraction layer includes an image input layer, a first convolutional layer, a first fully connected layer and a task classification output layer connected in sequence, and the task classification output layer is connected to the gating mechanism layer;

[0014] The road detection branch network also includes a feature image input layer and a feature classification layer, the feature image input layer is connected to the gating mechanism layer, the region of interest pooling layer is connected to the feature image input layer, the feature classification layer includes a second convolutional layer, a second fully connected layer and a feature classification output layer connected in sequence, and the second convolutional layer is connected to the region of interest pooling layer.

[0015] Optionally, the feature category identifier includes multiple abnormal feature major category identifiers, and each of the abnormal feature major category identifiers includes multiple abnormal feature minor category identifiers.

[0016] Optionally, the using the road detection image training set to complete the training of the shared feature extraction network and all the road detection branch networks comprises the following steps:

[0017] Initializing network parameters of the shared feature extraction network and all the road detection branch networks;

[0018] Input all training images in the road detection image training set into the shared feature extraction layer through the image input layer, train the shared feature extraction layer in combination with the training images and the feature category identifiers, and construct a first loss function of the shared feature extraction layer training process;

[0019] Inputting the training feature image output by the shared feature extraction layer into the gating mechanism layer, training the gating mechanism layer in combination with the training feature image and the task category identifier, and constructing a second loss function of the training process of the gating mechanism layer;

[0020] Passing the training feature image to the road detection branch network of the corresponding category through the gating mechanism layer and according to the task category identifier;

[0021] For each of the road detection branch networks, the region of interest pooling layer is trained in combination with the abnormal feature category identifier and the training feature image, and a third loss function of the region of interest pooling layer training process is constructed;

[0022] Input the training feature image segment output by the region of interest pooling layer into the feature classification layer, train the feature classification layer in combination with the training feature image segment and the abnormal feature subclass identifier, and construct a fourth loss function of the feature classification layer training process;

[0023] The total loss of model training is calculated by combining the first loss function, the second loss function, the third loss function and the fourth loss function, the total loss of model training is back-propagated and all the network parameters are iteratively updated, and all the above training steps are repeated until the number of iterative updates reaches a preset maximum number of iterations.

[0024] Optionally, the first loss function is a GIoU loss function, the second loss function is a binary cross entropy loss function, the third loss function is an L1 loss function, and the fourth loss function is a multivariate cross entropy loss function.

[0025] The calculation formula for the total loss of the model training is as follows:

[0026]

[0027] Where: L 0 represents the total loss of the model training, α represents the weight of the first loss function, L 1 represents the first loss calculated by the first loss function, β represents the weight of the second loss function, L 2 The second loss function is used to calculate the second loss, γ irepresents the weight of the third loss function in the i-th road detection branch network, represents the third loss calculated by the third loss function in the i-th road detection branch network, δ i represents the weight of the fourth loss function in the i-th road detection branch network, represents the fourth loss calculated by the fourth loss function in the i-th road detection branch network, and n represents the number of the road detection branch networks.

[0028] Optionally, the road safety detection device is further equipped with a laser radar module, the road detection task list includes a road disease detection task, a road safety facility detection task and a road foreign body detection task, and the method further includes the following steps:

[0029] When the road safety detection device starts to execute the road detection task according to the road detection task list, the laser radar module, the Beidou positioning module, the high-definition camera module and the edge computing module are synchronized using a timestamp alignment technology;

[0030] Acquiring initial positioning information of the road safety detection device through the Beidou positioning module, and collecting initial laser radar data of the surrounding road environment through the laser radar module, and switching the laser radar module to a standby state after collecting the initial laser radar data;

[0031] Acquiring initial operating status data of the road safety detection device;

[0032] The initial operation state data, the initial positioning information and the initial laser radar data are transmitted to the edge computing module, and the initial actual point cloud map of the surrounding road environment is generated by combining the initial operation state data, the initial positioning information and the initial laser radar data with the edge computing module;

[0033] In the process of the road safety detection device performing the road detection task according to the road detection task list, obtaining real-time operation status data of the road safety detection device, and transmitting the real-time operation status data to the edge computing module;

[0034] Using the edge computing module to predict the point cloud state of the initial actual point cloud map according to the real-time operating status data and using a Kalman filter method to obtain a real-time predicted point cloud map;

[0035] If there is an abnormal road result in the road detection result, the laser radar module is switched to a working state, and real-time laser radar data of the surrounding road environment at the current moment is collected by the laser radar module. After the real-time laser radar data is collected, the laser radar module is switched to a standby state;

[0036] The real-time laser radar data is transmitted to the edge computing module, and the edge computing module is used to generate a real-time actual point cloud map of the surrounding road environment at the current moment in combination with the real-time operation status data, the real-time positioning information and the real-time laser radar data;

[0037] Using a point cloud registration algorithm to calculate the point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image;

[0038] The road anomaly results are verified using the point cloud differences, and the road anomaly results that fail verification are removed.

[0039] Optionally, the step of calculating the point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image by using a point cloud registration algorithm comprises the following steps:

[0040] Preprocessing the real-time actual point cloud image and the real-time predicted point cloud image;

[0041] Using a registration method based on point cloud features to obtain an initial registration result between the real-time actual point cloud image and the real-time predicted point cloud image, and using a feature matching algorithm to obtain an initial registration transformation matrix corresponding to the initial registration result;

[0042] Combining the initial registration result and the initial registration transformation matrix and using a fine registration algorithm to complete fine registration between the real-time actual point cloud image and the real-time predicted point cloud image;

[0043] The point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image after fine registration is calculated, and the point cloud difference includes point cloud distance difference, point cloud density difference and point cloud fitting surface difference.

[0044] In a second aspect, the present invention also provides a road safety detection system based on image recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the road safety detection method based on image recognition as described in the first aspect is implemented.

[0045] In a third aspect, the present invention further provides a road safety detection device, comprising: the road safety detection system based on image recognition described in the second aspect;

[0046] A Beidou positioning module connected to the road safety detection system;

[0047] A high-definition camera module connected to the road safety detection system;

[0048] A laser radar module connected to the road safety detection system;

[0049] An edge computing module is connected to the road safety detection system.

[0050] The beneficial effects of the present invention are:

[0051] By building a shared feature extraction network and multiple road detection branch networks in the edge computing module, efficient detection of different types of road anomalies can be achieved. At the same time, the shared feature extraction network can reduce repeated calculations and improve the operating efficiency of the system. By using a pre-labeled road detection image training set for training, the accuracy and robustness of road detection can be improved, so that accurate safety detection can be performed in a complex road environment. By receiving a list of road detection tasks and obtaining the identification of the detection task currently required, and then using the gating mechanism layer to assign tasks to the road detection branch network, it is possible to flexibly select road detection branch networks for different types of road detection tasks, thereby improving the application scope and adaptability of the system. At the same time, each road detection branch network uses its own region of interest pooling layer to screen out redundant image data and only retain the image region of interest. Therefore, in the overall recognition and detection process, not only can the detection accuracy of road safety detection be guaranteed, but also the computing power usage of the edge computing module can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic flow chart of a road safety detection method based on image recognition in one embodiment of the present application.

[0053] Figure 2 This is a schematic diagram of the network structure of a shared feature extraction network and multiple road detection branch networks in one implementation of the present application.

[0054] Figure 3 This is a schematic diagram of the network structure of the first convolutional layer in one implementation manner of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0056] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0057] Figure 1 FIG. 1 is a flow chart of a road safety detection method based on image recognition in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0058] The present invention discloses a road safety detection device, including a road safety detection system based on image recognition, and the road safety detection device also includes:

[0059] Beidou positioning module, connected to the road safety detection system;

[0060] High-definition camera module, connected to the road safety detection system;

[0061] LiDAR module, connected to the road safety detection system;

[0062] Edge computing module, connected to the road safety detection system.

[0063] In one embodiment, when starting the road safety detection device, first ensure that all modules are working properly, including the Beidou positioning module, high-definition camera module, lidar module and edge computing module. The initial position and time information of the current device are obtained through the Beidou positioning module and synchronized with the road safety detection system. The Beidou positioning module uses satellite signals to determine the latitude, longitude and altitude position of the device to ensure that the device can accurately know its position on the earth. In addition, it is necessary to check the power and connection status of each module to ensure that they can communicate and work normally. The initialization process also includes loading the necessary software and algorithms to ensure that the device can correctly perform subsequent detection and control tasks.

[0064] Use high-definition camera modules and laser radar modules to collect images and distance information of the road environment in real time. The high-definition camera module captures image information in front of the road through a high-resolution camera. These images include lane lines, road signs, obstacles, etc. The laser radar module generates three-dimensional point cloud data around the road by emitting laser beams and receiving reflected signals, providing accurate distance and shape information. These data are transmitted to the edge computing module via a high-speed data bus. The edge computing module pre-processes the image data, including denoising, contrast enhancement, and color correction, and performs point cloud processing on the laser radar data to remove noise and generate an accurate three-dimensional model. The pre-processed data provides a reliable foundation for subsequent lane detection and path planning.

[0065] In the edge computing module, the shared feature extraction network and the road detection branch network are used to analyze the preprocessed image and lidar data to identify the road lane lines and surrounding environment features. The shared feature extraction network extracts representative features from the image and point cloud data, such as the shape and position of the lane lines, the road edges, obstacles, etc. through a deep learning algorithm. These features are then passed to multiple road detection branch networks, each of which specializes in a specific type of road detection task, such as lane line detection, obstacle detection, etc. In this way, the system can accurately identify various important information on the road and generate detailed detection results. These detection results include not only the position and shape of the lane lines, but also detailed information such as the type and position of obstacles.

[0066] Based on the lane detection results and real-time location data, the path planning algorithm in the edge computing module is used to generate a driving path that stays in the middle lane. The path planning algorithm first determines the lane the device is currently in based on the position and shape of the lane line, and then calculates a smooth and safe driving path to ensure that the device can stay in the middle lane. The algorithm also takes into account the curvature, slope and possible obstacles of the road ahead, and dynamically adjusts the driving path to avoid collisions and lane deviations. The result of path planning is a series of continuous control points that define how the device should adjust its direction and speed to stay on a safe driving path. The generated driving path is converted into specific control instructions (such as steering angle, acceleration, deceleration, etc.) and transmitted to the control unit of the road safety detection system. The control unit adjusts the driving state of the device according to these instructions to ensure that the device drives smoothly in the middle lane along the planned path. Specifically, the control unit adjusts the angle of the steering wheel according to the steering angle instruction, and adjusts the throttle and brake force according to the acceleration and deceleration instructions. These operations are accurately performed through the electronic control system to ensure that the device can drive smoothly and safely along the planned path. The control unit also monitors the driving state of the device in real time and adjusts the control instructions in time to ensure that the device can cope with various complex road environments.

[0067] During driving, the high-definition camera module and the laser radar module are continuously used to collect real-time data and transmit it to the edge computing module for analysis. If changes in the road environment are detected (such as blurred lane lines, obstacles in front, etc.), the edge computing module should re-plan the path in time and adjust the control instructions to ensure that the device always stays in the middle lane and drives safely. The process of real-time data collection and analysis is continuous, ensuring that the device can respond to changes in the road environment in a timely manner. For example, when the camera detects that the lane line is blurred, the system will compensate through the laser radar data to ensure the accuracy of the driving path. When an obstacle is detected in front, the system will immediately re-plan the path to bypass the obstacle to ensure driving safety. If road abnormalities (such as potholes, obstacles, etc.) are detected, road abnormality results are generated through the road detection branch network. Combined with the real-time positioning information provided by the Beidou positioning module, road defect feedback information is generated and stored or uploaded to the central monitoring system for subsequent processing and maintenance. Specifically, when a road abnormality is detected, the system will record the location, type and severity of the abnormality, and send this information to the central monitoring system through the wireless communication module. After receiving feedback information, the central monitoring system can dispatch maintenance personnel in a timely manner to handle the situation and ensure the safety and smooth flow of roads.

[0068] After completing the scheduled unmanned road safety inspection task, the device should be safely parked at the designated location. The road environment data, inspection results, path planning records and other information collected during the entire mission are stored in the road safety inspection system for subsequent analysis and improvement. After the task is completed, the system will automatically organize, compress and store all data in the internal storage or upload it to the cloud database. This data can not only be used for subsequent road maintenance and management, but also can be used to optimize and improve the algorithms and performance of road safety inspection devices, and improve the efficiency and accuracy of future tasks.

[0069] In one embodiment, the present invention further discloses a road safety detection system based on image recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following is achieved: Figure 1 The road safety detection method based on image recognition is shown.

[0070] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0071] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the computer device, etc., and the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0072] like Figure 1 As shown, a road safety detection method based on image recognition disclosed in the present invention is applied to a road safety detection device, and the road safety detection device is equipped with a Beidou positioning module, a high-definition camera module and an edge computing module. The road safety detection method based on image recognition disclosed in the present invention specifically includes the following steps:

[0073] S101. Construct a shared feature extraction network and multiple road detection branch networks in the edge computing module.

[0074] Among them, refer to Figure 2, multiple road detection branch networks are all connected to the last gating mechanism layer in the shared feature extraction network, and all road detection branch networks include a region of interest pooling layer. In this embodiment, the shared feature extraction network also includes a shared feature extraction layer, and the shared feature extraction layer includes an image input layer, a first convolutional layer, a first fully connected layer, and a task classification output layer connected in sequence, and the task classification output layer is connected to the gating mechanism layer. The road detection branch network also includes a feature image input layer and a feature classification layer, the feature image input layer is connected to the gating mechanism layer, the region of interest pooling layer is connected to the feature image input layer, the feature classification layer includes a second convolutional layer, a second fully connected layer, and a feature classification output layer connected in sequence, and the second convolutional layer is connected to the region of interest pooling layer.

[0075] In one embodiment, Figure 2 As shown, the road detection branch network includes a road disease detection branch network for detecting road diseases (such as road cracks, road potholes, etc.), a safety facility detection branch network for detecting safety facility anomalies (such as damaged road guardrails, abnormal green plants in median strips, etc.), and a road foreign object detection branch network for detecting road foreign objects (such as spilled objects, thrown objects, etc.). The shared convolutional neural network can preliminarily identify the road disease area, safety facility abnormal area and road foreign object area in the input image, and mark three different abnormal feature identifiers for the above three identified areas. The shared convolutional neural network can also mark three different task category identifiers for the three identified areas. The gating mechanism layer can identify the task category identifier and transmit the data stream of the shared convolutional neural network to the corresponding road detection branch network according to the identification result.

[0076] After the road detection branch network obtains the image with abnormal feature identification transmitted by the shared convolutional neural network, the road detection branch network can cut out the uninterested area through the region of interest pooling layer and only retain the area with abnormal feature identification in the image. Each road detection branch network can further identify the abnormal type of the abnormal area based on the retained image area with abnormal feature identification. For example, the pavement disease detection branch network receives the image transmitted by the shared convolutional neural network, and the image is marked with abnormal feature identification corresponding to the pavement disease. The pavement disease detection branch network will cut out all the areas without abnormal feature identification corresponding to the pavement disease, and further perform feature recognition on the retained part to determine whether the pavement disease in the image is a pavement crack, a pothole or other disease.

[0077] Through this structural design, the shared feature extraction network can effectively extract common features in the image, while multiple road detection branch networks focus on different types of road anomaly detection tasks. The shared feature extraction network can reduce repeated calculations, improve operating efficiency and reduce the computing power usage of the edge computing module. By using the pre-labeled road detection image training set for training, the accuracy and robustness of road detection can be improved, so that accurate safety detection can be performed in complex road environments. By receiving the road detection task list and obtaining the identification of the detection task currently required to be executed, and then using the gating mechanism layer to assign tasks to the road detection branch network, it is possible to flexibly select the road detection branch network for different types of road detection tasks, improve the application scope and adaptability of the system, and at the same time, each road detection branch network uses its own region of interest pooling layer to screen out redundant image data and only retain the image region of interest. Therefore, in the overall recognition and detection process, not only can the detection accuracy of road safety detection be guaranteed, but also the computing power usage of the edge computing module can be greatly reduced.

[0078] Reference Figure 3 , Figure 3 A schematic diagram of the network structure of the first convolutional layer in one implementation is shown. Figure 3 As shown, the first convolutional layer may include multiple convolutional layer groups connected in sequence, and the specific configuration of each convolutional layer group is as follows:

[0079] Convolutional Layer: Use multiple convolution kernels to extract image features such as edges and textures.

[0080] Batch Normalization Layer: Normalizes the convolution output to speed up training and stabilize the model.

[0081] Activation layer (ReLU Activation Layer): Introduces nonlinearity, enabling the network to learn more complex features.

[0082] Max Pooling Layer: Performs spatial downsampling to reduce the size of the feature map while retaining important features.

[0083] The convolution layers in each convolution layer group are configured with convolution kernels of different sizes and numbers, so that multi-dimensional features of the image can be extracted through multiple convolution layer groups.

[0084] S102. Obtain a road detection image training set, and use the road detection image training set to complete the training of the shared feature extraction network and all road detection branch networks.

[0085] Among them, all training images in the road detection image training set have pre-marked feature category identifiers and task category identifiers. In this embodiment, the feature category identifier includes multiple abnormal feature category identifiers, specifically including road surface disease feature identifiers, road safety facility abnormal feature identifiers, and road surface foreign body feature identifiers. Each abnormal feature category identifier contains multiple abnormal feature subcategory identifiers, among which the road surface disease feature identifier specifically includes road surface crack feature identifiers, road surface pothole feature identifiers, road surface repair plot feature identifiers, etc. The road safety facility abnormal feature identifier specifically includes traffic sign abnormal identifiers, curb abnormal identifiers, road guardrail abnormal identifiers, median strip green plant abnormal identifiers, anti-glare board abnormal identifiers, bridge head link device abnormal identifiers, etc. The road surface foreign body feature identifier specifically includes road surface dropped object identifiers, road surface cone barrel identifiers, etc.

[0086] During the training process, the shared feature extraction network is first trained to extract common features from the image, and then the individual road detection branch networks are trained to accurately identify specific types of road anomalies. The training process uses a supervised learning method, and the network parameters are continuously adjusted through the back-propagation algorithm to minimize the prediction error. In this way, the trained network can efficiently and accurately complete various road detection tasks, improving detection accuracy and efficiency.

[0087] S103. Receive a road detection task list and obtain a detection task identifier that currently needs to be executed based on the road detection task list.

[0088] Among them, the task list is usually generated by the central monitoring system and contains multiple detection tasks to be executed. Each task has a specific detection target. The road detection task list can specifically include pavement disease detection tasks, road safety facility detection tasks, and pavement foreign object detection tasks. The task identifier is used to indicate the specific road anomaly type that needs to be detected at the moment, such as pavement disease detection task identifier, road safety facility detection task identifier, pavement foreign object detection task identifier, etc. By analyzing the task list, the task identifier that needs to be executed is determined and passed to the shared feature extraction network in the edge computing module to guide subsequent image processing and feature detection. This step can flexibly adjust the detection strategy according to actual needs to improve detection efficiency and accuracy.

[0089] S104. While the road safety detection device is executing the road detection task according to the road detection task list, real-time image information of the surrounding road environment is collected through a high-definition camera module.

[0090] The high-definition camera module is installed at the front of the detection device, which can capture high-resolution images of the road ahead. The resolution and frame rate of the camera must be high enough to ensure that clear and continuous image data is captured, covering every detail of the road. Real-time image information is transmitted to the edge computing module via the data bus, and the edge computing module pre-processes and analyzes these images. The use of the high-definition camera module ensures that the system can obtain high-quality image data, providing a reliable foundation for subsequent feature extraction and anomaly detection.

[0091] S105. Preprocess the real-time image information, transmit the detection task identifier and the preprocessed real-time image information to the shared feature extraction network in the edge computing module, and output the road detection result through one or more road detection branch networks.

[0092] The preprocessing process includes steps such as image denoising, contrast enhancement, and color correction to improve the quality and clarity of the image. Denoising technology can eliminate random noise in the image, contrast enhancement technology can highlight important features in the image, and color correction is used to adjust the color balance of the image to make it more natural and realistic. Figure 2 As shown in the figure, the preprocessed image information is transmitted to the shared feature extraction network together with the detection task identifier, and the shared feature extraction network selects the appropriate road detection branch network for further processing according to the task identifier. In this way, various road detection tasks can be completed efficiently and accurately, and detailed detection results can be generated.

[0093] S106. If there are abnormal road results in the road detection results, the real-time positioning information of the road safety detection device is obtained through the Beidou positioning module.

[0094] Among them, the Beidou positioning module receives satellite signals and calculates the latitude, longitude and altitude of the device in real time, with positioning accuracy reaching sub-meter level. The positioning information is combined with the road detection results to generate road defect feedback information containing detailed information such as anomaly type and location. The use of the Beidou positioning module ensures that the anomaly detection results have accurate geographical locations, providing reliable data support for subsequent maintenance and management.

[0095] S107. Generate road defect feedback information by combining the road anomaly result and the real-time positioning information.

[0096] Among them, the road anomaly results include detailed information such as the anomaly type and severity, and the real-time positioning information provides the specific geographical location of the anomaly. This information is integrated to generate road defect feedback information, which includes details such as the anomaly type, location, and severity. The feedback information is transmitted to the central monitoring system through the wireless communication module. After receiving the feedback information, the central monitoring system can dispatch maintenance personnel in a timely manner to ensure the safety and smoothness of the road. For example, when a traffic sign is detected to be abnormal, the feedback information will include the sign's abnormality type (such as damage or missing), specific location (latitude and longitude) and severity (such as whether it affects traffic safety).

[0097] In one embodiment, the step of completing the training of the shared feature extraction network and all road detection branch networks using the road detection image training set in step S102 specifically includes the following steps:

[0098] Initialize the network parameters of the shared feature extraction network and all road detection branch networks;

[0099] Input all training images in the road detection image training set into the shared feature extraction layer through the image input layer, train the shared feature extraction layer in combination with the training images and feature category identifiers, and construct the first loss function of the shared feature extraction layer training process;

[0100] Input the training feature image output by the shared feature extraction layer into the gating mechanism layer, train the gating mechanism layer in combination with the training feature image and the task category identifier, and construct a second loss function for the training process of the gating mechanism layer;

[0101] The training feature image is passed to the road detection branch network of the corresponding category through the gating mechanism layer and according to the task category identification;

[0102] For each road detection branch network, the region of interest pooling layer is trained by combining the abnormal feature category identification and the training feature image, and the third loss function of the region of interest pooling layer training process is constructed;

[0103] Input the training feature image fragment output by the region of interest pooling layer into the feature classification layer, train the feature classification layer in combination with the training feature image fragment and the abnormal feature subclass identifier, and construct the fourth loss function of the feature classification layer training process;

[0104] The total loss of model training is calculated by combining the first loss function, the second loss function, the third loss function and the fourth loss function. The total loss of model training is back-propagated and all network parameters are iteratively updated. All the above training steps are repeated until the number of iterative updates reaches the preset maximum number of iterations.

[0105] In this embodiment, the shared feature extraction network is usually a deep convolutional neural network (such as ResNet, VGG, etc.), which is used to extract high-level features from the input image. Initializing these network parameters can be achieved by a variety of methods, such as random initialization, parameter initialization of pre-trained models, etc. Random initialization usually uses Gaussian distribution or uniform distribution to generate initial weights, while parameter initialization of pre-trained models uses model parameters pre-trained on large-scale data sets (such as ImageNet). The purpose of initialization is to provide a reasonable starting point for the network to ensure that the gradient update during training can be carried out effectively, thereby accelerating convergence and improving the performance of the model.

[0106] All training images in the road detection image training set are input to the shared feature extraction layer through the image input layer in order to allow the network to learn the underlying features in the image. The image input layer is usually a simple preprocessing module that normalizes and resizes the input image to make it suitable for input into the deep neural network. The shared feature extraction layer extracts feature maps with semantic information from the input image through multi-layer convolution, pooling and other operations. These feature maps contain information such as edges, textures, and shapes in the image, and are the basis for high-level understanding of subsequent tasks. When training the shared feature extraction layer, the training images and feature category identifiers are combined to guide the learning of the network by constructing the first loss function. The first loss function measures the gap between the network output and the true annotation, and updates the network parameters through the back-propagation algorithm, so that the network can gradually learn effective feature representations.

[0107] The gating mechanism layer is designed to dynamically allocate training feature images according to the task category identifier. Specifically, the gating mechanism layer consists of a fully connected layer and a Softmax activation function. The fully connected layer receives the output of the shared feature extraction layer and generates a vector, which is converted into the probability distribution of the task category after the Softmax activation function. The second loss function is constructed during the training process. By minimizing the second loss function, the parameters of the gating mechanism layer are adjusted so that it can accurately output the task category probability distribution. The task category probability distribution output by the gating mechanism layer is used to select the corresponding road detection branch network. In the specific implementation, the output of the gating mechanism layer is used as a weight, multiplied by the training feature image, and the weighted feature image is obtained, which is input into the corresponding road detection branch network. In this way, the training feature image can be passed to the corresponding category of road detection branch network through the gating mechanism layer and according to the task category identifier, ensuring that each branch network can focus on a specific task, improve the accuracy and efficiency of detection, and reduce the computing power consumption of each branch network.

[0108] Each road detection branch network is optimized for a specific road detection task. Each branch network further processes the input features through multiple layers of convolution, pooling and fully connected layers, extracts features that are highly relevant to the current task, and performs classification or regression operations. The entire network architecture can handle multiple road detection tasks at the same time, improving the versatility and efficiency of the model. For each road detection branch network, the region of interest pooling layer is trained in combination with the abnormal feature category identification and the training feature image, so that the region of interest (RoI) can be located and extracted in the feature map. The region of interest pooling layer (RoIPooling) can crop and resize the specified area in the feature map to a fixed size for subsequent processing. When training the RoIPooling layer, the abnormal feature category identification and the training feature image are combined, and a third loss function is constructed. The third loss function is usually a classification loss or regression loss based on the RoI, which guides the RoIPooling layer to learn how to accurately locate and extract the region of interest. In this way, the RoIPooling layer can effectively extract the feature area related to the current task and improve the accuracy and robustness of detection.

[0109] The training feature image fragments output by the region of interest pooling layer are input to the feature classification layer. The feature classification layer is trained in combination with the training feature image fragments and the abnormal feature sub-category identification in order to finely classify the extracted feature regions. The feature classification layer is usually composed of multiple layers of fully connected layers or convolutional layers, which further processes and classifies the input feature image fragments. In combination with the training feature image fragments and the abnormal feature sub-category identification, the fourth loss function is constructed to guide the training of the feature classification layer. The fourth loss function measures the gap between the output of the feature classification layer and the true annotation, and updates the network parameters through the back-propagation algorithm, so that the feature classification layer can gradually learn the fine category distinction ability.

[0110] The total loss of model training is calculated by combining the first loss function, the second loss function, the third loss function and the fourth loss function in order to comprehensively consider the learning effect of each subtask. The total loss function is usually the weighted sum of the loss functions of each subtask, where the weight can be adjusted according to the importance or difficulty of the task. Through this weighted sum method, the model can balance the learning effect of each task in multi-task learning, avoiding excessive focus on one task and ignoring other tasks. Back-propagating the total loss of model training and iteratively updating all network parameters are key steps in the training process. The back-propagation algorithm guides the update of parameters by calculating the gradient of the total loss function to the network parameters. Specifically, the gradient of each parameter to the total loss is first calculated by the chain rule, and then the optimization algorithm (such as stochastic gradient descent, Adam, etc.) is used to update the parameters. Through repeated iterative updates, the network parameters gradually converge, so that the performance of the model on the training set continues to improve. The whole process repeats all the above training steps until the number of iterative updates reaches the preset maximum number of iterations, thereby ensuring sufficient training and performance optimization of the model.

[0111] In this embodiment, the first loss function is the GIoU loss function. The generalized intersection-over-union (GIoU) ​​loss function is a loss function for target detection, which improves the traditional intersection-over-union (IoU) loss function. GIoU not only considers the overlapping area of ​​the target box and the prediction box, but also the distance between them. Compared with IoU loss, GIoU loss provides a more stable gradient, can converge faster, and reduce training time and computing power consumption. In addition, GIoU loss shows higher accuracy when processing target detection tasks, which helps the shared feature extraction layer to learn the target position in the image more accurately.

[0112] The second loss function is the binary cross entropy loss function, which is a standard tool for dealing with binary classification problems. The binary cross entropy loss function measures the difference between the predicted probability and the true label, which is crucial for the gating mechanism layer to distinguish task categories. The binary cross entropy loss function is suitable for the training of the gating mechanism layer because the task of this layer is to assign feature images according to the task category identifier (usually binary classification). The use of binary cross entropy loss can effectively measure the difference between the predicted probability and the actual category, prompting the network to assign tasks more accurately, thereby improving the accuracy and efficiency of the overall detection. Moreover, the calculation of the binary cross entropy loss function is very simple, requiring only a simple calculation of the predicted probability and the true label of each sample, which makes the binary cross entropy loss function very efficient, thereby reducing training time and computing power consumption.

[0113] The third loss function is the L1 loss function, which measures the absolute difference between the predicted value and the true value. The L1 loss function is suitable for the training of the region of interest pooling layer, because this layer needs to accurately locate and extract the key areas in the image. The L1 loss is insensitive to outliers and can update the network parameters more stably, ensuring that the pooling layer can accurately extract the region of interest and improve the ability to capture key features. In the task of processing the large category identification of abnormal features, a loss function that is insensitive to outliers is required, and the L1 loss function just meets this requirement. Since the L1 loss function is insensitive to outliers, the L1 loss function can provide more stable error feedback during training, thereby improving the robustness of the model. In addition, the calculation process of the L1 loss function is very simple, and only the absolute difference between the predicted value and the true value needs to be calculated, which makes its calculation efficiency very high, thereby reducing computing power consumption and improving training efficiency.

[0114] The fourth loss function is the multivariate cross entropy loss function, which is suitable for the training of the feature classification layer because this layer needs to classify the feature image fragments into multiple categories. The use of multivariate cross entropy loss can effectively measure the difference between the predicted distribution and the actual distribution, prompting the network to classify more accurately and improve the recognition ability of different abnormal feature subcategories. When dealing with multi-classification problems, a loss function that can handle multiple categories is required, and the multivariate cross entropy loss function just meets this requirement. The multivariate cross entropy loss function measures the difference between the predicted probability distribution and the true label distribution, which is crucial for the classification task of dealing with abnormal feature subcategories. Although the calculation process of the multivariate cross entropy loss function is slightly more complicated than that of the binary cross entropy loss function, its computational efficiency is still very high, so it can effectively reduce training time and computing power consumption. In addition, since the multivariate cross entropy loss function can handle classification problems of multiple categories, it can improve the accuracy and robustness of the model when dealing with complex classification tasks.

[0115] In this embodiment, the total loss of model training is calculated as follows:

[0116]

[0117] Where: L 0 represents the total loss of model training, α represents the weight of the first loss function, L 1 represents the first loss calculated by the first loss function, β represents the weight of the second loss function, L 2 Indicates that the second loss function is calculated to obtain the second loss, γ i represents the weight of the third loss function in the i-th road detection branch network, Indicates the third loss calculated by the third loss function in the i-th road detection branch network, δ irepresents the weight of the fourth loss function in the i-th road detection branch network, It indicates that the fourth loss is calculated by the fourth loss function in the i-th road detection branch network, and n indicates the number of road detection branch networks.

[0118] In one embodiment, the road safety detection device is further equipped with a laser radar module, the road detection task list includes a road disease detection task, a road safety facility detection task, and a road foreign body detection task, and the road safety detection method based on image recognition further includes the following steps:

[0119] When the road safety detection device starts to perform the road detection task according to the road detection task list, the timestamp alignment technology is used to synchronize the laser radar module, Beidou positioning module, high-definition camera module and edge computing module;

[0120] The initial positioning information of the road safety detection device is obtained through the Beidou positioning module, and the initial laser radar data of the surrounding road environment is collected through the laser radar module. After the initial laser radar data is collected, the laser radar module is switched to a standby state;

[0121] Obtaining initial operating status data of a road safety detection device;

[0122] The initial operation status data, initial positioning information and initial LiDAR data are transmitted to the edge computing module, and the initial actual point cloud map of the surrounding road environment is generated by combining the initial operation status data, initial positioning information and initial LiDAR data with the edge computing module;

[0123] When the road safety detection device performs the road detection task according to the road detection task list, real-time operation status data of the road safety detection device is obtained, and the real-time operation status data is transmitted to the edge computing module;

[0124] The edge computing module is used to predict the point cloud state of the initial actual point cloud map based on the real-time operation status data and the Kalman filter method to obtain a real-time predicted point cloud map;

[0125] If there is an abnormal road result in the road detection result, the laser radar module is switched to the working state, and the real-time laser radar data of the surrounding road environment at the current moment is collected by the laser radar module. After the real-time laser radar data is collected, the laser radar module is switched to the standby state;

[0126] The real-time LiDAR data is transmitted to the edge computing module, and the edge computing module is used to generate a real-time actual point cloud map of the surrounding road environment at the current moment by combining the real-time operation status data, real-time positioning information and real-time LiDAR data;

[0127] The point cloud registration algorithm is used to calculate the point cloud difference between the real-time actual point cloud map and the real-time predicted point cloud map;

[0128] Point cloud differences are used to verify road anomaly results, and road anomaly results that fail verification are removed.

[0129] In this embodiment, the timestamp alignment technology is to ensure the data synchronization of multiple sensor modules, thereby improving the accuracy of data fusion. First, when the laser radar module, Beidou positioning module, high-definition camera module and edge computing module collect data, they will be accompanied by a timestamp, which records the specific time of data collection. By aligning these timestamps, it can be ensured that the data collected by different sensors are consistent at the same time point. For example, when the laser radar collects a frame of data, this frame of data will be accompanied by a timestamp; similarly, the Beidou positioning module will obtain positioning information and the high-definition camera module will also be accompanied by a timestamp when taking images. The edge computing module will align the data of different sensors according to these timestamps. In specific implementation, a distributed clock synchronization protocol (such as PTP-Precision TimeProtocol) can be used to ensure that the clocks of each module are consistent, thereby achieving high-precision time alignment. In this way, it can be ensured that the laser radar, Beidou positioning and high-definition camera data can be accurately matched at the same time, avoiding data errors caused by time asynchrony.

[0130] When the road safety detection device starts to perform the road detection task according to the road detection task list, the Beidou positioning module first starts and obtains the initial positioning information of the current device, including data such as longitude, latitude and altitude. These positioning information can provide the initial geographical location of the device. At the same time, the laser radar module starts to work and collects initial laser radar data of the surrounding road environment. The laser radar measures the distance and shape of surrounding objects by emitting laser beams and receiving reflected signals to generate a three-dimensional point cloud map. These point cloud data can describe the geometric structure of the surrounding environment in detail. After collecting enough initial laser radar data, the laser radar module switches to standby mode to save energy and reduce unnecessary data processing burden. In specific implementation, a threshold can be set, and when the point cloud data collected by the laser radar reaches a certain number or covers a certain range, it automatically switches to standby mode. In this way, the integrity of the initial environmental data can be ensured while optimizing energy consumption.

[0131] The initial operating state data includes dynamic information such as the speed, acceleration, and azimuth of the device, which can be obtained through the built-in inertial measurement unit (IMU). The IMU can monitor the motion state of the device in real time and provide high-precision acceleration and angular velocity information. In specific implementation, this information can be obtained by reading the data register of the IMU sensor, and a timestamp is attached to ensure the time consistency of the data. The initial operating state data can help the edge computing module understand the initial motion state of the device and provide a reference for subsequent data processing and point cloud generation. For example, the speed and azimuth of the device can be used to predict future motion trajectories, thereby improving the accuracy of the point cloud map. The initial operating state data, initial positioning information, and initial lidar data are transmitted to the edge computing module through a high-bandwidth communication interface. After receiving these data, the edge computing module first performs data preprocessing, including denoising, filtering, and coordinate conversion. Then, the lidar point cloud data is converted to geographic coordinates using the initial positioning information to generate a three-dimensional point cloud map based on geographic coordinates. The edge computing module dynamically adjusts the point cloud data in combination with the initial operating state data to compensate for the point cloud error caused by the movement of the device. Finally, a high-precision initial actual point cloud map is generated, which describes the geometric structure and characteristics of the surrounding road environment in detail. In this way, the high accuracy and high consistency of the initial actual point cloud map can be ensured, providing a reliable environmental data foundation for subsequent road detection tasks.

[0132] In the process of performing road detection tasks, the operating status data of the road safety detection device, including speed, acceleration, direction angle, etc., are obtained in real time. These real-time operating status data are obtained through the built-in IMU sensor and are accompanied by a timestamp to ensure the time consistency of the data. The real-time operating status data can be transmitted to the edge computing module through a wireless communication interface (such as Wi-Fi or 5G). In specific implementation, the data acquisition frequency and transmission frequency can be set to ensure real-time and data integrity. The real-time operating status data provides the edge computing module with the current motion state information of the device to help it make dynamic adjustments and predictions. In this way, the accuracy and timeliness of the real-time operating status data can be ensured, and reliable dynamic data support can be provided for the edge computing module. After receiving the real-time operating status data, the edge computing module uses the Kalman filter method to predict the state of the initial actual point cloud map. Kalman filtering is a recursive algorithm that can optimally estimate the system state based on the dynamic model and observation data of the system. In specific implementation, the state space model of the point cloud map is first established, including the state transfer matrix and the observation matrix. Then, the real-time operating status data is used as the observation data to predict and update the state of the point cloud map.

[0133] During the execution of the road detection task, if there are abnormal road results in the detection results, such as road damage, obstacles, etc., the laser radar module is immediately switched to the working state. The laser radar module starts to collect real-time laser radar data of the surrounding road environment at the current moment. The laser radar measures the distance and shape of surrounding objects by emitting laser beams and receiving reflected signals to generate a high-precision three-dimensional point cloud map. After collecting enough real-time laser radar data, the laser radar module switches to the standby state to save energy and reduce unnecessary data processing burden. The edge computing module receives the real-time laser radar data and first performs data preprocessing, including denoising, filtering, and coordinate conversion. Then, the laser radar point cloud data is converted into geographic coordinates using real-time positioning information to generate a three-dimensional point cloud map based on geographic coordinates. The edge computing module combines the real-time operation status data to dynamically adjust the point cloud data, compensate for the point cloud error caused by the movement of the device, and finally generate a high-precision real-time actual point cloud map, which describes the geometric structure and characteristics of the surrounding road environment at the current moment in detail.

[0134] The point cloud registration algorithm is used to calculate the point cloud difference between the real-time actual point cloud map and the real-time predicted point cloud map. The point cloud registration algorithm is used to align and compare the point cloud maps at two different time points, so as to calculate the point cloud difference between them. In specific implementation, the ICP (Iterative Closest Point) algorithm can be used for point cloud registration. First, the real-time actual point cloud map and the real-time predicted point cloud map are preliminarily aligned, and then the distance error between the two is minimized through an iterative optimization process, and finally high-precision point cloud registration is achieved. By calculating the difference in the registered point cloud, the changed parts in the environment can be identified, such as new obstacles or road damage. The calculation of the point cloud difference can be quantified by indicators such as point cloud distance and shape change. In this way, changes in the road environment can be accurately identified, providing reliable data support for subsequent road anomaly verification.

[0135] Use the calculated point cloud difference to verify the abnormal part of the road detection results. In specific implementation, obtain the point cloud difference data between the real-time actual point cloud map and the real-time predicted point cloud map from the previous step, including point-to-point distance, point cloud density difference, surface difference, etc. Extract the preliminary detected road anomaly results, including information such as the location, type and severity of the anomaly. Define the verification standard based on the point cloud difference data, and the verification standard should correspond to the type and severity of the road anomaly. For each preliminary detected road anomaly result, perform the following verification steps:

[0136] Match the location of the road anomaly result with the point cloud difference data to find the corresponding point cloud area;

[0137] In the corresponding point cloud area, indicators such as point-to-point distance, point cloud density difference and surface difference are calculated.

[0138] The calculated difference index is compared with the pre-defined validation criteria.

[0139] Based on the comparison result, determine whether the road abnormality result has passed the verification.

[0140] For the road anomaly results that pass the verification, keep them as one of the final anomaly detection results. For the road anomaly results that fail the verification, remove them and consider them as false positives or detection errors. Update the road anomaly results that pass the verification to the final anomaly detection result list. Output the updated road anomaly detection results, including information such as the location, type, and severity of the anomaly. Through the above steps, the preliminary detected road anomaly results are verified using point cloud difference data, and the road anomaly results that fail the verification are removed, thereby improving the accuracy and reliability of road anomaly detection.

[0141] In one embodiment, the step of using a point cloud registration algorithm to calculate the point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image specifically includes the following steps:

[0142] Preprocessing real-time actual point cloud map and real-time predicted point cloud map;

[0143] A registration method based on point cloud features is used to obtain the initial registration result between the real-time actual point cloud image and the real-time predicted point cloud image, and a feature matching algorithm is used to obtain the initial registration transformation matrix corresponding to the initial registration result;

[0144] Combine the initial registration result and the initial registration transformation matrix and use the fine registration algorithm to complete the fine registration between the real-time actual point cloud image and the real-time predicted point cloud image;

[0145] The point cloud difference between the real point cloud image and the real-time predicted point cloud image after fine registration is calculated. The point cloud difference includes the point cloud distance difference, point cloud density difference and point cloud fitting surface difference.

[0146] In this embodiment, first of all, preprocessing the real-time actual point cloud map and the real-time predicted point cloud map is the basis of the entire technical solution. The main purpose of the preprocessing step is to clean and standardize the point cloud data so that subsequent processing is more efficient and accurate. Preprocessing includes steps such as denoising, downsampling and normal estimation. Denoising can be achieved by statistical filters or radius filters. The former determines whether it is a noise point based on the number of neighborhood points of each point, and the latter determines based on the number of points within the neighborhood radius of each point. Downsampling simplifies the point cloud data through a voxel grid filter to reduce the amount of calculation while maintaining the overall structure of the point cloud. Normal estimation is to calculate the normal direction of each point through the PCA (principal component analysis) method, which provides necessary information for subsequent feature extraction and alignment. The point cloud data after preprocessing will be neater and more standardized, laying a solid foundation for subsequent steps.

[0147] Next, a registration method based on point cloud features is used to obtain the initial registration results between the real-time actual point cloud map and the real-time predicted point cloud map. Feature extraction is the core of this step. Commonly used features include SIFT (scale-invariant feature transform) and FPFH (fast point feature histogram). These features can find corresponding feature points between different point cloud data, thereby achieving preliminary registration. In specific implementation, key point detection can be performed on the point cloud first to extract representative feature points. Then, these feature points are described by feature descriptors. For example, the FPFH feature descriptor can generate a descriptor by calculating the local geometric features of each point. Next, the feature points of the two point clouds are matched by a feature matching algorithm (such as the RANSAC random sampling consensus algorithm), and the initial registration transformation matrix is ​​calculated. The transformation matrix contains rotation and translation information, so that one point cloud can be roughly aligned to another point cloud. Although the initial registration result may have certain errors, it provides a good starting point for subsequent fine registration.

[0148] Combine the initial registration result and the initial registration transformation matrix, and use the fine registration algorithm to complete the fine registration between the real-time actual point cloud map and the real-time predicted point cloud map. Fine registration usually uses the ICP (Iterative Closest Point) algorithm, which continuously adjusts the relative positions between point clouds through iterative optimization to maximize their overlap. In specific implementation, the result of the initial registration is first used as the initial input of the ICP algorithm. Then, the ICP algorithm finds the closest point pair between the two point clouds in each iteration and calculates the error between these point pairs. Then, by minimizing these errors, the transformation matrix is ​​updated. This process is repeated until the error converges to a preset threshold or the maximum number of iterations is reached. Ultimately, the fine registration result will make the two point clouds highly aligned in space and minimize the error.

[0149] Finally, the point cloud difference between the real-time actual point cloud map and the real-time predicted point cloud map after fine registration is calculated. Point cloud differences include point cloud distance difference, point cloud density difference and point cloud fitting surface difference. Point cloud distance difference can be quantified by calculating the Euclidean distance from each point to its corresponding point, and counting the mean and variance of these distances. Point cloud density difference is measured by comparing the point density of two point clouds in the same spatial area. The voxel grid method can be used to divide the space into multiple small cubes, count the number of points in each cube and compare them. Point cloud fitting surface difference is calculated by fitting the point cloud surface (such as using a triangular mesh or surface fitting method). These difference indicators can fully reflect the geometric differences between the two point clouds and provide an important reference for subsequent analysis and application.

[0150] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as above, which are not provided in detail for the sake of simplicity.

[0151] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A road safety detection method based on image recognition, characterized in that: Applied to a road safety detection device, the road safety detection device is equipped with a Beidou positioning module, a high-definition camera module and an edge computing module, and the method comprises the following steps: A shared feature extraction network and a plurality of road detection branch networks are constructed in the edge computing module, wherein the plurality of road detection branch networks are all connected to the last gating mechanism layer in the shared feature extraction network, and all the road detection branch networks include a region of interest pooling layer. The shared feature extraction network also includes a shared feature extraction layer, and the shared feature extraction layer includes an image input layer, a first convolution layer, a first fully connected layer, and a task classification output layer connected in sequence, and the task classification output layer is connected to the gating mechanism layer. The road detection branch network also includes a feature image input layer and a feature classification layer, and the feature image input layer is connected to the gating mechanism layer, and the region of interest pooling layer is connected to the feature image input layer, and the feature classification layer includes a second convolution layer, a second fully connected layer, and a feature classification output layer connected in sequence, and the second convolution layer is connected to the region of interest pooling layer; Acquire a road detection image training set, and use the road detection image training set to complete the training of the shared feature extraction network and all the road detection branch networks, wherein all training images in the road detection image training set have pre-labeled feature category identifiers and task category identifiers, and the feature category identifiers include multiple abnormal feature category identifiers, and each of the abnormal feature category identifiers includes multiple abnormal feature subcategory identifiers; The method of using the road detection image training set to complete the training of the shared feature extraction network and all the road detection branch networks comprises the following steps: Initializing network parameters of the shared feature extraction network and all the road detection branch networks; Input all training images in the road detection image training set into the shared feature extraction layer through the image input layer, train the shared feature extraction layer in combination with the training images and the feature category identifiers, and construct a first loss function of the shared feature extraction layer training process; Inputting the training feature image output by the shared feature extraction layer into the gating mechanism layer, training the gating mechanism layer in combination with the training feature image and the task category identifier, and constructing a second loss function of the training process of the gating mechanism layer; Passing the training feature image to the road detection branch network of the corresponding category through the gating mechanism layer and according to the task category identifier; For each of the road detection branch networks, the region of interest pooling layer is trained in combination with the abnormal feature category identifier and the training feature image, and a third loss function of the region of interest pooling layer training process is constructed; Input the training feature image segment output by the region of interest pooling layer into the feature classification layer, train the feature classification layer in combination with the training feature image segment and the abnormal feature subclass identifier, and construct a fourth loss function of the feature classification layer training process; The total loss of model training is calculated by combining the first loss function, the second loss function, the third loss function and the fourth loss function, the total loss of model training is back-propagated and all the network parameters are iteratively updated, and all the above training steps are repeated until the number of iterative updates reaches a preset maximum number of iterations, the first loss function is a GIoU loss function, the second loss function is a binary cross entropy loss function, the third loss function is an L1 loss function, and the fourth loss function is a multivariate cross entropy loss function; The calculation formula for the total loss of the model training is as follows: ， Where: Represents the total loss of the model training, represents the weight of the first loss function, It means that the first loss is calculated by the first loss function. represents the weight of the second loss function, It means that the second loss is calculated by the second loss function. Indicates The weight of the third loss function in the road detection branch network, Indicates The third loss is calculated by the third loss function in the road detection branch network to obtain a third loss, Indicates The weight of the fourth loss function in the road detection branch network, Indicates The fourth loss is obtained by calculating the fourth loss function in the road detection branch network, represents the number of the road detection branch network; Receiving a road detection task list and obtaining a detection task identifier currently required to be executed based on the road detection task list; In the process of the road safety detection device performing the road detection task according to the road detection task list, collecting real-time image information of the surrounding road environment through the high-definition camera module; Preprocessing the real-time image information, transmitting the detection task identifier and the preprocessed real-time image information to the shared feature extraction network in the edge computing module, and outputting the road detection result through one or more road detection branch networks; If there is an abnormal road result in the road detection result, obtaining the real-time positioning information of the road safety detection device through the Beidou positioning module; The road defect feedback information is generated by combining the road anomaly result and the real-time positioning information.

2. The road safety detection method based on image recognition according to claim 1 is characterized in that: The road safety detection device is also equipped with a laser radar module, the road detection task list includes a road disease detection task, a road safety facility detection task and a road foreign body detection task, and the method further includes the following steps: When the road safety detection device starts to execute the road detection task according to the road detection task list, the laser radar module, the Beidou positioning module, the high-definition camera module and the edge computing module are synchronized using a timestamp alignment technology; Acquiring initial positioning information of the road safety detection device through the Beidou positioning module, and collecting initial laser radar data of the surrounding road environment through the laser radar module, and switching the laser radar module to a standby state after collecting the initial laser radar data; Acquiring initial operating status data of the road safety detection device; The initial operation state data, the initial positioning information and the initial laser radar data are transmitted to the edge computing module, and the initial actual point cloud map of the surrounding road environment is generated by combining the initial operation state data, the initial positioning information and the initial laser radar data with the edge computing module; In the process of the road safety detection device performing the road detection task according to the road detection task list, obtaining real-time operation status data of the road safety detection device, and transmitting the real-time operation status data to the edge computing module; Using the edge computing module to predict the point cloud state of the initial actual point cloud map according to the real-time operating status data and using a Kalman filter method to obtain a real-time predicted point cloud map; If there is an abnormal road result in the road detection result, the laser radar module is switched to a working state, and real-time laser radar data of the surrounding road environment at the current moment is collected by the laser radar module. After the real-time laser radar data is collected, the laser radar module is switched to a standby state; The real-time laser radar data is transmitted to the edge computing module, and the edge computing module is used to generate a real-time actual point cloud map of the surrounding road environment at the current moment in combination with the real-time operation status data, the real-time positioning information and the real-time laser radar data; Using a point cloud registration algorithm to calculate the point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image; The road anomaly results are verified using the point cloud differences, and the road anomaly results that fail verification are removed.

3. The road safety detection method based on image recognition according to claim 2 is characterized in that: The step of using a point cloud registration algorithm to calculate the point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image comprises the following steps: Preprocessing the real-time actual point cloud image and the real-time predicted point cloud image; Using a registration method based on point cloud features to obtain an initial registration result between the real-time actual point cloud image and the real-time predicted point cloud image, and using a feature matching algorithm to obtain an initial registration transformation matrix corresponding to the initial registration result; Combining the initial registration result and the initial registration transformation matrix and using a fine registration algorithm to complete fine registration between the real-time actual point cloud image and the real-time predicted point cloud image; The point cloud difference between the real-time actual point cloud image and the real-time predicted point cloud image after fine registration is calculated, and the point cloud difference includes point cloud distance difference, point cloud density difference and point cloud fitting surface difference.

4. A road safety detection system based on image recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the road safety detection method based on image recognition as described in any one of claims 1 to 3 is implemented.

5. A road safety detection device, characterized in that: include: The road safety detection system based on image recognition according to claim 4; A Beidou positioning module connected to the road safety detection system; A high-definition camera module connected to the road safety detection system; A laser radar module connected to the road safety detection system; An edge computing module is connected to the road safety detection system.

Images