YOLOv8-based curve target detection system

Abstract

First, a video surveillance system was built on rural roads to construct a curve target detection dataset. This dataset was then used to train a YOLOv8 algorithm model, achieving rapid identification of traffic targets on curves. To address the dynamic nature of rain and snow, a modified water accumulation sensor was used to assist in judgment. An expansion board with a dedicated GPIO interface was developed for the sensor, combined with temperature and humidity sensors, improving the accuracy of road water accumulation and rain / snow detection. Artificial intelligence algorithms were deployed on local edge nodes, effectively reducing costs, minimizing processing latency, increasing stability, and enabling flexible deployment unaffected by network conditions. The software framework is divided into four main parts, implementing information integration, judgment, and decision-making functions for intelligent interactive scenarios. This prioritizes the transmission of advanced warning information while minimizing the impact on drivers, and reports potential hazards to higher-level decision-making bodies to promptly prevent vehicles from entering the area.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation systems, and particularly relates to a real-time road condition detection system for curved roads and a method for providing early warnings to passing vehicles and pedestrians. By using the YOLOv8 algorithm, the road surface is monitored in real time. Water accumulation sensors and temperature sensors are used to determine whether there is water accumulation or ice and snow on the road. The monitoring status of the curve is displayed on the screen to remind passing pedestrians and drivers. Background Technology

[0002] Transportation is an essential infrastructure and vital lifeline for economic and social development. It is also an important indicator of a region's level of civilization and a concentrated display of its external image. Rural roads have the widest coverage, serve the largest number of people, provide the most widespread services, and have a strong public welfare nature. They are a basic condition for ensuring the production and life of rural residents, a leading and fundamental facility for agricultural development, and an important foundation for the continuous income growth of rural residents.

[0003] In recent years, my country's comprehensive transportation system has played an irreplaceable role. However, some prominent problems still exist within this system: mountain roads are characterized by sharp curves and steep slopes, with frequent rockfalls and debris spilled from vehicles. Due to terrain conditions, some sections are prone to water accumulation in low-lying areas, or severe ice and snow due to wind, seriously threatening public transportation safety. Heavy fog in mountainous areas renders safety warning systems relying solely on convex mirrors and radar speedometers inadequate, necessitating the enhancement of digital curve safety warning systems for rural roads. Curve target detection in the complex environment of rural roads differs from general target detection tasks, requiring the identification and detection of multiple targets such as curves, falling rocks, snow accumulation, water accumulation, road obstacles, and cracks. Currently available datasets mostly focus on identifying specific objects and are not entirely applicable to the road scenario proposed in this invention. Research on datasets for curve target detection in the complex environment of rural roads remains relatively scarce.

[0004] On the other hand, in curve target detection systems, dataset acquisition equipment needs to collect data at the same height as surveillance cameras. However, most existing datasets are taken by vehicle-mounted cameras at the same height as vehicles, resulting in significant differences in the acquisition perspective, target size, and sample features. Using existing publicly available road target detection datasets would directly lead to failure. Therefore, given the scarcity of datasets for curve target detection in the complex environment of rural roads, constructing a dataset suitable for curve target detection has significant guiding significance and reference value for accelerating the development of intelligent transportation in rural areas and rural road construction. There is an urgent need to improve the function of early warning for abnormal road conditions. This invention utilizes video data and intelligent sensing data from IoT sensors, employing data fusion, intelligent sensing, and artificial intelligence technologies to monitor, identify, and warn of targets such as vehicles on curves, road obstacles, and icy / water accumulation on the road surface. Information is disseminated through multiple channels, including scrolling screens and vehicle-road collaboration, to achieve timely early warning, reporting, and control, comprehensively improving the quality of safety services. Summary of the Invention

[0005] To achieve the above objectives, the technical approach of this invention is as follows: A detailed analysis and classification of various safety hazards and emergencies existing in the complex environment of rural roads is conducted. Based on rural roads, a video surveillance system is first built. Based on the large amount of comprehensive road data collected, a Curve Target Detection Dataset (RCODS) suitable for the complex environment of rural roads is constructed. Performance verification confirms the accuracy and usability of the dataset constructed in this invention. Subsequently, the YOLOv8 algorithm model is trained using the Curve Target Detection Dataset (RCODS) to achieve the goal of quickly identifying traffic targets on curves. Since rain and snow morphologies change freely and cannot be detected using standard target detection functions, a modified water accumulation sensor is used to assist in judgment. Simultaneously, to add a protocol logic chip to the modified water accumulation sensor, a dedicated expansion board for the GPIO interface is developed. Combined with the SHT3x-ARP temperature and humidity sensor from Sensirion, Switzerland, the system's performance is improved. This invention improves the accuracy of road water accumulation and rain / snow detection. Regarding hardware deployment, to avoid the problem in traditional AI solutions where software upgrades to detect new types of defects require placing the AI ​​algorithm in a data center, this invention deploys the AI ​​algorithm on local edge nodes. These system nodes include cameras, sensors, edge computing devices, antennas, and interactive devices. This effectively reduces costs, while also minimizing processing latency, increasing stability, and allowing for flexible deployment unaffected by network lines. As for the software component, the software framework is subdivided into four main parts: a system detection module, a Kafka module, a system analysis module, and a system information propagation module. The system detection module is responsible for target detection using the AI ​​algorithm; Kafka is the information transmission module; the system analysis module is the system's post-processing module, including sensor information post-processing and information release post-processing; and the information propagation module is mainly responsible for the interface between the system's main program output and the LED screen, thus integrating the hardware and software components of this invention.

[0006] Step 1: A video surveillance system is set up on rural roads to collect road data. Based on the large amount of comprehensive road data collected, a curve target detection dataset RCODS suitable for the complex environment of rural roads is constructed. The performance of the dataset is verified and the accuracy and usability of the dataset constructed in this invention are demonstrated. Then, the curve target detection dataset RCODS is used to train the YOLOv8 algorithm model to achieve the purpose of quickly identifying and detecting traffic targets on curves.

[0007] 1.1 Establish a video surveillance system suitable for rural roads to collect video data for specific road sections:

[0008] Firstly, the surveillance camera serves as the primary input device for data acquisition in this invention. All identification-related tasks within the system rely on the surveillance camera as the video input device, making its clarity and resolution paramount. The surveillance camera needs to provide a clear and sharp video stream during the day, while also possessing night vision capabilities to provide black-and-white night vision images, ensuring the system can still provide normal alerts at night. Therefore, this invention selected Dahua's 4-megapixel infrared zoom network camera, model DH-IPC-HFW3433F-Z. Two surveillance cameras were respectively installed on both sides of the curve to monitor the lane and collect video data.

[0009] 1.2 Video data is read and stored frame by frame, and suitable images are selected to construct the curve target detection dataset RCODS:

[0010] Since the acquired video data is continuous, extracting images frame by frame would generate a large amount of redundant data, occupying significant storage space and greatly increasing data processing time. Therefore, this invention adopts a method of extracting one image every ten frames. However, the video data acquisition process spans a considerable period, and extracting one image every ten frames results in an excessively large amount of image information. Therefore, further data redundancy removal is necessary. Thus, this invention introduces the imagehash function during the editing of the dataset algorithm script. The imagehash function is used to filter images, removing duplicates or highly similar images to further compress the dataset, but the resulting dataset is still large. Finally, every ten or so images in the compressed dataset are randomly selected and retained. This ensures the representativeness of the selected samples and effectively avoids problems caused by biases in the selection method. After the above series of operations, this invention processes long-term video footage, and the resulting sample images serve as the dataset required for the curve target detection system.

[0011] 1.3 Data Labeling and Classification:

[0012] The generated dataset images are complex, usually containing multiple categories of information such as weather, pedestrians, trucks, cars, traffic cones, headlights, and agricultural machinery. Select the information you need to identify and label and classify the target categories of the dataset. First, you need to download and install Anaconda and change the Python version to Python 3.8.5 to configure the environment. The detailed steps are as follows: (1) Click Start in the lower left corner of the screen, find the Anaconda folder in the application bar, and open Anaconda Prompt (anaconda). (2) Enter the following commands in the Prompt to create a virtual environment named labelme and install labelme. Note that when entering the commands, you need to change the default Python version to the installed Python version. Enter python-version in the opened command line window to check the installed Python version. (3) After the above operations are completed, if you want to check whether the installation is successful, you need to enter the command condainfo--env in the command line window to view all environments. If the labelme environment name appears, it means that the installation is successful.

[0013] Then start labeling and classifying the selected target categories in the system. The specific operations are as follows: (1) Put the dataset file dataset and the label name file label name into the same folder. Assume that the dataset is placed in the path D:\project_1\Qingling. (2) Open Anaconda Prompt, enter activate labelme to activate the environment, then enter D: to enter the D drive, and then enter cd D:\project_1\Qingling to enter the path where the dataset is placed. Here, you need to change the input to your own path. (3) Enter labelme dataset--labels labels_name.txt--flagssunny,rainy,snowy. A software window will appear. If there are corresponding labels on the right, it means that the creation is successful. The three options of Flags are sunny, rainy, and snowy. Label list is the target category you have selected. (4) Click opendir, select the dataset folder, then click File, select Save Automatically, uncheck Save with image, and click changeOutput dir. You can then create an empty label folder in the same path and select it. In this way, the labeled JSON file is saved in label. The completed operation is shown in the figure. (5) Select the image in the File list at the bottom right of the software and start annotation. (6) Select the weather in the Flags column, then click Edit, and select the rectangle to start annotating objects. Drag the rectangle to annotate the target category you selected in the image. Then click the corresponding object option. If the rectangle is dragged out of control, press ESC. If the annotation is wrong, click Edit polygons on the left to modify it. Finally, click Next Image on the left to annotate the next image.

[0014] 1.4 YOLOv8 Algorithm Model Training and Validation

[0015] After the dataset is constructed and labeled, this invention will perform dataset validation based on the YOLOv8 algorithm model. Since the dataset provides the necessary information for the curve target detection system, in order to enable the system to make accurate judgments, the YOLOv8 algorithm model needs to be trained on the dataset constructed in this invention. The YOLOv8 algorithm developed by Ultralytics can provide an efficient training mode for target detection systems. During training, the input dataset needs to be divided into training and test sets in a 9:1 ratio. Next, the environment and hyperparameters need to be configured. Hyperparameters are set and modified through a q1.yaml hyperparameter configuration file, setting the target detection categories to multiple required categories such as "car," "truck," and "person." Then, the YOLOv8 model is trained on the dataset constructed in this invention, iterating 150 times to obtain an optimal training weight for testing on the dataset.

[0016] Finally, after training the dataset to obtain the optimal training weights, we tested and verified the performance of the curve target detection system. We randomly selected other photos taken by curve video surveillance and put them into the curve target detection system to observe whether it could detect the pre-trained targets. We also derived the relationship between the confidence value, precision, and F1 score. We repeatedly set the confidence value until the curve target detection system could accurately identify the set targets and achieve high precision and F1 score.

[0017] Step 2: To address the issue that rain and snow patterns can change freely and cannot be detected using target detection functions, a modified water accumulation sensor was designed to assist in judgment. At the same time, a protocol logic chip was added to the modified water accumulation sensor, using a GPIO interface expansion board, combined with the Swiss Sensirion SHT3x-ARP temperature and humidity sensor, which improved the accuracy of road water accumulation and rain and snow detection.

[0018] 2.1 Fabrication and modification of the sensor

[0019] The most crucial element in the design of the water accumulation sensor is the comparator circuit. The circuit design first introduces an LM393 voltage comparator, which compares the voltage at the non-inverting and inverting inputs, outputting the result as a high or low level. A variable resistor is then connected to the inverting input to adjust the voltage threshold for determining the presence of water. A water accumulation sensing device is designed at the non-inverting input, consisting of two electrodes, DET_A and DET_B, placed at a certain distance, with one electrode grounded and the other connected to the non-inverting input. Simultaneously, an LED alarm is connected in parallel at the comparator output to indicate the high or low level of the output; a high level illuminates the LED, and a low level extinguishes it. Assuming a total voltage of 10V, when there is no water between the two electrodes, the resistance formed by the two electrodes (DET_A and DET_B) is assumed to be infinite. According to the voltage divider principle, the voltage at the non-inverting input approaches 10V. At this point, the voltage at the non-inverting input is greater than the voltage at the inverting input, resulting in a high level output and the LED lighting up. When there is water between the two electrode plates, it is equivalent to the presence of a medium between the two electrode plates, which is equivalent to the presence of resistance. By adjusting the sliding rheostat at the inverting terminal to set the voltage threshold, the voltage at the inverting terminal is just greater than the voltage at the non-inverting terminal. At this time, the output terminal outputs a low level, and the LED light turns off.

[0020] Since the water accumulation sensor node circuit is designed so that a high level indicates the absence of water and a low level indicates the presence of water, an SN74LVC1G04DBVR inverter is added to the circuit output to change the high and low levels for easier subsequent operation. Furthermore, because water accumulation between the two electrodes alters the circuit's structure and continuity, and the area covered by the two electrodes is too small, this invention expands the number of electrodes to 18: nine each for DET_A and DET_B, arranged in two groups of 3×3. DET_A and DET_B are placed on the electrodes with alternating intervals, and the electrodes are positioned in different locations for water accumulation detection, increasing the detection area. This circuit, integrating several modules, constitutes the water accumulation sensor node circuit designed in this invention. Regardless of where the water drips, as long as it connects any two red and blue electrodes of different polarities, a comparator circuit can be formed, thus achieving the purpose of detecting water accumulation.

[0021] 2.2 Connect the sensor to peripheral devices using the GPIO interface

[0022] The GPIO interface is used to connect peripheral devices of the sensor to the core system device. Peripheral devices using I2C and a self-developed protocol are used as IoT sensors for water accumulation sensors. The GPIO interface includes a large number of protocols, each with different interface definitions, requiring the corresponding interfaces to be brought out. The expansion board connects to the core device's GPIO interface via a 40-pin interface, bringing out interfaces such as I2C, SPI, and CAN. A protocol logic chip is also set on the expansion board, allowing the modified water accumulation sensor protocol to connect to the core device via the chip. Each expansion board can connect four sets of water accumulation sensors using the self-developed protocol, and the other brought-out interfaces provide the possibility of adding other sensors. The GPIO expansion board and the core device are placed in a rack, and other connected sensors are brought out from the rack by inserting into the corresponding interfaces on the expansion board. Therefore, each water accumulation detection node can be divided into two polarities, and each polarity can connect nine water accumulation detection nodes. Connecting water accumulation detection plates of different polarities activates the water accumulation detection node. The activated water accumulation detection node transmits a signal to the expansion board, at which point the core device receives an auxiliary judgment signal indicating the presence of water accumulation. Each detection plate can connect to 4 water accumulation detection nodes, so a system can connect to 36 sets of water accumulation detection plates. If any set detects water accumulation, the system will receive an auxiliary judgment signal indicating that there is water accumulation.

[0023] Step 3: Deploy the hardware system. The system nodes include cameras, sensors, edge computing devices, antennas, and interactive devices.

[0024] This system includes cameras, sensors, edge computing devices, antennas, and interactive devices. Deploying artificial intelligence algorithms on local edge nodes effectively reduces costs, reduces processing latency, increases stability, and allows for flexible deployment unaffected by network conditions. The interactive devices utilize large-area LED bulletin boards with strong anti-reflective capabilities to ensure drivers can clearly see the interactive content from a distance. The sensors used are the modified sensors from step two, providing cross-validation between physical sensors and artificial intelligence methods for road infrastructure and road anomaly warnings, creating redundancy and ensuring normal operation of the warning functions. The cameras are 1080P resolution cameras with infrared night vision capabilities, providing effective protection for nighttime road detection and curve warnings. The system configuration of this invention is flexible; each node can freely combine multiple edge computing devices, cameras, sensors, and interactive devices according to its scale, achieving on-demand configuration. Not only in curve warning scenarios, where two sets of cameras and two sets of edge computing devices can be assumed on both sides of the curve, and the sensors can be deployed according to actual conditions, in road infrastructure and road anomaly warning applications, nodes can be densely arranged in key road sections, or the interactive devices can be omitted, with only the detection results uploaded to higher-level decision-making agencies via antennas through a mobile network.

[0025] To achieve modularity and scalability, this invention uses an edge computing device as its core, switches and expansion boards as node switching devices, and LED screens, surveillance cameras, water accumulation sensors, and temperature and humidity sensors as peripheral devices. All three types of devices have the capability for individual replacement or upgrades, allowing for flexible deployment in complex environments. The LED screen and surveillance camera are connected to the switch, communicating bidirectionally with the edge computing device via RJ-45 interfaces. The water accumulation sensor, temperature and humidity sensor, and other IoT sensor peripherals are connected to the expansion board, providing the edge computing device with environmental awareness, enhancing the accuracy of machine learning judgments, and reducing false positives and false negatives. The switch connects to all connected devices via network cables, offering advantages such as strong anti-interference capabilities and long-distance deployment. The expansion board connects to the edge computing device via a dedicated 40-pin ribbon cable, placed next to the edge computing device, and uses a self-developed protocol and I2C protocol for connections with the sensors, both offering a certain degree of anti-interference capability.

[0026] Step 4: Integrate the software and hardware components of this invention.

[0027] The various software programs are integrated into the hardware to form an intelligent interactive scenario. The software framework is subdivided into four main parts: system detection module, Kafka module, system analysis module, and system information dissemination module. The system detection module is responsible for target detection using artificial intelligence algorithms; Kafka is the information transmission module; the system analysis module is the system's post-processing module, including post-processing of sensor information and post-processing of information dissemination; and the information dissemination module is mainly responsible for the interface between the system's main program output and the LED screen.

[0028] 4.1 System Detection Module

[0029] The system detection module is responsible for target detection using artificial intelligence algorithms. Based on the DeepStream framework and the YOLOv8 network, a pruning optimization algorithm was developed to improve the accuracy and speed of the original network framework, enabling it to be deployed on edge computing modules. To improve operational efficiency, this module was developed in C++, avoiding the efficiency issues associated with using Python in the original network architecture. Compiling it into a native application based on the aarch64 architecture further enhances its performance.

[0030] The AI ​​processing module requires pre-trained weights to run, but the original weights have a 32-bit floating-point precision, which, while highly accurate, results in a large computational burden. Although the network algorithm has been significantly optimized, its performance is still insufficient to support simultaneous processing of multiple 2K video streams. Therefore, precision pruning was performed on the network, adjusting the FP32 precision to FP16 precision. This significantly improved speed with minimal decrease in accuracy, making it possible for a scalable, high-precision network to process multiple high-resolution video streams in real time. The C++ FP16 pruning-optimized network architecture can achieve a detection speed of 128 FPS while meeting the accuracy requirements. It can process multiple 30 FPS 2K resolution video streams in real time. When a small number of video streams are connected, it can reduce the power consumption of the edge processing module while providing real-time performance and high accuracy, and extend battery life when using a battery as a power source.

[0031] 4.2 Kafka Module

[0032] To ensure system stability, this invention divides the program into several subroutines. The artificial intelligence part is executed by the system detection module, which outputs the detection results to the system analysis module via Kafka for analysis, and then outputs the analysis results. Adding subroutine modules prevents the main program from crashing and thus the system from crashing, and also makes the system more stable. In this invention, the Kafka Producer is the system detection module, and the Consumer is the system analysis module. The system detection module outputs the detection result string and transmits it to the system analysis module via Kafka. The protocol layer for communicating with Kafka is embedded in the system detection module and can directly interface with the Kafka Server. Before Kafka starts, the ZooKeeper process needs to be started. After ZooKeeper starts, it starts Kafka and initiates message passing functionality. A daemon program was written according to system requirements; if Kafka stops working, it restarts Kafka to ensure stable system operation.

[0033] 4.3 System Analysis Module

[0034] The system analysis module is a post-processing module, including sensor information post-processing and information dissemination post-processing. Sensor information post-processing includes further weighting and conditional judgment of sensor information, while information dissemination post-processing includes post-processing of message data transmitted by the system detection module. The strings transmitted by the system detection module via Kafka mainly include the results of identified objects, as well as information on ice, snow, and water accumulation; these strings are all real-time identification information. Due to limitations in screen refresh rate and output information rules, directly outputting real-time detection information would result in disorganized information, directly affecting drivers' ability to obtain effective information in curve warning scenarios. Therefore, the information needs to be cached and processed before being handed over to the information dissemination module for final display on the LED screen.

[0035] Sensor post-processing not only involves comparing and weighting the water and snow warnings output by the AI ​​network with the raw data, but also includes a watchdog program that checks for erroneous frames transmitted by the sensor or sensor non-response. If more than five erroneous or empty frames are transmitted, a reset command is issued to the sensor. Information dissemination post-processing is a comprehensive display strategy process that integrates various information. If the result string transmitted by the system detection module is valid, and after weighting it with the raw sensor data, the frequency of the result and sensor data parameters are evaluated. The result is then processed according to the display strategy and finally output as a string to the information dissemination module for display on the LED screen.

[0036] 4.4 Information Dissemination Module

[0037] The information dissemination module is primarily responsible for interfacing the system's main program output with the LED screen. The LED screen controller uses the Dahua DH-PHTOC-PT11 control card, which is compatible with the Nova traffic protocol and requires development based on this communication protocol. While the traffic protocol has a plug-and-play host computer program, it is incompatible with the Nvidia Jetson AGX Orin architecture and system core equipment used in this system, necessitating redevelopment. After porting and testing, the control card, which includes complete communication protocol source code, successfully interfacing with the core equipment of this system. The information dissemination module is also responsible for LED screen layout, cutting excessively long strings and displaying them in large font on the corresponding LED screens to remind drivers to drive carefully and slowly.

[0038] Furthermore, intelligent interactive scenarios enable comprehensive information research, judgment, and decision-making. Information comprehensive research and judgment refer to prioritizing the warning signals output by the artificial intelligence network and making comprehensive decisions based on the existing information priorities. High-priority information is displayed on the interactive LED screen, prioritizing the delivery of advanced warning information while minimizing the impact on the driver. Simultaneously, in the event of a major emergency, a high-alert warning is given to the driver, reminding the vehicle to slow down, and the emergency is reported to higher-level decision-making bodies to promptly prevent the vehicle from entering the area.

[0039] Compared with existing traffic target detection methods, this invention has the following advantages:

[0040] 1. Compared with existing traffic target detection, this invention uses edge computing devices as the core, switches and expansion boards as node switching devices, and LED screens, surveillance cameras, water accumulation sensors and temperature and humidity sensors as peripheral devices. All three types of devices have the ability to be replaced or updated individually, which can meet the flexible deployment in complex usage environments.

[0041] 2. The edge computing device of this invention provides ambient environment awareness, enhances the accuracy of machine learning judgments, and reduces false positives and false negatives. The switch and each connected device are connected via network cables, offering advantages such as strong anti-interference capabilities and long-distance deployment.

[0042] 3. This invention uses advanced target detection algorithms to reduce computation while ensuring accuracy, thereby reducing the power consumption of edge computing devices and further reducing the latency of output information;

[0043] 4. This invention innovatively combines an improved water accumulation sensor and a temperature sensor to determine the water and snow accumulation conditions, avoiding the situation where target detection cannot identify shapeless water and snow accumulation. Attached Figure Description

[0044] Figure 1 The diagram shows the proposed solution for the system of this invention.

[0045] Figure 2 A combined solution for road infrastructure early warning and road anomaly early warning nodes;

[0046] Figure 3 A simulation diagram of water accumulation detected by the water accumulation sensor node circuit;

[0047] Figure 4 This is a diagram of the overall hardware framework of the system.

[0048] Figure 5 This is a diagram of the overall system software framework.

[0049] Figure 6 This is a diagram illustrating the workflow of the Kafka module.

[0050] Figure 7 A flowchart of the system analysis module;

[0051] Figure 8 This is a flowchart illustrating the implementation of the method of the present invention; Detailed Implementation

[0052] To make the above and other objects, features and advantages of the present invention more apparent, specific embodiments of the present invention are described below in detail with reference to the accompanying drawings: Refer to Figure 1 The diagram shows the proposed solution for the system of this invention; see reference. Figure 2 A combined scheme for road infrastructure early warning and road anomaly early warning nodes; refer to Figure 3 A simulation diagram of the water accumulation sensor node circuit detecting accumulated water; refer to... Figure 4 This is a diagram of the overall system hardware framework; refer to Figure 5 This is a diagram of the overall system software framework; refer to Figure 6 This is a diagram illustrating the workflow of the Kafka module; refer to... Figure 7 This is a flowchart of the system analysis module workflow; refer to... Figure 8 This is a flowchart illustrating the implementation of the method of the present invention; the specific steps of the method of the present invention are as follows:

[0053] Step 1: A video surveillance system is set up on rural roads to collect road data. Based on the large amount of comprehensive road data collected, a curve target detection dataset RCODS suitable for the complex environment of rural roads is constructed. The performance of the dataset is verified and the accuracy and usability of the dataset constructed in this invention are demonstrated. Then, the curve target detection dataset RCODS is used to train the YOLOv8 algorithm model to achieve the purpose of quickly identifying and detecting traffic targets on curves.

[0054] 1.1 Establish a video surveillance system suitable for rural roads to collect video data for specific road sections:

[0055] Firstly, the surveillance camera serves as the primary input device for data acquisition in this invention. All identification-related tasks within the system rely on the surveillance camera as the video input device, making its clarity and resolution paramount. The surveillance camera needs to provide a clear and sharp video stream during the day, while also possessing night vision capabilities to provide black-and-white night vision images, ensuring the system can still provide normal alerts at night. Therefore, this invention selected Dahua's 4-megapixel infrared zoom network camera, model DH-IPC-HFW3433F-Z. Two surveillance cameras were respectively installed on both sides of the curve to monitor the lane and collect video data.

[0056] 1.2 Video data is read and stored frame by frame, and suitable images are selected to construct the curve target detection dataset RCODS:

[0057] Since the acquired video data is continuous, extracting images frame by frame would generate a large amount of redundant data, occupying significant storage space and greatly increasing data processing time. Therefore, this invention adopts a method of extracting one image every ten frames. However, the video data acquisition process spans a considerable period, and extracting one image every ten frames results in an excessively large amount of image information. Therefore, further data redundancy removal is necessary. Thus, this invention introduces the imagehash function during the editing of the dataset algorithm script. The imagehash function is used to filter images, removing duplicates or highly similar images to further compress the dataset, but the resulting dataset is still large. Finally, every ten or so images in the compressed dataset are randomly selected and retained. This ensures the representativeness of the selected samples and effectively avoids problems caused by biases in the selection method. After the above series of operations, this invention processes long-term video footage, and the resulting sample images serve as the dataset required for the curve target detection system.

[0058] 1.3 Data Labeling and Classification:

[0059] The generated dataset images are complex, usually containing multiple categories of information such as weather, pedestrians, trucks, cars, traffic cones, headlights, and agricultural machinery. Select the information you need to identify and label and classify the target categories of the dataset. First, you need to download and install Anaconda and change the Python version to Python 3.8.5 to configure the environment. The detailed steps are as follows: (1) Click Start in the lower left corner of the screen, find the Anaconda folder in the application bar, and open Anaconda Prompt (anaconda). (2) Enter the following commands in the Prompt to create a virtual environment named labelme and install labelme. Note that when entering the commands, you need to change the default Python version to the installed Python version. Enter python-version in the opened command line window to check the installed Python version. (3) After the above operations are completed, if you want to check whether the installation is successful, you need to enter the command condainfo--env in the command line window to view all environments. If the labelme environment name appears, it means that the installation is successful.

[0060] Then start labeling and classifying the selected target categories in the system. The specific operations are as follows: (1) Put the dataset file dataset and the label name file label name into the same folder. Assume that the dataset is placed in the path D:\project_1\Qingling. (2) Open Anaconda Prompt, enter activate labelme to activate the environment, then enter D: to enter the D drive, and then enter cd D:\project_1\Qingling to enter the path where the dataset is placed. Here, you need to change the input to your own path. (3) Enter labelme dataset--labels labels_name.txt--flagssunny,rainy,snowy. A software window will appear. If there are corresponding labels on the right, it means that the creation is successful. The three options of Flags are sunny, rainy, and snowy. Label list is the target category you have selected. (4) Click opendir, select the dataset folder, then click File, select Save Automatically, uncheck Save with image, and click changeOutput dir. You can then create an empty label folder in the same path and select it. In this way, the labeled JSON file is saved in label. The completed operation is shown in the figure. (5) Select the image in the File list at the bottom right of the software and start annotation. (6) Select the weather in the Flags column, then click Edit, and select the rectangle to start annotating objects. Drag the rectangle to annotate the target category you selected in the image. Then click the corresponding object option. If the rectangle is dragged out of control, press ESC. If the annotation is wrong, click Edit polygons on the left to modify it. Finally, click Next Image on the left to annotate the next image.

[0061] 1.4 YOLOv8 Algorithm Model Training and Validation

[0062] After the dataset is constructed and labeled, this invention will perform dataset validation based on the YOLOv8 algorithm model. Since the dataset provides the necessary information for the curve target detection system, in order to enable the system to make accurate judgments, the YOLOv8 algorithm model needs to be trained on the dataset constructed in this invention. The YOLOv8 algorithm developed by Ultralytics can provide an efficient training mode for the target detection system. During training, the input dataset needs to be divided into training and test sets in a 9:1 ratio. Next, the environment and hyperparameters need to be configured. Hyperparameters are set and modified through a ql.yaml hyperparameter configuration file, and the target detection categories are set to multiple required categories such as "car", "truck", and "person". Then, the YOLOv8 model is trained on the dataset constructed in this invention, iterating 150 times to obtain an optimal training weight for testing on the dataset.

[0063] Finally, after training the dataset to obtain the optimal training weights, we tested and verified the performance of the curve target detection system. We randomly selected other photos taken by curve video surveillance and put them into the curve target detection system to observe whether it could detect the pre-trained targets. We also derived the relationship between the confidence value, precision, and F1 score. We repeatedly set the confidence value until the curve target detection system could accurately identify the set targets and achieve high precision and F1 score.

[0064] Step 2: To address the issue that rain and snow patterns can change freely and cannot be detected using target detection functions, a modified water accumulation sensor was designed to assist in judgment. At the same time, a protocol logic chip was added to the modified water accumulation sensor, using a GPIO interface expansion board, combined with the Swiss Sensirion SHT3x-ARP temperature and humidity sensor, which improved the accuracy of road water accumulation and rain and snow detection.

[0065] 2.1 Fabrication and modification of the sensor

[0066] The most crucial element in the design of the water accumulation sensor is the comparator circuit. The circuit design first introduces an LM393 voltage comparator, which compares the voltage at the non-inverting and inverting inputs, outputting the result as a high or low level. A variable resistor is then connected to the inverting input to adjust the voltage threshold for determining the presence of water. A water accumulation sensing device is designed at the non-inverting input, consisting of two electrodes, DET_A and DET_B, placed at a certain distance, with one electrode grounded and the other connected to the non-inverting input. Simultaneously, an LED alarm is connected in parallel at the comparator output to indicate the high or low level of the output; a high level illuminates the LED, and a low level extinguishes it. Assuming a total voltage of 10V, when there is no water between the two electrodes, the resistance formed by the two electrodes (DET_A and DET_B) is assumed to be infinite. According to the voltage divider principle, the voltage at the non-inverting input approaches 10V. At this point, the voltage at the non-inverting input is greater than the voltage at the inverting input, resulting in a high level output and the LED lighting up. When there is water between the two electrode plates, it is equivalent to the presence of a medium between the two electrode plates, which is equivalent to the presence of resistance. By adjusting the sliding rheostat at the inverting terminal to set the voltage threshold, the voltage at the inverting terminal is just greater than the voltage at the non-inverting terminal. At this time, the output terminal outputs a low level, and the LED light turns off.

[0067] Since the water accumulation sensor node circuit is designed so that a high level indicates the absence of water and a low level indicates the presence of water, an SN74LVC1G04DBVR inverter is added to the circuit output to change the high and low levels for easier subsequent operation. Furthermore, because water accumulation between the two electrodes alters the circuit's structure and continuity, and the area covered by the two electrodes is too small, this invention expands the number of electrodes to 18: nine each for DET_A and DET_B, arranged in two groups of 3×3. DET_A and DET_B are placed on the electrodes with alternating intervals, and the electrodes are positioned in different locations for water accumulation detection, increasing the detection area. This circuit, integrating several modules, constitutes the water accumulation sensor node circuit designed in this invention. Regardless of where the water drips, as long as it connects any two red and blue electrodes of different polarities, a comparator circuit can be formed, thus achieving the purpose of detecting water accumulation.

[0068] 2.2 Connect the sensor to peripheral devices using the GPIO interface

[0069] The GPIO interface is used to connect peripheral devices of the sensor to the core system device. Peripheral devices using I2C and a self-developed protocol are IoT sensors for water accumulation sensors. The GPIO interface includes a large number of protocols, each with different interface definitions, requiring the corresponding interfaces to be brought out. The expansion board connects to the core device's GPIO interface via a 40-pin interface, bringing out interfaces such as I2C, SPI, and CAN. A protocol logic chip is also set on the expansion board, allowing the modified water accumulation sensor protocol to connect to the core device via the chip. Each expansion board can connect four sets of water accumulation sensors using the self-developed protocol, and the other brought-out interfaces provide the possibility of adding other sensors. The GPIO expansion board and the core device are placed in a rack, and other connected sensors are brought out from the rack by inserting into the corresponding interfaces on the expansion board. Therefore, each water accumulation detection node can be divided into two polarities, and each polarity can connect nine water accumulation detection nodes. Connecting water accumulation detection plates of different polarities activates the water accumulation detection node. The activated water accumulation detection node transmits a signal to the expansion board, at which point the core device receives an auxiliary judgment signal indicating the presence of water accumulation. Each detection plate can connect to 4 water accumulation detection nodes, so a system can connect to 36 sets of water accumulation detection plates. If any set detects water accumulation, the system will receive an auxiliary judgment signal indicating that there is water accumulation.

[0070] Step 3: Deploy the hardware system. The system nodes include cameras, sensors, edge computing devices, antennas, and interactive devices.

[0071] This system includes cameras, sensors, edge computing devices, antennas, and interactive devices. Deploying artificial intelligence algorithms on local edge nodes effectively reduces costs, reduces processing latency, increases stability, and allows for flexible deployment unaffected by network conditions. The interactive devices utilize large-area LED bulletin boards with strong anti-reflective capabilities to ensure drivers can clearly see the interactive content from a distance. The sensors used are the modified sensors from step two, providing cross-validation between physical sensors and artificial intelligence methods for road infrastructure and road anomaly warnings, creating redundancy and ensuring normal operation of the warning functions. The cameras are 1080P resolution cameras with infrared night vision capabilities, providing effective protection for nighttime road detection and curve warnings. The system configuration of this invention is flexible; each node can freely combine multiple edge computing devices, cameras, sensors, and interactive devices according to its scale, achieving on-demand configuration. Not only in curve warning scenarios, where two sets of cameras and two sets of edge computing devices can be assumed on both sides of the curve, and the sensors can be deployed according to actual conditions, in road infrastructure and road anomaly warning applications, nodes can be densely arranged in key road sections, or the interactive devices can be omitted, with only the detection results uploaded to higher-level decision-making agencies via antennas through a mobile network.

[0072] To achieve modularity and scalability, this invention uses an edge computing device as its core, switches and expansion boards as node switching devices, and LED screens, surveillance cameras, water accumulation sensors, and temperature and humidity sensors as peripheral devices. All three types of devices have the capability for individual replacement or upgrades, allowing for flexible deployment in complex environments. The LED screen and surveillance camera are connected to the switch, communicating bidirectionally with the edge computing device via RJ-45 interfaces. The water accumulation sensor, temperature and humidity sensor, and other IoT sensor peripheral devices are connected to the expansion board, providing the edge computing device with environmental awareness, enhancing the accuracy of machine learning judgments, and reducing false positives and false negatives. The switch connects to all connected devices via network cables, offering advantages such as strong anti-interference capabilities and long-distance deployment. The expansion board connects to the edge computing device via a dedicated 40-pin ribbon cable, placed next to the edge computing device, and uses a self-developed protocol and I2C protocol for connections with the sensors, both offering a certain degree of anti-interference capability.

[0073] Step 4: Integrate the software and hardware components of this invention.

[0074] The various software programs are integrated into the hardware. The software framework is subdivided into four main parts: system detection module, Kafka module, system analysis module, and system information dissemination module. The system detection module is responsible for target detection using artificial intelligence algorithms. Kafka is the information transmission module. The system analysis module is the system's post-processing module, including post-processing of sensor information and post-processing of information dissemination. The information dissemination module is mainly responsible for the interface between the system's main program output and the LED screen.

[0075] 4.1 System Detection Module

[0076] The system detection module is responsible for target detection using artificial intelligence algorithms. Based on the DeepStream framework and the YOLOv8 network, a pruning optimization algorithm was developed to improve the accuracy and speed of the original network framework, enabling it to be deployed on edge computing modules. To improve operational efficiency, this module was developed in C++, avoiding the efficiency issues associated with using Python in the original network architecture. Compiling it into a native application based on the aarch64 architecture further enhances its performance.

[0077] The AI ​​processing module requires pre-trained weights to run, but the original weights have a 32-bit floating-point precision, which, while highly accurate, results in a large computational burden. Although the network algorithm has been significantly optimized, its performance is still insufficient to support simultaneous processing of multiple 2K video streams. Therefore, precision pruning was performed on the network, adjusting the FP32 precision to FP16 precision. This significantly improved speed with minimal decrease in accuracy, making it possible for a scalable, high-precision network to process multiple high-resolution video streams in real time. The C++ FP16 pruning-optimized network architecture can achieve a detection speed of 128 FPS while meeting the accuracy requirements. It can process multiple 30 FPS 2K resolution video streams in real time. When a small number of video streams are connected, it can reduce the power consumption of the edge processing module while providing real-time performance and high accuracy, and extend battery life when using a battery as a power source.

[0078] 4.2 Kafka Module

[0079] To ensure system stability, this invention divides the program into several subroutines. The artificial intelligence part is executed by the system detection module, which outputs the detection results to the system analysis module via Kafka for analysis, and then outputs the analysis results. Adding subroutine modules prevents the main program from crashing and thus the system from crashing, and also makes the system more stable. In this invention, the Kafka Producer is the system detection module, and the Consumer is the system analysis module. The system detection module outputs the detection result string and transmits it to the system analysis module via Kafka. The protocol layer for communicating with Kafka is embedded in the system detection module and can directly interface with the Kafka Server. Before Kafka starts, the ZooKeeper process needs to be started. After ZooKeeper starts, it starts Kafka and initiates message passing functionality. A daemon program was written according to system requirements; if Kafka stops working, it restarts Kafka to ensure stable system operation.

[0080] 4.3 System Analysis Module

[0081] The system analysis module is a post-processing module, including sensor information post-processing and information dissemination post-processing. Sensor information post-processing includes further weighting and conditional judgment of sensor information, while information dissemination post-processing includes post-processing of message data transmitted by the system detection module. The strings transmitted by the system detection module via Kafka mainly include the results of identified objects, as well as information on ice, snow, and water accumulation; these strings are all real-time identification information. Due to limitations in screen refresh rate and output information rules, directly outputting real-time detection information would result in disorganized information, directly affecting drivers' ability to obtain effective information in curve warning scenarios. Therefore, the information needs to be cached and processed before being handed over to the information dissemination module for final display on the LED screen.

[0082] Sensor post-processing not only involves comparing and weighting the water and snow warnings output by the AI ​​network with the raw data, but also includes a watchdog program to detect sensor transmission errors or sensor non-response. If more than five error or empty frames are transmitted, a reset command is issued to the sensor. Information dissemination post-processing is a comprehensive display strategy process that integrates various information. If the result string transmitted by the system detection module is valid, it is weighted with the raw sensor data, and the frequency of the result and sensor data parameters are determined. The result is then processed according to the display strategy and finally output as a string to the information dissemination module for display on the LED screen.

[0083] 4.4 Information Dissemination Module

[0084] The information dissemination module is primarily responsible for interfacing the system's main program output with the LED screen. The LED screen controller uses the Dahua DH-PHTOC-PT11 control card, which is compatible with the Nova traffic protocol and requires development based on this communication protocol. While the traffic protocol has a plug-and-play host computer program, it is incompatible with the Nvidia Jetson AGX Orin architecture and system core equipment used in this system, necessitating redevelopment. After porting and testing, the control card, which includes complete communication protocol source code, successfully interfacing with the core equipment of this system. The information dissemination module is also responsible for LED screen layout, cutting excessively long strings and displaying them in large font on the corresponding LED screens to remind drivers to drive carefully and slowly.

[0085] Furthermore, intelligent interactive scenarios enable comprehensive information research, judgment, and decision-making. Information comprehensive research and judgment refer to prioritizing the warning signals output by the artificial intelligence network and making comprehensive decisions based on the existing information priorities. High-priority information is displayed on the interactive LED screen, prioritizing the delivery of advanced warning information while minimizing the impact on the driver. Simultaneously, in the event of a major emergency, a high-alert warning is given to the driver, reminding the vehicle to slow down, and the emergency is reported to higher-level decision-making bodies to promptly prevent the vehicle from entering the area.

Claims

1. A non-uniform array design and direction of arrival estimation method, comprising the following steps: Step one: rural road video monitoring system is used to collect road data, and a curve target detection data set RCODS suitable for the complex environment of rural roads is constructed on the basis of a large amount of collected road comprehensive data, the performance of the data set is verified, and the accuracy and availability of the constructed data set are proved, then the YOLOv8 algorithm model is trained using the curve target detection data set RCODS, and the purpose of quickly identifying the curve traffic target detection is achieved; (1a) Build a video monitoring system suitable for rural roads to collect video data of a specific section of road: First, the monitoring camera is the main input device for data collection, and all recognition-related tasks of the system are video input devices of the monitoring camera, so the definition and resolution of the device are particularly important. The monitoring camera needs to provide clear and sharp video stream input during the day, and needs to have night vision capability to provide black and white night vision pictures for the system at night, so that the system can still provide normal early warning at night. Therefore, Dahua's 4 million infrared zoom gun network camera, model DH-IPC-HFW3433F-Z, is selected. Two monitoring cameras are set up on both sides of the curve to monitor the lane and collect video data; (1b) Video data is read and stored frame by frame, and suitable pictures are selected to construct the curve target detection data set RCODS: Since continuous video data is collected, extracting pictures frame by frame will generate a large amount of redundant data, occupying a large amount of storage space and also greatly increasing the data processing time. Therefore, we use the method of taking one picture every ten frames. However, during the video data collection process, the time span is relatively large, and the amount of information generated by taking one picture every ten frames is also too large, so further data redundancy reduction operation is needed. Therefore, we introduced the imagehash function in the process of editing the data set algorithm script, which uses the imagehash function to filter pictures, and removes repeated or similar images to further compress the data set. However, the data set obtained is still large, and finally one picture is randomly selected every few pictures in the compressed data set to keep, which can effectively avoid the problem caused by the bias of the selection method. After a series of operations on the video taken for a long time, the sample images obtained are used as the data set required by the curve target detection system; (1c) Data label annotation and classification: The generated data set image is complex and contains weather, pedestrians, trucks, cars, traffic cones, car headlights, and agricultural machinery. Select the information you need to identify and label and classify the target categories of the data set. First, download and install Anaconda and replace the python version with python3.8.

5. Perform environment configuration. The detailed steps are as follows: (1) Click the start button at the bottom left of the screen, find the Anaconda folder in the application bar, and open AnacondaPrompt (anaconda); (2) Enter the following commands in Prompt to create a virtual environment named labelme and install labelme. Note that you need to change the default python version to the installed python version when entering the command; enter python-version in the opened command line window to view the installed python version; (3) After the above operation is completed, if you want to check whether the installation is successful, you need to enter the command conda info --env in the command line window to view all environments. The appearance of the labelme environment name indicates that the installation is successful. Then start in the system to select the target category annotation and classification, the specific operation as follows: (1) put the dataset file dataset and label name file labelname in the same folder; assume that the data set is put into D: \project_1\Qingling this path; (2) open Anaconda Prompt, input activate labelme to activate the environment, and then input D: enter D disk, and then input cd D: \project_1\Qingling enter the path where the data set is placed, here input to change to your own path; (3) input labelme dataset--labels labels_name.txt-flags sunny,rainy,snowy, a software window will appear, and the corresponding label on the right side represents the successful creation; The three options of Flags are sunny, rainy and snowy, and Label list is the target category selected by yourself; (4) click opendir, select dataset folder, click File, select Save Automatically, deselect Save with image, click change Output dir, and a new label empty folder will be created in the same path; The JSON file after labeling is saved in label, (5) select the picture in the lower right corner of the software File list, and start labeling; (6) select the weather in the Flags column, then click Edit, and select the rectangular box to start labeling the object; Drag the rectangular box to label the target category selected in the picture; Then select the corresponding object option; If the rectangular box is dragged with deviation, press ESC to modify the labeling error; Finally, press the left Next Image to label the next picture; (1d)YOLOv8 algorithm model training and verification After the data set is constructed and labeled, the data set based on YOLOv8 algorithm model will be verified; Since the data set provides the necessary information required in the curve target detection system, in order to make accurate judgments with the system, it is necessary to train the YOLOv8 algorithm model on the constructed data set, and the YOLOv8 algorithm developed by Ultralytics company can provide efficient training mode for the target detection system; During training, the input data set needs to be divided into training set and test set according to the mode of 9:1; Then, the environment and hyperparameters need to be configured, and a ql.yaml hyperparameter configuration file is set to set and modify the hyperparameters, and the target detection categories are set as "car", "truck", "person" and other required categories; Then train the YOLOv8 model on the constructed data set, iterate 150 times, and get a best training weight for data set test; Then in the data set training completed to obtain the best training weight, the performance of the curved target detection system is tested and verified, randomly selected other photos of curved video monitoring, put into the curved target detection system, observe whether the target can be detected in advance training, and export the confidence value and the relationship between the accuracy and F1 score, repeat the confidence value setting, until the curved target detection system can accurately identify the set target; Step two: for the rain and snow shape transformation freedom, cannot use the target detection function to achieve, design an improved water sensor to assist in judgment, at the same time in order to improve the water sensor increases the protocol logic chip, using the expansion board of GPIO interface, combined with the temperature and humidity sensor SHT3x-ARP of Switzerland Sensirion company, improve the accuracy of road water and snow detection; (2a) making improved sensor In the design of water sensor, the most core is the design of comparator circuit. In the invention circuit design, first introduce a LM393 voltage comparator, which can compare the voltage of same phase terminal and opposite phase terminal, and output the result in the form of high and low level. Then connect a sliding rheostat to the opposite phase terminal to adjust the voltage threshold of water judgment. Design a water sensing device in the same phase input terminal, which is composed of two electrode pieces DET_A and DET_B with opposite polarity and a certain distance. One electrode piece is grounded and the other is connected to the same phase input terminal. At the same time, connect an LED alarm device to the output terminal of the comparator to judge the high and low level of output. High level is bright and low level is off. Assuming the total voltage is 10V, when there is no water between the two electrode pieces, the resistance composed of DET_A and DET_B is infinite. According to the circuit voltage division principle, the voltage divided by the same phase terminal is close to 10V. At this time, the voltage of same phase terminal is greater than that of opposite phase terminal, and the output terminal outputs high level, and the LED lamp is bright. When there is water between the two electrode pieces, it is equivalent to that there is medium between the two electrode pieces, which is equivalent to that there is resistance. By adjusting the sliding rheostat of opposite phase terminal to set the voltage threshold, the voltage of opposite phase terminal is just greater than that of same phase terminal. At this time, the output terminal outputs low level, and the LED lamp is off. As the water sensor node circuit design is high level detection is no water, low level detection is the case of water, in order to facilitate the subsequent work, in the output of the circuit added a SN74LVC1G04DBVR inverter for changing the high and low level of the circuit output; In addition, in the design of the circuit, if there is water between the two plates, it will change its organization, change its on-off, but the range of two plates is too small, so the two plates are expanded to 18, DET_A and DET_B are 9 each, which are placed in two groups according to 3x3, DET_A and DET_B are placed alternately on the plate, and the plate is placed in different positions for water detection, which increases the detection area; Such a circuit integrated by several modules is the designed water sensor node circuit; When water drops anywhere, as long as it can connect any two different polarity red and blue plates, it can form a comparator circuit, and then achieve the purpose of detecting water; (2b) using GPIO interface for connecting sensor peripheral devices and system core devices The GPIO interface is used for connecting the sensor peripheral devices and the system core devices. The sensor peripheral devices are water sensors and temperature and humidity sensors using I2C and self-developed protocols. The GPIO interface includes a large number of protocols, and the interface definition of each protocol is different. The protocol corresponding interface needs to be introduced. The expansion board is connected with the core device GPIO interface through the 40PIN interface, and the I2C, SPI, CAN interfaces are introduced. Protocol logic chips are set on the expansion board, so that the improved water sensor protocol can be connected with the core device through the chip. Each expansion board can connect four groups of self-developed protocol water sensors. The other interfaces introduced provide the possibility of adding other sensors. The GPIO expansion board and the core device are placed in the cabinet, and the other sensors connected are introduced from the cabinet through the corresponding interface of the expansion board. Each water detection node can be divided into two polarities, and each polarity can connect nine water detection nodes. The water detection nodes of different polarities are connected to activate the water detection nodes. The activated water detection nodes transmit signals to the expansion board. At this time, the core device receives the auxiliary judgment signal of water. Each detection board can connect four water detection nodes, so a set of system can connect 36 groups of water detection plates. Any group that detects water will receive the auxiliary judgment signal of water. Step three: deploy the hardware system. The system node includes camera, sensor, edge computing device, antenna and interactive device: The system node includes camera, sensor, edge computing device, antenna and interactive device. The artificial intelligence algorithm is deployed in the local edge node, and the interactive device uses LED bulletin board to ensure that the driver can clearly see the interactive content from a distance. The sensor uses the self-developed sensor of step two, and the physical sensor and artificial intelligence method are cross-verified for road infrastructure warning and road anomaly warning, and redundancy is formed to ensure the normal operation of the warning function; the camera uses a 1080P resolution camera with infrared night vision function to provide effective protection for night road detection and curve warning; In order to realize the modularity and scalability of the system, the system takes the edge computing device as the core, the switch and the expansion board as the node switching device, the LED screen, the monitoring camera, the water sensor and the temperature and humidity sensor as the peripheral device, and the three devices have the ability of separate replacement or updating, which meets the flexible deployment in complex use environment; The LED screen, the monitoring camera and the switch are connected, the switch communicates with the edge computing device through the RJ-45 interface in both directions; the water sensor and the temperature and humidity sensor are connected with the expansion board to provide the edge computing device with the peripheral environment sensing ability, to strengthen the accuracy of machine learning judgment and to reduce misjudgment and omission; the switch and each connected device are connected through a network cable, the expansion board and the edge computing device are connected through a special 40PIN cable, which is placed beside the edge computing device, and the connection between the sensor and the expansion board uses self-developed protocol and I2C protocol; Step four: integrate the software part and the hardware part: The software framework is divided into four parts: system detection module, Kafka module, system analysis module and system information dissemination module. The system detection module is responsible for artificial intelligence algorithm target detection, Kafka is the information transmission module, the system analysis module is the post-processing module of the system, including sensor information post-processing and information release post-processing, and the information dissemination module is responsible for the output of the system main program and the docking of the LED screen; (4a) System detection module The system detection module is responsible for artificial intelligence algorithm target detection, and a pruning optimization algorithm is developed based on the YOLOV8 network using DeepStream as the framework, which optimizes the accuracy and speed of the original network framework, so that it can be deployed in the edge computing module; In order to improve the running efficiency, this module is developed using C++ language, which avoids the running efficiency problem caused by using python in the original network architecture, and the running efficiency is also improved after compiling it into aarch64 architecture native application; The artificial intelligence processing module needs to load the trained weight to run, and the network has been pruned for precision, adjusting the FP32 precision to FP16 precision. The network architecture optimized by C++ FPl6 pruning can achieve a detection speed of 128FPS, while the precision meets the requirements, and can process multiple 30FPS 2K resolution video streams in real time; (4b) Kafka module In order to ensure the stable operation of the system, the program is divided into several sub-programs, the artificial intelligence part is executed by the system detection module, the system detection module outputs the detection results to the system analysis module through Kafka for analysis, and then the analysis results are output, the addition of the sub-program module can avoid the crash of the main program and the crash of the system, and also make the system more stable; The producer of Kafka is the system detection module, and the consumer is the system analysis module. The system detection module outputs the detection result string, which is transmitted to the system analysis module through Kafka; The protocol layer for communication with Kafka has been embedded in the system detection module, which can be directly connected with Kafka-Server; Before starting Kafka, the pre-process ZooKeeper needs to be started, and after starting ZooKeeper, Kafka is pulled up to start the message transmission function. According to the system requirements, a daemon program is written, if Kafka stops working, Kafka will be restarted to ensure the stable operation of the system; (4c) System analysis module The system analysis module is a post-processing module, including sensor information post-processing and information publishing post-processing; Sensor information post-processing includes further weighting processing and condition judgment processing of sensor information, and information publishing post-processing includes post-processing of message data transmitted by the system detection module; The string transmitted by the system detection module through Kafka includes the recognized object result information and the ice and snow and water identification information. These strings are all real-time identification information; Limited by the screen refresh rate and the output information rule, directly outputting real-time detection information will lead to disorganized information, which will directly affect the driver in the curve warning scene and also cannot obtain effective information, therefore, the information needs to be cached and processed, and only the processed information can be finally realized on the LED screen by the information dissemination module; Sensor post-processing not only includes comparing and weighting the water and ice warning and original data output by the artificial intelligence network, but also includes a watchdog program to judge the transmission error frame or non-response of the sensor. If the transmission error frame or empty frame is greater than 5 times, a reset command will be sent to the sensor; Information publishing post-processing is a display strategy processing process of various information. If the result string transmitted by the system detection module is legal and weighted with the sensor original data, the result frequency and sensor data parameters in the judgment result are processed according to the display strategy, and finally output to the information dissemination module in the form of a string for display on the LED screen; (4d) Information dissemination module The information dissemination module is mainly responsible for the docking of the system main program output and the LED screen. The LED screen controller uses Dahua DH-PHTOC-PT11 control card, compatible with Nova traffic protocol, which needs to be developed on the basis of this communication protocol. The traffic protocol has a plug-and-play host computer program, but it is not compatible with the core device Nvidia Jetson AGX Orin architecture and the system, which needs to be redeveloped. After transplantation and testing of the complete communication protocol source code attached to the control card, the control protocol is successfully docked with the core device used in the system. The information dissemination module is also responsible for the layout of the LED screen, cutting long strings to display on the corresponding LED screen in large font, reminding the drivers of passing vehicles to drive carefully and pass slowly. And the intelligent interaction scene can realize information comprehensive research, judgment and decision research, information comprehensive research and judgment and decision research refers to the priority sorting of the early warning signals output by the artificial intelligence network, and the comprehensive decision-making using the existing information priority. High-priority information is displayed through the interactive device LED screen, and the impact on the driver is minimized while delivering high-level early warning information. At the same time, when a major danger occurs, the driver is given a high alert reminder to slow down the vehicle, and the danger is reported to the superior decision-making agency in time to block the vehicle from entering.

Images