YOLOv8-based curve target detection system

Abstract

The method comprises the following steps: firstly, building a video monitoring system on a rural road, constructing a curve target detection data set, and training a YOLOv8 algorithm model by using the curve target detection data set to achieve the purpose of rapidly identifying curve traffic target detection; aiming at free conversion of rain and snow forms, a modified ponding sensor is used for assisting judgment, an expansion board special for a GPIO interface is developed for the sensor, and the accuracy of road ponding and rain and snow detection is improved in combination with a temperature and humidity sensor; an artificial intelligence algorithm is deployed at a local edge node, so that the cost can be effectively reduced, the processing delay is small, the stability is high, the arrangement is flexible, and the influence of a line is avoided; a software part and a software framework are subdivided into four parts, comprehensive information research, judgment and decision-making research functions of an intelligent interaction scene are realized step by step, the influence on a driver can be reduced to the minimum while advanced early warning information is transmitted preferentially, a dangerous case is reported to a superior decision-making mechanism, and a vehicle is stopped from entering in time.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation systems, and in particular relates to a system for real-time detection of curved road conditions and a method for early warning of passing vehicles and pedestrians. The YOLOv8 algorithm is used to monitor the road surface in real time, and a water accumulation sensor and a temperature sensor are used to determine whether there is water accumulation and ice and snow on the traffic road. The screen displays the monitoring status of the curve to remind passing pedestrians and drivers. Background Art

[0002] Transportation is the infrastructure and important lifeline of economic and social development, as well as an important yardstick of a region's level of civilization and a concentrated display of its external image. Rural roads have the widest coverage, serve the largest number of people, provide the most universal services, and have a strong public welfare nature. They are the basic conditions for ensuring the production and life of rural residents, and are also the leading and basic facilities for agricultural development, and an important basis for the continuous increase in rural residents' income.

[0003] In recent years, my country's comprehensive transportation system has played an irreplaceable role. However, there are still some outstanding problems in such a transportation system: mountain roads are steep and winding, rolling stones and vehicle spills on the road surface are common, and some low-lying sections are prone to water accumulation due to the influence of terrain conditions, or due to the influence of wind, there will be serious ice and snow on the road, which seriously threatens the safety of public transportation. There is heavy fog in the mountains, and the safety warning system relying on convex mirrors and radar speed guns cannot meet the actual needs. It is urgent to improve the function of the digital curve safety warning system for rural roads. The curve target detection in the complex environment of rural roads is different from the general target detection task scenario. It is necessary to identify and detect multiple targets, such as curves, falling rocks, snow, water, road obstacles, and cracks. Most of the currently disclosed data sets are for the identification of a specific object target, which is not completely applicable to the road scene proposed in this invention. The data set research based on the curve target detection in the complex environment of rural roads is still relatively scarce.

[0004] On the other hand, in the curve target detection system, the data set acquisition equipment needs to be collected at the same height as the surveillance camera, while most of the existing data sets are taken by on-board cameras at the same height as the vehicle. The angle of view, target size and sample characteristics collected by the two are significantly different. If the existing public road target detection data set is used for detection, it will directly fail. Therefore, based on the current situation of the lack of data sets for curve target detection in the complex environment of rural roads, constructing a data set that meets the curve target detection has significant guiding significance and reference value for accelerating the development of rural intelligent transportation and rural road construction. There is an urgent need to improve the function of highway abnormal situation warning. The present invention studies and utilizes video data and intelligent perception data of Internet of Things sensors. Through the information technology of data fusion, intelligent perception and artificial intelligence technology, it monitors, identifies and warns targets such as curve vehicles, road obstacles, and ice and snow on the road surface, and releases information through multiple channels of scrolling and vehicle-road collaboration to achieve timely warning, reporting and control, and comprehensively improve the quality of safety services. Summary of the invention

[0005] To achieve the above objectives, the technical idea of ​​the present invention is as follows: a detailed analysis and classification of various safety hazards and emergencies existing in the complex environment of rural roads is carried out. Relying on rural roads, a video surveillance system is first built on the rural roads, and a curve target detection dataset (RCODS) suitable for the complex environment of rural roads is constructed on the basis of a large amount of collected comprehensive road data. The accuracy and availability of the dataset constructed by the present invention are verified through performance verification, and then the curve target detection dataset (RCODS) is used to train the YOLOv8 algorithm model to achieve the purpose of rapid identification of curve traffic target detection; in view of the free transformation of rain and snow forms, the target detection function cannot be used, so a modified water accumulation sensor is used to assist in judgment, and at the same time, a protocol logic chip is added to the modified water accumulation sensor, an expansion board dedicated to the GPIO interface is developed, and combined with the temperature and humidity sensor SHT3x-ARP of the Swiss Sensirion company, the The accuracy of road water accumulation and rain and snow detection is improved; for hardware deployment, in order to avoid the problem that in traditional artificial intelligence solutions, the function of software upgrade to detect new types of defects requires the artificial intelligence algorithm to be placed in the data center, the present invention deploys the artificial intelligence algorithm on the local edge node. The system node includes cameras, sensors, edge computing devices, antennas and interactive devices, which can effectively reduce costs. At the same time, the processing delay is small, the stability is high, and the layout can be flexibly realized without being affected by the line; for the software part of the present invention, the software framework is subdivided into four parts: system detection module, Kafka module, system analysis module and system information dissemination module. The system detection module is responsible for artificial intelligence algorithm target detection, Kafka is an information transmission module, and the system analysis module is the post-processing module of the system, including sensor information post-processing and information release post-processing. The information dissemination module is mainly responsible for the docking of the system main program output with the LED screen, thereby integrating the software and hardware parts of the present invention.

[0006] Step 1: A video surveillance system is built on rural roads to collect road data. Based on the large amount of comprehensive road data collected, a curve target detection dataset RCODS suitable for the complex environment of rural roads is constructed. The performance of the dataset is verified and the accuracy and usability of the dataset constructed by the present invention are proved. Subsequently, the curve target detection dataset RCODS is used to train the YOLOv8 algorithm model to achieve the purpose of rapid identification of curve traffic target detection.

[0007] 1.1 Build a video surveillance system suitable for rural roads and collect video data of specific sections:

[0008] First of all, the surveillance camera is the main input device for data collection in the present invention. All recognition-related tasks of the system are performed by the surveillance camera as the video input device, so the clarity and resolution of the device are particularly important. The surveillance camera needs to provide clear and sharp video stream input during the day, and at the same time needs to have night vision capabilities to provide the system with black and white night vision images at night, so that the system can still be normally warned at night. Therefore, the present invention selects Dahua's 4 million infrared zoom gun-type network camera, model DH-IPC-HFW3433F-Z. Two surveillance cameras are set on both sides of the curve to monitor the lane and collect video data.

[0009] 1.2 Video data is read and stored frame by frame, and appropriate images are selected to build the curve target detection dataset RCODS:

[0010] Since continuous video data is collected, extracting pictures frame by frame will generate a large amount of redundant data, which will greatly increase the time of data processing while occupying a large amount of storage space. Therefore, the present invention adopts the method of taking a picture every ten frames. However, in the process of video data collection, the time span is relatively large, and the amount of picture information generated by taking a picture every ten frames is also too large. Therefore, further data redundancy removal operations are required. Therefore, the present invention introduces the imagehash function in the process of editing the data set algorithm script. The imagehash function is used to screen the pictures, and the repeated or highly similar images are eliminated to further compress the data set, but the obtained data set is still huge. Finally, in the compressed data set, a picture is randomly selected every ten pictures to be retained, which ensures that the selected sample is representative, and can also effectively avoid the problems caused by the deviation of the selection method. After the above series of operations, the present invention processes the video shot for a long time, and the sample image finally obtained is used as the data set required by the curve target detection system.

[0011] 1.3 Data labeling and classification:

[0012] The generated dataset images are complex and usually contain multiple categories of information such as weather, pedestrians, trucks, cars, traffic cones, headlights, and agricultural machinery. Select the information you need to identify and label and classify the target categories of the dataset. First, you need to download and install Anaconda, and change the Python version to Python 3.8.5 to configure the environment. The detailed steps are as follows: (1) Click Start in the lower left corner of the screen, find the Anaconda folder in the application bar, and open Anaconda Prompt (anaconda). (2) Enter the following commands in the prompt to create a virtual environment named labelme and install labelme. Note that when entering the command, you need to change the default Python version to the installed Python version. Enter python-version in the command line window to view the installed Python version. (3) After the above operation is completed, if you want to check whether the installation is successful, you need to enter the command condainfo --env in the command line window to view all environments. If the labelme environment name appears, it means that the installation is successful.

[0013] Then start to label and classify the selected target category in the system. The specific operations are as follows: (1) Put the dataset file dataset and the label name file label name into the same folder. Assume that the present invention puts the dataset in the path D:\project_1\Qingling. (2) Open Anaconda Prompt, enter activate labelme to activate the environment, then enter D: to enter the D drive, and then enter cd D:\project_1\Qingling to enter the path where the dataset is placed. The input here should be changed to your own path. (3) Enter labelme dataset--labels labels_name.txt--flagssunny,rainy,snowy, a software window will appear, and the corresponding label on the right means that the creation is successful. The three options of Flags are sunny, rainy, and snowy, and the Label list is the target category selected by yourself. (4) Click opendir, select the dataset folder, then click File, select Save Automatically, uncheck Save with image, and click changeOutput dir to create a new label empty folder in the same path and select it. In this way, the labeled JSON file is saved in label. The completed state is shown in the figure. (5) Select the image in the File list in the lower right corner of the software and start annotating. (6) Select Weather in the Flags column, then click Edit, and select the rectangular box to start annotating the object. Drag the rectangular box to annotate the target category you selected in the image. Then click the option corresponding to the object. If the rectangular box deviates when dragging, press ESC. If the annotation is wrong, press Edit polygons on the left to modify it. Finally, press Next Image on the left to annotate the next image.

[0014] 1.4YOLOv8 algorithm model training and verification

[0015] After the data set is constructed and annotated, the present invention will perform data set verification based on the YOLOv8 algorithm model. Since the data set provides the necessary information required in the curve target detection system, in order to cooperate with the system to make accurate judgments, the YOLOv8 algorithm model needs to be trained on the data set constructed by the present invention, and the YOLOv8 algorithm developed by Ultralytics can provide an efficient training mode for the target detection system. During the training process, the input data set needs to be divided into a training set and a test set according to a 9:1 mode. Next, the environment and hyperparameters need to be configured, and the hyperparameters are set and modified by setting a q1.yaml hyperparameter configuration file, and the target detection category is set to "car", "truck", "pedestrian (person)" and multiple required categories. Then the YOLOv8 model is trained on the data set constructed by the present invention, iterated 150 rounds, and an optimal training weight is obtained for the test of the data set.

[0016] Finally, after the training of the dataset was completed and the optimal training weights were obtained, we tested and verified the performance of the curved object detection system. We randomly selected other photos taken by the curved video surveillance and put them into the curved object detection system to observe whether the pre-trained targets can be detected, and derived the relationship between the confidence value, precision rate and F1 score. We repeatedly set the confidence value until the curved object detection system can accurately identify the set target with high precision rate and F1 score.

[0017] Step 2: As the rain and snow can change freely and cannot be detected using the target detection function, a modified water accumulation sensor is designed to assist in judgment. At the same time, a protocol logic chip is added to the modified water accumulation sensor, and an expansion board with a GPIO interface is used. Combined with the temperature and humidity sensor SHT3x-ARP from the Swiss company Sensirion, the accuracy of road water accumulation and rain and snow detection is improved.

[0018] 2.1 Making a modified sensor

[0019] In the design of water accumulation sensor, the core is the design of comparator circuit. In the invention circuit design, a LM393 voltage comparator is first introduced, which can compare the voltage of the same phase terminal with the voltage of the opposite phase terminal, and output the result of the comparison in the form of high and low levels. Then a sliding rheostat is connected to the opposite phase terminal to adjust the voltage threshold for judging whether there is water accumulation. A water accumulation sensing device is designed at the same phase input terminal. It is composed of two electrodes DET_A and DET_B with opposite polarities placed at a certain distance, one electrode is grounded, and the other is connected to the same phase input terminal. At the same time, an LED alarm device is connected to the output terminal of the comparator to judge the high and low levels of the output. The high level is on and the low level is off. Assuming the total voltage is 10V, when there is no water between the two electrodes, the resistance composed of the two electrodes DET_A and DET_B is infinite by default. According to the circuit voltage division principle, the voltage divided by the same phase terminal approaches 10V. At this time, the voltage of the same phase terminal is greater than the voltage of the opposite phase terminal, the output terminal outputs a high level, and the LED light is on. When there is water between the two electrodes, it is equivalent to the presence of a medium between the two electrodes, which is equivalent to the presence of resistance. The voltage threshold is set by adjusting the sliding rheostat at the inverting end so that the voltage at the inverting end is just greater than the voltage at the inverting end. At this time, the output end outputs a low level and the LED light goes out.

[0020] Since the design of the water accumulation sensor node circuit is that the high level detects the absence of water, and the low level detects the presence of water accumulation, in order to facilitate the subsequent work, the present invention adds a SN74LVC1G04DBVR inverter at the output end of the circuit to change the high and low levels of the circuit output. In addition, in the design of the circuit, if there is water accumulation between the two plates, its organization will be changed and its on and off will be changed. However, the range of the two plates is too small, so the present invention expands the two plates to 18, DET_A and DET_B are divided into two groups of 9 each, and arranged according to 3×3, DET_A and DET_B are placed on the plates in pairs and spaced apart, and the plates are placed in different positions for water accumulation detection, which increases the detection area. Such a circuit integrated with several modules is the water accumulation sensor node circuit designed by the present invention. No matter where the water drips, as long as any two red and blue pole pieces of different polarities can be connected, a comparator circuit can be formed, thereby achieving the purpose of detecting water accumulation.

[0021] 2.2 Use the GPIO interface to connect the sensor to peripheral devices

[0022] The GPIO interface is used to connect the sensor's peripheral devices to the system's core devices. The peripheral devices that use I2C and self-developed protocols are IoT sensors for water accumulation sensors. The GPIO interface contains a large number of protocols, and the interface definitions of each protocol are different, so the corresponding interfaces of the protocols need to be brought out. The expansion board is connected to the GPIO interface of the core device through a 40PIN interface, and interfaces such as I2C, SPI, and CAN are brought out. A protocol logic chip is set on the expansion board so that the modified water accumulation sensor protocol can be connected to the core device through the chip. Each expansion board can connect four sets of water accumulation sensors with self-developed protocols, and the other brought-out interfaces provide the possibility of adding other sensors. The GPIO expansion board and the core device are placed in the cabinet, and the other connected sensors are brought out from the cabinet by inserting the corresponding interfaces of the expansion board. Therefore, each water accumulation detection node can be divided into two polarities, and each polarity can connect 9 water accumulation detection nodes. The water accumulation detection node can be activated by connecting water accumulation detection plates of different polarities. The activated water accumulation detection node transmits the signal to the expansion board. At this time, the core device receives an auxiliary judgment signal of water accumulation. Each detection board can be connected to 4 water accumulation detection nodes, so a system can be connected to 36 groups of water accumulation detection plates. Any group that detects water accumulation will receive an auxiliary judgment signal of water accumulation.

[0023] Step 3: Deploy the hardware system. The system nodes include cameras, sensors, edge computing devices, antennas and interactive devices.

[0024] The nodes of this system include cameras, sensors, edge computing devices, antennas and interactive devices. Deploying artificial intelligence algorithms on local edge nodes can effectively reduce costs, while processing delays are small, stability is high, and flexible layout can be achieved without being affected by lines. The interactive device uses a large-area LED bulletin board with strong anti-reflection ability to ensure that the driver can see the interactive content clearly from a distance; the sensor uses the modified sensor of step 2 to provide physical sensors and artificial intelligence methods for road infrastructure warning and road abnormality warning, and forms redundancy to ensure the normal operation of the warning function; the camera uses a 1080P resolution camera with infrared night vision function to provide effective protection for night road detection and curve warning. The system configuration of the present invention is flexible, and each node can freely combine multiple edge computing devices, cameras, sensors, and interactive devices according to different scales to achieve on-demand configuration. Not only in the curve warning scenario, two groups of cameras and two groups of edge computing devices can be assumed on both sides of the curve, and the sensors can be arranged according to the actual situation. In the application scenarios of road infrastructure warning and road abnormality warning, the nodes can be densely arranged in key sections, and the interactive device can be omitted, and only the detection results are uploaded to the superior decision-making body through the mobile network using antennas.

[0025] In order to realize the modularity and scalability of the system, the system of the present invention takes the edge computing device as the core, the switch and the expansion board as the node switching devices, and the LED screen, the surveillance camera, the water accumulation sensor and the temperature and humidity sensor as the peripheral devices. The three devices are capable of being replaced or updated separately to meet the flexible deployment requirements of complex usage environments. The LED screen and the surveillance camera are connected to the switch and communicate with the edge computing device in both directions through the RJ-45 interface; the IoT sensor peripheral devices such as the water accumulation sensor and the temperature and humidity sensor are connected to the expansion board to provide the edge computing device with the ability to perceive the surrounding environment, enhance the accuracy of machine learning judgment, and reduce misjudgments and missed judgments. The switch is connected to each connected device through a network cable, which has the advantages of strong anti-interference ability and long-distance laying. The expansion board is connected to the edge computing device through a dedicated 40PIN cable and is placed next to the edge computing device. The connection with the sensor uses the self-developed protocol and the I2C protocol, both of which have certain anti-interference capabilities.

[0026] Step 4: Integrate the software part and the hardware part of the present invention.

[0027] Integrate various software programs into hardware to form intelligent interactive scenes, where the software framework is divided into four parts: system detection module, Kafka module, system analysis module and system information dissemination module. The system detection module is responsible for artificial intelligence algorithm target detection, Kafka is the information transmission module, the system analysis module is the post-processing module of the system, including sensor information post-processing and information release post-processing, and the information dissemination module is mainly responsible for the connection between the system main program output and the LED screen.

[0028] 4.1 System Detection Module

[0029] The system detection module is responsible for the target detection of artificial intelligence algorithms. It uses DeepStream as the framework and develops a pruning optimization algorithm based on the YOLOV8 network, which optimizes the accuracy and speed of the original network framework so that it can be deployed on the edge computing module. In order to improve the operating efficiency, the module is developed in C++ language, avoiding the operating efficiency problems caused by the original network architecture using Python. The operating efficiency is also improved after being compiled into a native application of the aarch64 architecture.

[0030] The AI ​​processing module needs to load the trained weights before it can run, but the original weight precision is 32-bit floating point. Although the precision is high, the amount of calculation is large. Although a lot of optimization has been done on the network algorithm part, the performance is still not enough to support the simultaneous processing of multiple 2K videos. Therefore, the network was pruned again, and the FP32 precision was adjusted to FP16 precision, which greatly improved the speed with less accuracy loss, making it possible for the scalable high-precision network to process multiple high-resolution video streams in real time. The network architecture optimized by C++FP16 pruning can achieve a detection speed of 128FPS, and the accuracy meets the requirements. It can process multiple 30FPS 2K resolution video streams in real time. When a small number of video streams are connected, it can reduce the energy consumption of the edge processing module while providing real-time performance and high accuracy, and extend the battery life when using batteries as energy.

[0031] 4.2 Kafka Module

[0032] To ensure the stable operation of the system, the present invention splits the program into several subroutines for execution. The artificial intelligence part is executed by the system detection module, and the system detection module outputs the detection results to the system analysis module through Kafka for analysis, and then outputs the analysis results. Adding subroutine modules can avoid the main program crash and thus the system crash, and can also make the system more stable. In the present invention, the Producer of Kafka is the system detection module, and the Consumer is the system analysis module. The system detection module outputs the detection result string, which is passed to the system analysis module through Kafka. The protocol layer that communicates with Kafka has been embedded in the system detection module and can be directly connected to Kafka-Server. Before Kafka is started, the front-end process ZooKeeper needs to be started. After ZooKeeper is started, Kafka is pulled up and the message transmission function is started. According to the system requirements, a daemon is written. If Kafka stops working, Kafka is restarted to ensure the stable operation of the system.

[0033] 4.3 System Analysis Module

[0034] The system analysis module is a post-processing module, including sensor information post-processing and information release post-processing. Sensor information post-processing includes further weighted processing of sensor information and conditional judgment processing, and information release post-processing includes post-processing of message data transmitted by the system detection module. The character strings transmitted by the system detection module through Kafka mainly include the result information of the identified objects, as well as ice, snow and water identification information. These character strings are all real-time identification information. Due to the restrictions of the screen refresh rate and output information rules, directly outputting real-time detection information will result in disorganized information, which will directly affect the curve warning scenario and the driver will not be able to obtain effective information. Therefore, the information needs to be cached and processed, and the processed information can be handed over to the information dissemination module for final display on the LED screen.

[0035] Sensor post-processing not only includes comparing and weighting the water accumulation, ice and snow warning output by the artificial intelligence network with the original data, but also includes a watchdog program to judge the sensor transmission error frame or sensor no response. If the error frame or empty frame transmitted is greater than 5 times, a reset command is issued to the sensor. Information release post-processing is a display strategy processing process that integrates various information. If the result string transmitted by the system detection module is legal and weighted with the original data of the sensor, the result frequency and sensor data parameters appearing in the result are judged, and the result is processed according to the display strategy, and finally output to the information dissemination module in the form of a string and displayed on the LED screen.

[0036] 4.4 Information dissemination module

[0037] The information dissemination module is mainly responsible for the connection between the system main program output and the LED screen. The LED screen controller uses the Dahua DH-PHTOC-PT11 control card, which is compatible with the Nova traffic protocol and needs to be developed based on this communication protocol. The traffic protocol has a plug-and-play host computer program, but it is incompatible with the Nvidia Jetson AGX Orin architecture and system used in this system. It needs to be redeveloped. After transplantation and testing, the control card comes with a complete communication protocol source code, and the control protocol is successfully connected to the core device used in this system. The information dissemination module is also responsible for the layout of the LED screen, cutting the overlong character string and displaying it in large font on the corresponding LED screen to remind the drivers of passing vehicles to drive carefully and pass slowly.

[0038] In addition, intelligent interactive scenarios can realize information comprehensive research, judgment and decision research. Information comprehensive research and judgment refers to the priority sorting of warning signals output by the artificial intelligence network, and the use of existing information priorities to make comprehensive decisions, displaying high-priority information on the LED screen of the interactive device, giving priority to the transmission of advanced warning information while minimizing the impact on the driver. At the same time, when a major dangerous situation occurs, the driver is given a high-alert reminder to remind the vehicle to slow down, and the dangerous situation is reported to the superior decision-making body to prevent the vehicle from entering in time.

[0039] Compared with the existing traffic target detection, the present invention has the following advantages:

[0040] 1. Compared with the existing traffic target detection, the present invention takes the edge computing device as the core, the switch and the expansion board as the node switching device, the LED screen, the surveillance camera, the water accumulation sensor and the temperature and humidity sensor as the peripheral devices. The three devices are capable of being replaced or updated separately, which can meet the flexible deployment in complex use environments;

[0041] 2. The edge computing device of the present invention provides the ability to perceive the surrounding environment, enhances the accuracy of machine learning judgment, reduces misjudgment and missed judgment, and the switch is connected to each connected device through a network cable, which has the advantages of strong anti-interference ability and long-distance laying.

[0042] 3. The present invention uses advanced target detection algorithms to reduce the amount of calculation while ensuring accuracy, so that the delay of output information is further reduced while the power consumption of edge computing devices is reduced;

[0043] 4. The present invention innovatively combines the use of improved water accumulation sensors and temperature sensors to determine the conditions of water accumulation and snow accumulation, thereby avoiding the situation where target detection cannot identify shapeless water accumulation and snow accumulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the system of the present invention;

[0045] Figure 2 It is a combination solution for road infrastructure warning and road abnormality warning nodes;

[0046] Figure 3 The simulation diagram of water accumulation detected by the water accumulation sensor node circuit;

[0047] Figure 4 This is the overall framework diagram of the system hardware;

[0048] Figure 5 This is the overall framework diagram of the system software;

[0049] Figure 6 This is a schematic diagram of the Kafka module workflow;

[0050] Figure 7 It is the workflow diagram of the system analysis module;

[0051] Figure 8 A flowchart for implementing the method of the present invention; DETAILED DESCRIPTION

[0052] In order to make the above and other purposes, features and advantages of the present invention more obvious, the following is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings: Figure 1 The scheme diagram proposed for the system of the present invention; Figure 2 It is a combination scheme of road infrastructure warning and road abnormality warning nodes; refer to Figure 3 This is a simulation diagram of the water accumulation sensor node circuit detecting water accumulation; refer to Figure 4 This is the overall framework diagram of the system hardware; refer to Figure 5 This is the overall framework diagram of the system software; refer to Figure 6 This is a diagram of the Kafka module workflow; refer to Figure 7 This is the workflow diagram of the system analysis module; refer to Figure 8 The flowchart of the method of the present invention is as follows:

[0053] Step 1: A video surveillance system is built on rural roads to collect road data. Based on the large amount of comprehensive road data collected, a curve target detection dataset RCODS suitable for the complex environment of rural roads is constructed. The performance of the dataset is verified and the accuracy and usability of the dataset constructed by the present invention are proved. Subsequently, the curve target detection dataset RCODS is used to train the YOLOv8 algorithm model to achieve the purpose of rapid identification of curve traffic target detection.

[0054] 1.1 Build a video surveillance system suitable for rural roads and collect video data of specific sections:

[0055] First of all, the surveillance camera is the main input device for data collection in the present invention. All recognition-related tasks of the system are performed by the surveillance camera as the video input device, so the clarity and resolution of the device are particularly important. The surveillance camera needs to provide clear and sharp video stream input during the day, and at the same time needs to have night vision capabilities to provide the system with black and white night vision images at night, so that the system can still be normally warned at night. Therefore, the present invention selects Dahua's 4 million infrared zoom gun-type network camera, model DH-IPC-HFW3433F-Z. Two surveillance cameras are set on both sides of the curve to monitor the lane and collect video data.

[0056] 1.2 Video data is read and stored frame by frame, and appropriate images are selected to build the curve target detection dataset RCODS:

[0057] Since continuous video data is collected, extracting pictures frame by frame will generate a large amount of redundant data, which will greatly increase the time of data processing while occupying a large amount of storage space. Therefore, the present invention adopts the method of taking a picture every ten frames. However, in the process of video data collection, the time span is relatively large, and the amount of picture information generated by taking a picture every ten frames is also too large. Therefore, further data redundancy removal operations are required. Therefore, the present invention introduces the imagehash function in the process of editing the data set algorithm script. The imagehash function is used to screen the pictures, and the repeated or highly similar images are eliminated to further compress the data set, but the obtained data set is still huge. Finally, in the compressed data set, a picture is randomly selected every ten pictures to be retained, which ensures that the selected sample is representative, and can also effectively avoid the problems caused by the deviation of the selection method. After the above series of operations, the present invention processes the video shot for a long time, and the sample image finally obtained is used as the data set required by the curve target detection system.

[0058] 1.3 Data labeling and classification:

[0059] The generated dataset images are complex and usually contain multiple categories of information such as weather, pedestrians, trucks, cars, traffic cones, headlights, and agricultural machinery. Select the information you need to identify and label and classify the target categories of the dataset. First, you need to download and install Anaconda, and change the Python version to Python 3.8.5 to configure the environment. The detailed steps are as follows: (1) Click Start in the lower left corner of the screen, find the Anaconda folder in the application bar, and open Anaconda Prompt (anaconda). (2) Enter the following commands in the prompt to create a virtual environment named labelme and install labelme. Note that when entering the command, you need to change the default Python version to the installed Python version. Enter python-version in the command line window to view the installed Python version. (3) After the above operation is completed, if you want to check whether the installation is successful, you need to enter the command condainfo --env in the command line window to view all environments. If the labelme environment name appears, it means that the installation is successful.

[0060] Then start to label and classify the selected target category in the system. The specific operations are as follows: (1) Put the dataset file dataset and the label name file label name into the same folder. Assume that the present invention puts the dataset in the path D:\project_1\Qingling. (2) Open Anaconda Prompt, enter activate labelme to activate the environment, then enter D: to enter the D drive, and then enter cd D:\project_1\Qingling to enter the path where the dataset is placed. The input here should be changed to your own path. (3) Enter labelme dataset--labels labels_name.txt--flagssunny,rainy,snowy, a software window will appear, and the corresponding label on the right means that the creation is successful. The three options of Flags are sunny, rainy, and snowy, and the Label list is the target category selected by yourself. (4) Click opendir, select the dataset folder, then click File, select Save Automatically, uncheck Save with image, and click changeOutput dir to create a new label empty folder in the same path and select it. In this way, the labeled JSON file is saved in label. The completed state is shown in the figure. (5) Select the image in the File list in the lower right corner of the software and start annotating. (6) Select Weather in the Flags column, then click Edit, and select the rectangular box to start annotating the object. Drag the rectangular box to annotate the target category you selected in the image. Then click the option corresponding to the object. If the rectangular box deviates when dragging, press ESC. If the annotation is wrong, press Edit polygons on the left to modify it. Finally, press Next Image on the left to annotate the next image.

[0061] 1.4YOLOv8 algorithm model training and verification

[0062] After the data set is constructed and annotated, the present invention will perform data set verification based on the YOLOv8 algorithm model. Since the data set provides the necessary information required in the curve target detection system, in order to cooperate with the system to make accurate judgments, the YOLOv8 algorithm model needs to be trained on the data set constructed by the present invention, and the YOLOv8 algorithm developed by Ultralytics can provide an efficient training mode for the target detection system. During the training process, the input data set needs to be divided into a training set and a test set according to a 9:1 mode. Next, the environment and hyperparameters need to be configured, and the hyperparameters are set and modified by setting a ql.yaml hyperparameter configuration file, and the target detection categories are set to "car", "truck", "pedestrian (person)" and multiple required categories. Then the YOLOv8 model is trained on the data set constructed by the present invention, iterated 150 rounds, and an optimal training weight is obtained for the test of the data set.

[0063] Finally, after the training of the dataset was completed and the optimal training weights were obtained, we tested and verified the performance of the curved object detection system. We randomly selected other photos taken by the curved video surveillance and put them into the curved object detection system to observe whether the pre-trained targets can be detected, and derived the relationship between the confidence value, precision rate and F1 score. We repeatedly set the confidence value until the curved object detection system can accurately identify the set target with high precision rate and F1 score.

[0064] Step 2: As the rain and snow can change freely and cannot be detected using the target detection function, a modified water accumulation sensor is designed to assist in judgment. At the same time, a protocol logic chip is added to the modified water accumulation sensor, and an expansion board with a GPIO interface is used. Combined with the temperature and humidity sensor SHT3x-ARP from the Swiss company Sensirion, the accuracy of road water accumulation and rain and snow detection is improved.

[0065] 2.1 Making a modified sensor

[0066] In the design of water accumulation sensor, the core is the design of comparator circuit. In the invention circuit design, a LM393 voltage comparator is first introduced, which can compare the voltage of the same phase terminal with the voltage of the opposite phase terminal, and output the result of the comparison in the form of high and low levels. Then a sliding rheostat is connected to the opposite phase terminal to adjust the voltage threshold for judging whether there is water accumulation. A water accumulation sensing device is designed at the same phase input terminal. It is composed of two electrodes DET_A and DET_B with opposite polarities placed at a certain distance, one electrode is grounded, and the other is connected to the same phase input terminal. At the same time, an LED alarm device is connected to the output terminal of the comparator to judge the high and low levels of the output. The high level is on and the low level is off. Assuming the total voltage is 10V, when there is no water between the two electrodes, the resistance composed of the two electrodes DET_A and DET_B is infinite by default. According to the circuit voltage division principle, the voltage divided by the same phase terminal approaches 10V. At this time, the voltage of the same phase terminal is greater than the voltage of the opposite phase terminal, the output terminal outputs a high level, and the LED light is on. When there is water between the two electrodes, it is equivalent to the presence of a medium between the two electrodes, which is equivalent to the presence of resistance. The voltage threshold is set by adjusting the sliding rheostat at the inverting end so that the voltage at the inverting end is just greater than the voltage at the inverting end. At this time, the output end outputs a low level and the LED light goes out.

[0067] Since the design of the water accumulation sensor node circuit is that the high level detects the absence of water, and the low level detects the presence of water accumulation, in order to facilitate the subsequent work, the present invention adds a SN74LVC1G04DBVR inverter at the output end of the circuit to change the high and low levels of the circuit output. In addition, in the design of the circuit, if there is water accumulation between the two plates, its organization will be changed and its on and off will be changed. However, the range of the two plates is too small, so the present invention expands the two plates to 18, DET_A and DET_B are divided into two groups of 9 each, and arranged according to 3×3, DET_A and DET_B are placed on the plates in pairs and spaced apart, and the plates are placed in different positions for water accumulation detection, which increases the detection area. Such a circuit integrated with several modules is the water accumulation sensor node circuit designed by the present invention. No matter where the water drips, as long as any two red and blue pole pieces of different polarities can be connected, a comparator circuit can be formed, thereby achieving the purpose of detecting water accumulation.

[0068] 2.2 Use the GPIO interface to connect the sensor to peripheral devices

[0069] The GPIO interface is used to connect sensor peripheral devices to the core device of the system. The peripheral device using I2C and self-developed protocol is an IoT sensor of water accumulation sensor. The GPIO interface contains a large number of protocols. The interface definition of each protocol is different, and the corresponding interface of the protocol needs to be brought out. The expansion board is connected to the GPIO interface of the core device through the 40PIN interface, and interfaces such as I2C, SPI, and CAN are brought out. A protocol logic chip is set on the expansion board so that the modified water accumulation sensor protocol can be connected to the core device through the chip. Each expansion board can connect four groups of water accumulation sensors with self-developed protocols, and the other brought out interfaces provide the possibility of adding other sensors. The GPIO expansion board and the core device are placed in the cabinet, and the other connected sensors are brought out from the cabinet by inserting the corresponding interface of the expansion board. Therefore, each water accumulation detection node can be divided into two polarities, and each polarity can connect 9 water accumulation detection nodes. The water accumulation detection node can be activated by connecting water accumulation detection plates of different polarities. The activated water accumulation detection node transmits the signal to the expansion board. At this time, the core device receives an auxiliary judgment signal of water accumulation. Each detection board can be connected to 4 water accumulation detection nodes, so a system can be connected to 36 groups of water accumulation detection plates. Any group that detects water accumulation will receive an auxiliary judgment signal of water accumulation.

[0070] Step 3: Deploy the hardware system. The system nodes include cameras, sensors, edge computing devices, antennas and interactive devices.

[0071] The nodes of this system include cameras, sensors, edge computing devices, antennas and interactive devices. Deploying artificial intelligence algorithms on local edge nodes can effectively reduce costs, while processing delays are small, stability is high, and flexible layout can be achieved without being affected by lines. The interactive device uses a large-area LED bulletin board with strong anti-reflection ability to ensure that the driver can see the interactive content clearly from a distance; the sensor uses the modified sensor of step 2 to provide physical sensors and artificial intelligence methods for road infrastructure warning and road abnormality warning, and forms redundancy to ensure the normal operation of the warning function; the camera uses a 1080P resolution camera with infrared night vision function to provide effective protection for night road detection and curve warning. The system configuration of the present invention is flexible, and each node can freely combine multiple edge computing devices, cameras, sensors, and interactive devices according to different scales to achieve on-demand configuration. Not only in the curve warning scenario, two groups of cameras and two groups of edge computing devices can be assumed on both sides of the curve, and the sensors can be arranged according to the actual situation. In the application scenarios of road infrastructure warning and road abnormality warning, the nodes can be densely arranged in key sections, and the interactive device can be omitted, and only the detection results are uploaded to the superior decision-making body through the mobile network using antennas.

[0072] In order to realize the modularity and scalability of the system, the system of the present invention takes the edge computing device as the core, the switch and the expansion board as the node switching devices, and the LED screen, the surveillance camera, the water accumulation sensor and the temperature and humidity sensor as the peripheral devices. The three devices are capable of being replaced or updated separately to meet the flexible deployment requirements of complex usage environments. The LED screen and the surveillance camera are connected to the switch and communicate with the edge computing device in both directions through the RJ-45 interface; the water accumulation sensor, the temperature and humidity sensor, and the IoT sensor peripheral devices are connected to the expansion board to provide the edge computing device with the ability to perceive the surrounding environment, enhance the accuracy of machine learning judgments, and reduce misjudgments and missed judgments. The switch is connected to each connected device through a network cable, which has the advantages of strong anti-interference ability and long-distance laying. The expansion board is connected to the edge computing device through a dedicated 40PIN cable and is placed next to the edge computing device. The connection with the sensor uses the self-developed protocol and the I2C protocol, both of which have certain anti-interference capabilities.

[0073] Step 4: Integrate the software part and the hardware part of the present invention.

[0074] Integrate each software program into the hardware, and the software framework is divided into four parts: system detection module, Kafka module, system analysis module and system information dissemination module. The system detection module is responsible for artificial intelligence algorithm target detection, Kafka is the information transmission module, the system analysis module is the post-processing module of the system, including sensor information post-processing and information release post-processing, and the information dissemination module is mainly responsible for the connection between the system main program output and the LED screen.

[0075] 4.1 System Detection Module

[0076] The system detection module is responsible for the target detection of artificial intelligence algorithms. It uses DeepStream as the framework and develops a pruning optimization algorithm based on the YOLOV8 network, which optimizes the accuracy and speed of the original network framework so that it can be deployed on the edge computing module. In order to improve the operating efficiency, the module is developed in C++ language, avoiding the operating efficiency problems caused by the original network architecture using Python. The operating efficiency is also improved after being compiled into a native application of the aarch64 architecture.

[0077] The AI ​​processing module needs to load the trained weights before it can run, but the original weight precision is 32-bit floating point. Although the precision is high, the amount of calculation is large. Although a lot of optimization has been done on the network algorithm part, the performance is still not enough to support the simultaneous processing of multiple 2K videos. Therefore, the network was pruned again, and the FP32 precision was adjusted to FP16 precision, which greatly improved the speed with less accuracy loss, making it possible for the scalable high-precision network to process multiple high-resolution video streams in real time. The network architecture optimized by C++FP16 pruning can achieve a detection speed of 128FPS, and the accuracy meets the requirements. It can process multiple 30FPS 2K resolution video streams in real time. When a small number of video streams are connected, it can reduce the energy consumption of the edge processing module while providing real-time performance and high accuracy, and extend the battery life when using batteries as energy.

[0078] 4.2 Kafka Module

[0079] To ensure the stable operation of the system, the present invention splits the program into several subroutines for execution. The artificial intelligence part is executed by the system detection module, and the system detection module outputs the detection results to the system analysis module through Kafka for analysis, and then outputs the analysis results. Adding subroutine modules can avoid the main program crash and thus the system crash, and can also make the system more stable. In the present invention, the Producer of Kafka is the system detection module, and the Consumer is the system analysis module. The system detection module outputs the detection result string, which is passed to the system analysis module through Kafka. The protocol layer that communicates with Kafka has been embedded in the system detection module and can be directly connected to Kafka-Server. Before Kafka is started, the front-end process ZooKeeper needs to be started. After ZooKeeper is started, Kafka is pulled up and the message transmission function is started. According to the system requirements, a daemon is written. If Kafka stops working, Kafka is restarted to ensure the stable operation of the system.

[0080] 4.3 System Analysis Module

[0081] The system analysis module is a post-processing module, including sensor information post-processing and information release post-processing. Sensor information post-processing includes further weighted processing of sensor information and conditional judgment processing, and information release post-processing includes post-processing of message data transmitted by the system detection module. The character strings transmitted by the system detection module through Kafka mainly include the result information of the identified objects, as well as ice, snow and water identification information. These character strings are all real-time identification information. Due to the restrictions of the screen refresh rate and output information rules, directly outputting real-time detection information will result in disorganized information, which will directly affect the curve warning scenario and the driver will not be able to obtain effective information. Therefore, the information needs to be cached and processed, and the processed information can be handed over to the information dissemination module for final display on the LED screen.

[0082] Sensor post-processing not only includes comparing and weighting the water accumulation, ice and snow warning output by the artificial intelligence network with the original data, but also includes a watchdog program to judge the sensor transmission error frame or sensor no response. If the error frame or empty frame transmitted is greater than 5 times, a reset command is issued to the sensor. Information release post-processing is a display strategy processing process that integrates various information. If the result string transmitted by the system detection module is legal and weighted with the original data of the sensor, the result frequency and sensor data parameters appearing in the result are judged, and the result is processed according to the display strategy, and finally output to the information dissemination module in the form of a string and displayed on the LED screen.

[0083] 4.4 Information dissemination module

[0084] The information dissemination module is mainly responsible for the connection between the system main program output and the LED screen. The LED screen controller uses the Dahua DH-PHTOC-PT11 control card, which is compatible with the Nova traffic protocol and needs to be developed based on this communication protocol. The traffic protocol has a plug-and-play host computer program, but it is incompatible with the Nvidia Jetson AGX Orin architecture and system used in this system. It needs to be redeveloped. After transplantation and testing, the control card comes with a complete communication protocol source code, and the control protocol is successfully connected to the core device used in this system. The information dissemination module is also responsible for the layout of the LED screen, cutting the overlong character string and displaying it in large font on the corresponding LED screen to remind the drivers of passing vehicles to drive carefully and pass slowly.

[0085] In addition, intelligent interactive scenarios can realize information comprehensive research, judgment and decision research. Information comprehensive research and judgment refers to the priority sorting of warning signals output by the artificial intelligence network, and the use of existing information priorities to make comprehensive decisions, displaying high-priority information on the LED screen of the interactive device, giving priority to the transmission of advanced warning information while minimizing the impact on the driver. At the same time, when a major dangerous situation occurs, the driver is given a high-alert reminder to remind the vehicle to slow down, and the dangerous situation is reported to the superior decision-making body to prevent the vehicle from entering in time.

Claims

1. A YOLOv8-based curve target detection system, comprising the following steps: Step 1: A video surveillance system is built on rural roads to collect road data. Based on the collected large amount of comprehensive road data, a curve target detection dataset RCODS suitable for the complex environment of rural roads is constructed. The performance of the dataset is verified and the accuracy and usability of the constructed dataset are proved. Then, the curve target detection dataset RCODS is used to train the YOLOv8 algorithm model to achieve the purpose of rapid identification of curve traffic target detection. (1a) Build a video surveillance system suitable for rural roads and collect video data of specific road sections: First of all, the surveillance camera is the main input device for data collection. All recognition-related tasks of the system are performed by the surveillance camera as the video input device, so the clarity and resolution of the device are particularly important. The surveillance camera needs to provide clear and sharp video stream input during the day, and at the same time needs to have night vision capabilities to provide the system with black and white night vision images at night so that the system can still provide normal warnings at night. Therefore, Dahua's 4 million infrared zoom gun-type network camera, model DH-IPC-HFW3433F-Z, was selected; two surveillance cameras were set on both sides of the curve to monitor the lanes and collect video data; (1b) Video data is read and stored frame by frame, and appropriate images are selected to construct the curve object detection dataset RCODS: Since the collected video data is continuous, extracting pictures frame by frame will generate a large amount of redundant data, which will occupy a large amount of storage space and greatly increase the data processing time. Therefore, we adopt the method of taking a picture every ten frames. However, in the process of video data collection, the time span is relatively large, and the amount of picture information generated by taking a picture every ten frames is too large. Therefore, further data redundancy removal operations are required. Therefore, we introduced the imagehash function in the process of editing the dataset algorithm script, and used the imagehash function to filter the pictures, and removed the repeated or highly similar images to further compress the dataset. However, the obtained dataset is still huge. Finally, in the compressed dataset, one picture is randomly selected every ten pictures to be retained. While ensuring that the selected samples are representative, it can also effectively avoid problems caused by the deviation of the selection method. After the above series of operations, the long-shot video is processed, and the sample images finally obtained are used as the dataset required by the curve target detection system; (1c) Data labeling and classification: The generated dataset images are complex and usually contain multiple categories of information such as weather, pedestrians, trucks, cars, traffic cones, headlights, and agricultural machinery. Select the information you need to identify and label and classify the target categories of the dataset. First, you need to download and install Anaconda, and change the Python version to Python 3.8.5 to configure the environment. The detailed steps are as follows: (1) Click Start in the lower left corner of the screen, find the Anaconda folder in the application bar, and open Anaconda Prompt (anaconda); (2) Enter the following commands in the prompt to create a virtual environment named labelme and install labelme. Note that when entering the command, you need to change the default Python version to the installed Python version; enter python-version in the command line window to view the installed Python version; (3) After the above operations are completed, if you want to check whether the installation is successful, you need to enter the command conda info--env in the command line window to view all environments. The presence of the labelme environment name indicates that the installation is successful; Then start to label and classify the selected target category in the system. The specific operations are as follows: (1) Put the dataset file dataset and the label name file label name into the same folder; assume that the dataset is placed in the path D:\project_1\Qingling; (2) Open Anaconda Prompt, enter activate labelme to activate the environment, then enter D: to enter the D drive, and then enter cd D:\project_1\Qingling to enter the path where the dataset is placed. The path here should be changed to your own path; (3) Enter labelme dataset--labels labels_name.txt--flags sunny,rainy,snowy, and the software window will appear. If there is a corresponding label on the right, it means that the creation is successful; the three options of Flags are sunny, rainy, and snowy, and the Label list is the target category you selected; (4) Click opendir, select the dataset folder, then click File, select Save Automatically, uncheck Save with image, and click change Output dir. You can create a new label empty folder in the same path and select it; the labeled JSON file is saved in label. (5) Click File in the lower right corner of the software Select a picture from the list and start labeling; (6) Select Weather in the Flags column, then click Edit, select a rectangular box to start labeling objects; drag the rectangular box to label the target category you selected in the picture; then click the option corresponding to the object; if the rectangular box is dragged inaccurately, press ESC; if the labeling is wrong, press Edit polygons on the left to modify it; finally, press Next Image on the left to label the next picture; (1d) YOLOv8 algorithm model training and verification After the dataset is constructed and labeled, the dataset verification based on the YOLOv8 algorithm model will be carried out; since the dataset provides the necessary information required in the curve target detection system, in order to cooperate with the system to make accurate judgments, the YOLOv8 algorithm model needs to be trained on the constructed dataset, and the YOLOv8 algorithm developed by Ultralytics can provide an efficient training mode for the target detection system; during the training process, the input dataset needs to be divided into training sets and test sets in a 9:1 mode; then, the environment and hyperparameters need to be configured, and the hyperparameters are set and modified by setting a ql.yaml hyperparameter configuration file, and the target detection categories are set to "car", "truck", "pedestrian" and other required categories; then the YOLOv8 model is trained on the constructed dataset, iterated 150 times, and an optimal training weight is obtained for the dataset test; After the training of the dataset was completed and the optimal training weights were obtained, the performance of the curved object detection system was tested and verified. Other photos taken by the curved video surveillance were randomly selected and put into the curved object detection system to observe whether the pre-trained targets could be detected. The relationship between the confidence value, precision rate and F1 score was derived, and the confidence value was repeatedly set until the curved object detection system could accurately identify the set target with high precision rate and F1 score. Step 2: As the rain and snow can change freely and cannot be detected using the target detection function, an improved water accumulation sensor is designed to assist in judgment. At the same time, a protocol logic chip is added to the improved water accumulation sensor, and an expansion board with a GPIO interface is used. Combined with the temperature and humidity sensor SHT3x-ARP from the Swiss company Sensirion, the accuracy of road water accumulation and rain and snow detection is improved; (2a) Fabrication of improved sensors In the design of water accumulation sensor, the core is the design of comparator circuit. In the invention circuit design, a LM393 voltage comparator is first introduced, which can compare the voltage at the same phase terminal and the voltage at the opposite phase terminal, and output the comparison result in the form of high and low levels. Then, a sliding rheostat is connected to the opposite phase terminal to adjust the voltage threshold for judging whether there is water accumulation. A water accumulation sensing device is designed at the same phase input terminal. It consists of two electrodes DET_A and DET_B with opposite polarities placed at a certain distance. One electrode is grounded and the other is connected to the same phase input terminal. At the same time, an LED alarm device is connected to the output terminal of the comparator to judge The high and low levels of the output, high level is on, low level is off; assuming the total voltage is 10V, when there is no water between the two electrodes, the resistance composed of the two electrodes DET_A and DET_B is infinite by default. According to the circuit voltage division principle, the voltage divided by the same phase end approaches 10V. At this time, the voltage at the same phase end is greater than the voltage at the reverse end. The output end outputs a high level and the LED light is on; when there is water between the two electrodes, it is equivalent to having a medium between the two electrodes, which is equivalent to the existence of a resistor. The voltage threshold is set by adjusting the sliding rheostat at the reverse end so that the voltage at the reverse end is just greater than the voltage at the same phase end. At this time, the output end outputs a low level and the LED light is off; Since the design of the water accumulation sensor node circuit is that a high level detects the absence of water and a low level detects the presence of water, in order to facilitate subsequent work, an SN74LVC1G04DBVR inverter is added to the output end of the circuit to change the high and low levels of the circuit output; In addition, in the design of the circuit, if there is water accumulation between the two plates, it will change its organization and change its on and off, but the range of the two plates is too small, so the two plates are expanded to 18, DET_A and DET_B are divided into two groups of 9 each, and arranged in 3×3, DET_A and DET_B are placed on the plates in pairs, and the plates are placed in different positions for water accumulation detection, which increases the detection area; Such a circuit integrated with several modules is the designed water accumulation sensor node circuit; No matter where the water drips, as long as any two red and blue electrodes of different polarities can be connected, a comparator circuit can be formed, thereby achieving the purpose of detecting water accumulation; (2b) Use the GPIO interface to connect the sensor to peripheral devices The GPIO interface is used to connect sensor peripheral devices to the core devices of the system. The peripheral devices that use I2C and self-developed protocols are IoT sensors of water accumulation sensors. The GPIO interface contains a large number of protocols. The interface definition of each protocol is different. It is necessary to lead out the interface corresponding to the protocol. The expansion board is connected to the GPIO interface of the core device through the 40PIN interface, and interfaces such as I2C, SPI, and CAN are led out. A protocol logic chip is set on the expansion board, so that the improved water accumulation sensor protocol can be connected to the core device through the chip. Each expansion board can connect four groups of water accumulation sensors with self-developed protocols, and other interfaces are led out to increase its Other sensors provide the possibility. The GPIO expansion board and the core equipment are placed in the cabinet. The other connected sensors are led out of the cabinet by inserting the corresponding interface of the expansion board. Therefore, each water accumulation detection node can be divided into two polarities. Each polarity can be connected to 9 water accumulation detection nodes. The water accumulation detection node can be activated by connecting water accumulation detection plates of different polarities. The activated water accumulation detection node transmits the signal to the expansion board. At this time, the core equipment receives the auxiliary judgment signal of water accumulation. Each detection board can be connected to 4 water accumulation detection nodes. Therefore, a system can be connected to 36 groups of water accumulation detection plates. Any group of systems that detect water accumulation will receive an auxiliary judgment signal of water accumulation. Step 3: Deploy the hardware system. The system nodes include cameras, sensors, edge computing devices, antennas, and interactive devices: The nodes of this system include cameras, sensors, edge computing devices, antennas and interactive devices. The artificial intelligence algorithm is deployed on the local edge nodes, which can effectively reduce costs. At the same time, the processing delay is small, the stability is high, and the layout can be flexible and not affected by the line. The interactive device uses a large-area LED bulletin board with strong anti-reflection ability to ensure that the driver can see the interactive content clearly from a distance; The sensors use the self-developed sensors in step 2 to provide cross-validation of physical sensors and artificial intelligence methods for road infrastructure warning and road anomaly warning, and form redundancy to ensure the normal operation of the warning function; the cameras use 1080P resolution cameras with infrared night vision functions to provide effective protection for night road detection and curve warning. The system configuration is flexible. Each node can freely combine multiple edge computing devices, cameras, sensors, and interactive devices according to different scales to achieve on-demand configuration. Not only in the curve warning scenario, two groups of cameras and two groups of edge computing devices can be assumed on both sides of the curve, and the sensors can be arranged according to the actual situation. In the application scenarios of road infrastructure warning and road anomaly warning, the nodes can be densely arranged on key sections, and the interactive equipment can be omitted. Only the detection results are uploaded to the superior decision-making body through the mobile network using antennas; In order to realize the modularity and scalability of the system, the system takes edge computing devices as the core, switches and expansion boards as node switching devices, LED screens, surveillance cameras, water accumulation sensors and temperature and humidity sensors as peripheral devices. All three devices have the ability to be replaced or updated separately to meet the flexible deployment requirements in complex usage environments. The LED screen and surveillance camera are connected to the switch, and communicate with the edge computing device in two directions through the RJ-45 interface; the water accumulation sensor, temperature and humidity sensor, and IoT sensor peripherals are connected to the expansion board to provide the edge computing device with the ability to perceive the surrounding environment, enhance the accuracy of machine learning judgment, and reduce misjudgments and missed judgments; the switch is connected to each connected device through a network cable, which has the advantages of strong anti-interference ability and long-distance laying. The expansion board is connected to the edge computing device through a dedicated 40PIN cable and placed next to the edge computing device. The connection between the sensor uses the self-developed protocol and the I2C protocol, both of which have a certain degree of anti-interference ability; Step 4: Integrate the software and hardware parts: The software framework is divided into four parts: system detection module, Kafka module, system analysis module and system information dissemination module. The system detection module is responsible for artificial intelligence algorithm target detection, Kafka is the information transmission module, the system analysis module is the post-processing module of the system, including sensor information post-processing and information release post-processing, and the information dissemination module is mainly responsible for the connection between the system main program output and the LED screen; (4a) System detection module The system detection module is responsible for AI algorithm target detection. It uses DeepStream as the framework and develops a pruning optimization algorithm based on the YOLOV8 network, which optimizes the accuracy and speed of the original network framework so that it can be deployed on the edge computing module. In order to improve the operating efficiency, the module is developed in C++ language, avoiding the operating efficiency problems caused by the original network architecture using Python. After being compiled into a native application of the aarch64 architecture, the operating efficiency is also improved. The AI ​​processing module needs to load the trained weights before it can run. However, the original weight precision is 32-bit floating point. Although the precision is high, the amount of calculation is large. Although the network algorithm has been optimized a lot, the performance is still not enough to support the simultaneous processing of multiple 2K videos. Therefore, the network is pruned and the FP32 precision is adjusted to FP16 precision. The speed is greatly improved with less accuracy loss, making it possible for the scalable high-precision network to process multiple high-resolution video streams in real time. The network architecture after C++FP16 pruning optimization can achieve a detection speed of 128FPS, and the accuracy meets the requirements. It can process multiple 30FPS 2K resolution video streams in real time. When a small number of video streams are connected, the energy consumption of the edge processing module can be reduced while providing real-time performance and high accuracy, and the battery life can be extended when using batteries as energy. (4b) Kafka module To ensure the stable operation of the system, the program is divided into several subroutines for execution. The artificial intelligence part is executed by the system detection module. The system detection module outputs the detection results to the system analysis module through Kafka for analysis, and then outputs the analysis results. Adding subroutine modules can avoid the main program crash and thus the system crash, and can also make the system more stable; the Producer of Kafka is the system detection module, and the Consumer is the system analysis module. The system detection module outputs the detection result string and passes it to the system analysis module through Kafka; the protocol layer that communicates with Kafka has been embedded in the system detection module and can be directly connected to Kafka-Server; before starting Kafka, the front-end process ZooKeeper needs to be started. After ZooKeeper is started, Kafka is pulled up and the message transmission function is started. According to system requirements, a daemon is written. If Kafka stops working, Kafka is restarted to ensure the stable operation of the system; (4c) System Analysis Module The system analysis module is a post-processing module, including sensor information post-processing and information release post-processing; sensor information post-processing includes further weighted processing of sensor information and conditional judgment processing, and information release post-processing includes post-processing of message data transmitted by the system detection module; the character strings transmitted by the system detection module through Kafka mainly include the identified object result information, as well as ice, snow and water identification information, and these character strings are all real-time identification information; limited by the screen refresh rate and output information rules, directly outputting real-time detection information will result in disorganized information, which will directly affect the curve warning scene and the driver will not be able to obtain effective information. Therefore, the information needs to be cached and processed, and the processed information can be handed over to the information dissemination module for final display on the LED screen; Sensor post-processing not only includes comparing and weighting the water accumulation, ice and snow warning output by the artificial intelligence network with the original data, but also includes a watchdog program to judge the sensor transmission error frame or sensor no response. If the error frame or empty frame transmitted is greater than 5 times, a reset command is issued to the sensor; information release post-processing is a display strategy processing process that integrates various information. If the result string transmitted by the system detection module is legal and weighted with the original data of the sensor, the result frequency and sensor data parameters appearing in the result are judged, and the result is processed according to the display strategy, and finally output to the information dissemination module in the form of a string and displayed on the LED screen; (4d) Information dissemination module The information dissemination module is mainly responsible for the connection between the system main program output and the LED screen. The LED screen controller uses the Dahua DH-PHTOC-PT11 control card, which is compatible with the Nova traffic protocol and needs to be developed based on this communication protocol. The traffic protocol has a plug-and-play host computer program, but it is incompatible with the Nvidia Jetson AGX Orin architecture and system used by the core device of the system and needs to be redeveloped. After transplantation and testing, the control card comes with a complete communication protocol source code, which successfully connects the control protocol with the core device used in this system. The information dissemination module is also responsible for the layout of the LED screen, cutting the extra-long character string and displaying it in large font on the corresponding LED screen to remind the drivers of passing vehicles to drive carefully and pass slowly; In addition, intelligent interactive scenarios can realize information comprehensive research, judgment and decision-making research. Information comprehensive research and judgment refer to prioritizing the warning signals output by the artificial intelligence network, and making comprehensive decisions based on the existing information priority. High-priority information can be displayed on the LED screen of the interactive device, while giving priority to transmitting advanced warning information while minimizing the impact on the driver. At the same time, when a major dangerous situation occurs, the driver will be given a high-alert reminder to remind the vehicle to slow down. At the same time, the dangerous situation will be reported to the superior decision-making body to prevent the vehicle from entering in time.

Images