Road intelligent sensing and real-time monitoring system

Abstract

The invention relates to the technical field of intelligent traffic, in particular to a highway intelligent sensing and real-time monitoring system, which comprises an intelligent sensing end, a monitoring end and a monitoring end, and is characterized in that the intelligent sensing end comprises a PTZ (Pan / Tilt / Zoom) camera and an all-sky imager; the monitoring cloud platform is in real-time communication with the intelligent sensing end, and comprises a vehicle detection model carrying a neural network algorithm, a road detection model carrying the neural network algorithm, an imaging adjustment module, a sample set updating module, a plan updating module and an alarm management module; the method is used for judging whether vehicle stop, vehicle congestion and slow driving, vehicle traffic accidents, vehicle retrograde driving, road flaws, road fire disasters and road spilled objects occur in the road images or not, and then alarming and plan display are carried out. According to the invention, high-accuracy detection of various road events is realized, and effective support is provided for road traffic safety management and operation.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a highway intelligent perception and real-time monitoring system. Background Art

[0002] The intelligent operation and management of highways has always been a complex challenge that highway managers have been facing. Despite the accumulation of rich data, video resources and image resources, managers have been pursuing more effective analysis and utilization methods to deal with the problem of "sensing but not knowing, sensing but not being smart". At present, highway management still relies on the traditional manual inspection and monitoring mode, but this method has problems such as heavy workload, huge data and low efficiency. In general, highway management still faces a series of challenges in terms of data diversity and insufficient processing capacity.

[0003] At the highway management level, the monitoring images are not clear enough, making it difficult to accurately judge, quickly make decisions and take decisive measures for accidents in the images. At the same time, due to the lack of intelligent statistical analysis capabilities for data from multiple devices, the convenience of application and management is affected to a certain extent.

[0004] There is a common problem with the currently accessible road equipment, data, and technological achievements, namely the lack of a unified, intuitive, and easy-to-operate software platform for standardized management and output. In addition, artificial intelligence technology lacks effective means of utilization and display, making it difficult to provide a complete basis and practical operation handle for business management.

[0005] For example, the invention patent document with application number 201910898587.5 discloses an automatic detection system for highway traffic incidents, which can only determine whether a vehicle is illegal during driving.

[0006] Therefore, how to utilize the high-precision images and environmental monitoring devices at the highway sensing end to improve the detection accuracy of various highway events and provide effective support for road traffic safety management and operations is a technical problem that needs to be solved. Summary of the invention

[0007] To this end, the present invention provides a highway intelligent perception and real-time monitoring system, which controls the exposure parameters of the PTZ camera through the interconnection of the all-sky imager and the monitoring cloud platform, thereby enhancing the environmental adaptability of the PTZ camera, and then through the clearer and more reliable highway images collected by the PTZ camera, high-accuracy detection of various highway events is achieved, and effective support is provided for road traffic safety management and operation through regularly updated detection models and plan sets.

[0008] To achieve the above objectives, the present invention proposes a highway intelligent perception and real-time monitoring system, comprising: An intelligent sensing terminal, including a PTZ camera and an all-sky imager, wherein the PTZ camera is used to collect road images capable of controlling exposure parameters, and the all-sky imager is used to collect cloud images; A monitoring cloud platform, which communicates with the intelligent sensing terminal in real time, and is equipped with a vehicle detection model of a neural network algorithm, a highway detection model of a neural network algorithm, an imaging adjustment module, a sample set update module, a plan update module, and an alarm management module; The vehicle detection model is used to detect vehicle violations and vehicle accidents in the highway image; The highway detection model is used to detect highway events in the highway image; The imaging adjustment module is used to control the PTZ camera to collect the highway image with exposure parameters corresponding to the weather type and light intensity, wherein the weather type is determined according to the cloud condition information represented by the cloud image; The sample set updating module is used to update the vehicle sample set of the vehicle detection model and the highway sample set of the highway detection model, wherein the vehicle sample set is generated by classifying and automatically annotating pictures of vehicle violations and vehicle accidents obtained from the network according to their text descriptions, and the highway sample set is generated by classifying and automatically annotating pictures of highway events obtained from the network according to their text descriptions; The plan updating module is used to intercept the text description corresponding to the picture obtained by the sample set updating module, and generate a plan set corresponding to the picture through a keyword extraction algorithm; The alarm management module corresponds the detection results of the highway detection model and the vehicle detection model to the plan set, and performs alarm and plan display.

[0009] Further, the exposure parameters of the PTZ camera include an initial aperture value, an initial shutter speed value, an initial sensitivity value, and a target brightness value; The imaging adjustment module is used to set the corresponding initial aperture value, initial shutter speed value and initial sensitivity value according to the weather type, and is also used to set the target brightness value that is in inverse proportion to the light intensity according to the light intensity; When the PTZ camera runs to the adjustment time, the initial aperture value, the initial shutter speed value and the initial sensitivity value are executed to automatically adjust the exposure with the target brightness value as the target.

[0010] Furthermore, the exposure parameters of the PTZ camera also include aperture range constraints and shutter speed range constraints; The imaging adjustment module is further used to set an aperture range constraint according to the aperture initial value and the light intensity, and to set a shutter speed range constraint according to the shutter speed initial value and the light intensity.

[0011] Furthermore, the imaging adjustment module is used to determine the weather type and its corresponding amount of rain, snow, fog or rain and fog according to the cloud condition information; The imaging adjustment module is also used to adjust the initial aperture value and the initial shutter speed value according to the amount of rain / snow / fog / rain and fog when it is determined to be a rainy day type, a snowy day type or a foggy day type.

[0012] Furthermore, the monitoring cloud platform also includes an image processing module; The image processing module is used to adjust the contrast, sharpness and white balance of the highway image according to the light intensity.

[0013] Furthermore, the monitoring cloud platform also includes an image stabilization module; The image stabilization module is used to fuse the multiple highway images collected by the PTZ camera within a set time period according to the target vehicle.

[0014] The above solution breaks through traditional limitations and has the excellent ability to conduct all-weather detection under various complex environmental conditions such as day, night, rain, snow, etc., demonstrating excellent environmental adaptability.

[0015] Furthermore, sources of the vehicle sample set and the highway sample set include an autonomously annotated picture set, an automatically annotated website picture library set and a public traffic picture set, wherein the automatically annotated website picture library set and the public traffic picture set are pictures acquired from the Internet, and the autonomously annotated picture set is generated by annotating pictures collected by historical PTZ cameras.

[0016] Furthermore, the sample set updating module is used to determine whether the pictures and texts in the automatic annotation website picture library set and the public traffic picture set are adopted according to the similarity between the pictures and the texts.

[0017] Furthermore, the sample set updating module is also used to determine whether to adopt the sampled review text based on the overall similarity between the sampled review text and all pictures in the corresponding automatic annotation website picture library set and the public traffic picture set.

[0018] Furthermore, the vehicle detection model includes a vehicle detection sub-model, a driving detection sub-model, an accident detection sub-model and a wrong-way detection sub-module; The vehicle detection sub-model, whose sample set is a normal driving vehicle picture, is used to identify the vehicle in the highway image and generate a labeled vehicle; The driving detection sub-model is used to determine whether a vehicle is stopped or slow-moving due to vehicle congestion based on the labeled vehicles in the plurality of highway images; The accident detection sub-model, whose sample set is a vehicle traffic accident picture, is used to determine whether a vehicle traffic accident occurs in the highway image; The wrong-travel detection submodule has a sample set of wrong-travel vehicle images, which is used to determine whether a vehicle is wrong-travelling in the highway image.

[0019] In the above scheme, the self-learning function of the detection model is realized. By continuously accumulating, analyzing and updating the sample set, the detection model can be gradually optimized to intelligently adapt to the complexity of various scenarios, and can dynamically adapt to the characteristics and changes of new scenarios. In the continuous iteration process, an intelligent detection model with relatively powerful capabilities is obtained.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. The exposure parameters of the PTZ camera are controlled through the interconnection of the all-sky imager and the monitoring cloud platform, which enhances the environmental adaptability of the PTZ camera. The clearer and more reliable highway images collected by the PTZ camera can achieve high-accuracy detection of various highway events. The regularly updated detection models and emergency plan sets provide effective support for road traffic safety management and operations.

[0021] 2. It has broken through traditional limitations and has the excellent ability to conduct all-weather detection under various complex environmental conditions such as day, night, rain, snow, etc., showing excellent environmental adaptability.

[0022] 3. The self-learning function of the detection model is realized. By continuously accumulating, analyzing and updating sample sets, the detection model can be gradually optimized to intelligently adapt to the complexity of various scenarios. It can dynamically adapt to the characteristics and changes of new scenarios. In the continuous iteration process, an intelligent detection model with relatively powerful capabilities is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the structure of a highway intelligent perception and real-time monitoring system according to an embodiment of the present invention; Figure 2 A detailed flow chart of a highway intelligent perception and real-time monitoring system according to an embodiment of the present invention; Figure 3 A highway defect marking diagram of the highway intelligent perception and real-time monitoring system according to an embodiment of the present invention; Figure 4 A vehicle annotation diagram of the highway intelligent perception and real-time monitoring system according to an embodiment of the present invention; Figure 5 Schematic diagram of the process of detecting road defects using the YOLOv10 neural network model of the highway intelligent perception and real-time monitoring system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0026] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0027] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0028] like Figures 1 to 5 As shown, the present invention provides a highway intelligent perception and real-time monitoring system, which controls the viewing angle, focal length and exposure parameters of the PTZ camera in rain, snow or fog through the interconnection of the all-sky imager and the monitoring cloud platform, thereby enhancing the environmental adaptability of the PTZ camera, and further realizing high-accuracy detection of various highway events through clearer and more reliable highway images collected by the PTZ camera and more accurately identified highway vehicle models of the monitoring cloud platform, thereby providing effective support for road traffic safety management and operation.

[0029] like Figures 1 to 5 As shown, this embodiment proposes a highway intelligent perception and real-time monitoring system, including: an intelligent perception terminal, including a PTZ camera and an all-sky imager, the PTZ camera is used to collect highway images capable of controlling exposure parameters, and the all-sky imager is used to collect cloud images; A monitoring cloud platform, which communicates with the intelligent sensing terminal in real time, and is equipped with a vehicle detection model of a neural network algorithm, a highway detection model of a neural network algorithm, an imaging adjustment module, a sample set update module, a plan update module, and an alarm management module; The vehicle detection model is used to detect vehicle violations and vehicle accidents in the highway image; The highway detection model is used to detect highway events in the highway image; The imaging adjustment module is used to control the PTZ camera to collect the highway image with exposure parameters corresponding to the weather type and light intensity, wherein the weather type is determined according to the cloud condition information represented by the cloud image; The sample set updating module is used to update the vehicle sample set of the vehicle detection model and the highway sample set of the highway detection model, wherein the vehicle sample set is generated by classifying and automatically annotating pictures of vehicle violations and vehicle accidents obtained from the network according to their text descriptions, and the highway sample set is generated by classifying and automatically annotating pictures of highway events obtained from the network according to their text descriptions; The plan updating module is used to intercept the text description corresponding to the picture obtained by the sample set updating module, and generate a plan set corresponding to the picture through a keyword extraction algorithm; The alarm management module corresponds the detection results of the highway detection model and the vehicle detection model to the plan set, and performs alarm and plan display.

[0030] The vehicle violations and vehicle accidents include vehicle parking, vehicle congestion and slow movement, vehicle traffic accidents, and vehicle reversal; the highway events include highway defects, highway fires, and highway spills. The PTZ camera can also collect highway images with controlled viewing angle and focal length. The vehicle detection model is used to determine whether a vehicle parking, vehicle congestion and slow movement, vehicle traffic accident, or vehicle reversal occurs in the highway image. If so, the viewing angle and focal length of the PTZ camera are controlled to track the target vehicle of the vehicle event closest to it, and generate a vehicle violation / accident video.

[0031] The vehicle detection model is a neural network model based on the YOLOv8 neural network. The vehicle detection model includes a vehicle detection sub-model, a driving detection sub-model, an accident detection sub-model and a reverse detection sub-module; the vehicle detection sub-model, whose sample set is a picture of a normally driving vehicle, is used to identify the vehicle in the highway image and generate a labeled vehicle with a label; the driving detection sub-model is used to determine whether a vehicle is stopped and a vehicle is congested and slow according to the labeled vehicles in the highway image; the accident detection sub-model, whose sample set is a picture of a vehicle traffic accident, is used to determine whether a vehicle traffic accident occurs in the highway image; the reverse detection sub-module, whose sample set is a picture of a reverse vehicle, is used to determine whether a vehicle reverses in the highway image.

[0032] Specifically, PyTorch is used to implement the vehicle target detection algorithm, and target detection is performed based on the YOLOv8 algorithm. In the training phase, the pre-trained model is used as the initial model for training, and then the network parameters are optimized through multiple iterations to achieve better detection performance. During the training process, learning rate decay and data enhancement are used to enhance the generalization ability and robustness of the model. The model is automatically trained by automatically increasing or decreasing the hyperparameters by set values ​​until the fitting state is reached for the testing phase. The hyperparameters include learning rate and batch size. In order to further improve the generalization ability of the model, we also use data enhancement techniques such as random rotation, scaling, cropping, and color transformation to expand the sample set while reducing the risk of overfitting.

[0033] In the testing phase, the set loaded images are used for detection. By setting the threshold, the detection boxes with confidence lower than the threshold are filtered out, and finally the detection results are obtained. At the same time, the detection results are saved in image or video format. The main libraries used in the code include PyTorch, NumPy, OpenCV, Pyside6, etc.

[0034] Among them, the highway detection model is based on the YOLOv10 neural network model. The road defects include cracks, inspection wells, crack blocks, pothole blocks and convex blocks. The highway sample set includes 5 categories of road defects and highway fires, highway broken plants, highway domestic garbage, and highway spilled building materials, a total of 9 categories. As the initial highway training sample set, there are a total of 4822 pictures and 1178 pictures in the validation set. Each category in the highway training sample set has a large number of rotations and different lighting conditions, which helps to train a more robust detection model. The training sample set of this article contains 4822 pictures in the training set and 1178 pictures in the validation set. Since the YOLOv10 algorithm has restrictions on the size of the input picture, all pictures are adjusted to the same size of 640x640 and the original aspect ratio is maintained. In order to enhance the generalization ability and robustness of the model, data enhancement techniques are used, including random rotation, scaling, cropping and color transformation, to expand the data set and reduce the risk of overfitting.

[0035] Specifically, see Figure 5 ,in Figure 5 (a) is the original road image captured by the PTZ camera. Figure 5 (a) Image preprocessing and grayscale generation Figure 5 (b) Figure 5(b) Input is based on the YOLOv10 neural network model for feature extraction, enhancement and recognition detection result output. The YOLOv10 neural network model includes the first to third convolutional layers. Each convolutional layer includes convolution, dynamic convolution layer and maximum pooling layer. The convolution is as follows: using a 3×3 convolution kernel, a step size of 1, the number of output channels is set to 64, and the activation function uses LeakyReLU, which increases the nonlinear characteristics and allows a certain amount of negative value transmission. Then, another convolution is performed, the number of output channels is set to 128, and the same 3×3 convolution kernel and LeakyReLU activation are used. The dynamic convolution layer DynamicConv generates a dynamic convolution kernel according to the input feature map. The maximum pooling layer further reduces the spatial size of the feature map to provide a more streamlined input for the subsequent layers, ensuring that the network retains the most significant features, that is, the pixel value of each position in the output image is determined based on whether it is a road surface. If it is a road surface, the pixel value is set to 255, otherwise it is 0, to achieve image binarization. In order to more accurately identify the real road surface area, all connected domains marked as road surfaces obtained by binarization are sorted by area, and the connected domain with the largest area is selected as the first connected domain of the actual road surface. Figure 5 (c) When calculating the contour area of ​​a connected domain, Green’s formula is used to effectively identify and filter noise and errors.

[0036] It should be noted that through the two convolutions, the underlying features of the image are extracted layer by layer to ensure rich feature information. The LeakyReLU activation function enables the network to capture more complex feature patterns, avoids the problem of neuron death, and improves the learning ability of the network. DynamicConv generates a dynamic convolution kernel based on the input feature map, and convolves the input feature map based on the generated dynamic convolution kernel to adapt to the differences in input features, thereby enhancing the adaptability of the network to features. The maximum pooling layer uses a 2×2 pooling window and a step size of 2 to reduce the spatial resolution of the feature map. By selecting the Green's formula in the pooling window to calculate the maximum value of the contour surface, the most significant connected domain is retained, enhancing the representativeness of the connected domain.

[0037] Other objects in the road image will appear as black holes in the first connected domain generated by the first convolutional layer. Figure 5 In (c), these hole regions are considered as candidate regions containing potential road defects.

[0038] In order to extract these potential road defect targets, the first connected domain of the road surface Figure 5 The binary image of (c) is inverted, and then the first connected domain of the original road surface after inversion is formed in the form of a white connected domain of potential road defects. Figure 5 The black hole region in (c) is marked as the first connected graph.

[0039] In addition to the potential road defect connectivity domain, Figure 5 (c) may also contain other areas outside the road surface area. In order to extract the potential road defect connected domain separately, the largest closed contour and its associated area in the first connected graph are identified in the second convolution layer, and the largest closed contour and its associated area are filled to generate a hole-free, complete connected domain recorded as the second connected domain of the road surface. Figure 5 (d) The second connected domain Figure 5 (d) with the first connected graph (including the first connected domain Figure 5 (c) The black hole area of ​​potential road defects) is bitwise ANDed to generate the second connected graph Figure 5 (e). The second connected domain Figure 5 (d) is generated in the second convolutional layer. The task objectives of the second convolutional layer are different from those of the first and third convolutional layers. The third connected domain is located in the third convolutional layer, that is, the second connected graph Figure 5 (e) The corresponding connected domain is further identified through the third convolutional layer to generate the third connected domain and the second connected graph. Figure 5 (e) and the third connected domain are bitwise ANDed to generate a third connected graph, and the recognition result of the third connected graph is used as the output of the three-level convolutional neural network model to more accurately locate potential road defects through the three-level convolution. The formula for the bitwise AND operation after the second-level convolutional layer is: ,

[0040] In the formula, is a first connected graph including a first connected domain Figure 5 The black hole area of ​​potential road defects in (c) is the second connected domain, Figure 5 (d), is the second connected graph, Figure 5 (e). The bitwise AND operation ensures that only the first connected domain Figure 5 (c) and the second connected domain Figure 5 Only areas where both (d) and (d) exist are selected as candidate areas for spillage.

[0041] It should be noted that the YOLOv10 neural network model identifies a potential road defect as a road defect only if the conditions are met at the same time: the defect is detected as a potential road defect, the centroid of the potential road defect is located within the road area, the connected domain is less than the connected domain threshold, the credibility IoU threshold of the road defect is greater than the defect credibility threshold, and the credibility IoU threshold of the road is greater than the road credibility threshold.

[0042] Specifically, the credibility IoU threshold adopts improved IOU to improve the recognition speed of the model for road defects. The loss function expression of the improved IOU is: , In the formula, To improve IOU, is the intersection-over-union ratio, which is the conventional calculation of the loss function, is the distance penalty term, which is the conventional calculation of the loss function, is the shape penalty term, which is the conventional calculation of the loss function, is the angle penalty term.

[0043] The angle penalty term calculates the difference between the real frame and the predicted frame by the following formula, so that the predicted frame converges quickly to the real frame and improves the model's recognition speed of road defects: , In the formula, is the angle penalty term, and are the horizontal and vertical coordinates of the center point of the real frame, respectively. and are the horizontal and vertical coordinates of the prediction box.

[0044] In the experimental analysis section, we use indicators such as precision and recall to evaluate the performance of the model, and use loss curves and PR curves to analyze the training process. We use tensorboard to record the loss curves of the model on the training sample set and the validation set. As the number of training times increases, the training loss and validation loss of the model gradually decrease, indicating that by expanding and updating the highway sample set, the highway detection model can continuously learn more accurate features.

[0045] Furthermore, the monitoring cloud platform also includes the spilled object fitting model, which is used to control the viewing angle and focal length of the PTZ camera to track the road spilled objects when the road detection model detects the presence of road spilled objects in the road image, fit the scattering trajectory of the road spilled objects according to the position of the road spilled objects in multiple road images, and determine the scattering vehicle that generates the road spilled objects according to the scattering trajectory.

[0046] Specifically, the shutter speed of the PTZ camera can reach 1 / 600 second and it is equipped with a point cloud map. The spilled object fitting model will generate corresponding position coordinates of the spilled objects in two road images with an interval of 1 / 30 second in the point cloud map and connect them to generate a spilling trajectory, and then determine the tagged vehicle from which the spilled object originated and record it. If the spilling trajectory of the spilled object exceeds the viewing angle of the PTZ camera, it is determined to be a passive spill.

[0047] Furthermore, the exposure parameters of the PTZ camera include an initial aperture value, an initial shutter speed value, an initial sensitivity value and a target brightness value; the imaging adjustment module is used to set the corresponding initial aperture value, initial shutter speed value and initial sensitivity value according to the weather type, and is also used to set a target brightness value that is inversely proportional to the light intensity according to the light intensity; when the PTZ camera runs to the adjustment time, the initial aperture value, initial shutter speed value and initial sensitivity value are executed, and the exposure is automatically adjusted with the target brightness value as the target.

[0048] The adjustment time is preferably 3 hours, which can be adjusted according to local meteorological changes.

[0049] Further, the exposure parameters of the PTZ camera also include aperture range constraints and shutter speed range constraints; the weather types include rainy day type, snowy day type and foggy day type; the imaging adjustment module is used to set the aperture initial value to a small aperture initial value, the shutter speed initial value to a faster shutter speed, and the sensitivity initial value to a medium sensitivity initial value when it is determined to be a rainy day type; the imaging adjustment module is used to set the aperture initial value to a medium aperture initial value, the shutter speed initial value to an extremely fast shutter speed, and the sensitivity initial value to a low sensitivity initial value when it is determined to be a snowy day type; the imaging adjustment module is used to set the aperture initial value to a medium aperture initial value, the shutter speed initial value to a slow shutter speed, and the sensitivity initial value to a high sensitivity initial value when it is determined to be a foggy day type; the imaging adjustment module is also used to set the aperture range constraint according to the aperture initial value and the light intensity, and to set the shutter speed range constraint according to the shutter speed initial value and the light intensity.

[0050] It is understandable that in complex lighting environments, such as the presence of reflected light from rain, fog, and snow, the automatic exposure system of the PTZ camera may not be able to provide ideal exposure effects. When shooting with strong reflected light, the background may be too bright, resulting in insufficient exposure of the foreground, or dead white or dead black, resulting in reduced image quality. In addition, the accuracy of the automatic exposure system of the PTZ camera depends to a large extent on the sensor performance of the camera, and high-performance lenses are often more expensive. Therefore, this embodiment sets up an all-sky imager to perceive accurate weather conditions in real time, and then controls multiple PTZ cameras around the all-sky imager through the monitoring cloud platform at the same time to reduce the cost of obtaining clear images.

[0051] Specifically, the small aperture initial value is smaller than the medium aperture initial value, the faster shutter speed is slower than the extremely fast shutter speed and faster than the slow shutter speed, and the medium sensitivity initial value is larger than the low sensitivity initial value and smaller than the low sensitivity initial value. Specifically, the initial values ​​of the rainy day type include a small aperture initial value smaller than f / 8, a relatively small shutter speed smaller than 1 / 60 second and larger than 1 / 700 second, and a medium sensitivity initial value larger than 200 and smaller than 300; the initial values ​​of the snowy day type include a medium aperture initial value smaller than f / 5.6, an extremely small shutter speed smaller than 1 / 800 second, and a low sensitivity initial value larger than 100 and smaller than 200; the initial values ​​of the foggy day type include a small aperture initial value smaller than f / 8, a slow shutter speed larger than 1 / 100 second, and a high sensitivity initial value larger than 400.

[0052] Specifically, the aperture range constraint is the aperture initial value plus or minus the light intensity parameter, and the light parameter is obtained by subtracting the light intensity (unit kLx) from the sunny day standard 150, divided by 150, multiplied by the aperture correction coefficient, and multiplied by the aperture standard value. The aperture correction coefficient is 8 to 12, preferably 10. The aperture standard value is determined according to the type of PTZ camera. The aperture standard value of the PTZ camera used in this embodiment is f / 16.

[0053] The shutter speed range constraint is the shutter speed initial value plus the shutter speed parameter, and the shutter speed parameter is obtained by subtracting the light intensity (unit kLx) from the sunny day standard 150, dividing by 150, multiplying by the shutter correction coefficient, and multiplying by the shutter speed standard value. The shutter correction coefficient is preferably 150. The aperture standard value is determined according to the type of PTZ camera. The aperture standard value of the PTZ camera used in this embodiment is 1 / 600s, that is, the shutter is opened 600 times in one second.

[0054] The aperture is a device used to control the amount of light that passes through the lens and enters the photosensitive surface inside the camera body. The size of the aperture determines the amount of light that passes through the camera lens. The larger the aperture (the smaller the F value), the more light passes through and the brighter the photo; the smaller the aperture (the larger the F value), the less light passes through and the darker the photo. Shutter speed (exposure time): The shutter speed determines the length of time the camera lens is open, that is, the shutter speed. The longer the shutter speed, the brighter the photo; the shorter the shutter speed, the darker the photo. The shutter speed also affects the ability to capture motion. A high-speed shutter can freeze the moment of a moving object, while a low-speed shutter can record the movement trajectory of a moving object. The sensitivity (ISO) determines the sensitivity of the camera to light. The higher the ISO, the higher the sensitivity to light and the brighter the photo; the lower the ISO, the lower the sensitivity to light and the darker the photo.

[0055] Further, the cloud condition information includes cloud height, cloud amount and cloud type, and the cloud type includes cumulus, stratus and cirrus; the imaging adjustment module is used to determine the weather type and its corresponding rain level, snowfall level and fog level according to the cloud height, cloud amount and cloud type; the imaging adjustment module is also used to adjust the aperture initial value and the shutter speed initial value of the set value according to the rain level / snowfall level / fog level when it is determined to be a rainy day type, a snowy day type or a foggy day type. The rainy day aperture setting value is the rainy day coefficient of 0.2 to 0.3 multiplied by the precipitation deviation between the rain level and the moderate rain level. For example, if the heavy rain has 25 ml more precipitation than the moderate rain, the precipitation deviation is 25. The setting values ​​for snowy and foggy days are similar. The setting value for rainy day shutter speed is the rainy day coefficient of 300 to 350 multiplied by the precipitation deviation between the rain level and the moderate rain level, and the result is taken inversely. The setting values ​​for snowy and foggy days are similar. Sensitivity is determined by adjusting shutter speed and aperture.

[0056] The imaging adjustment module is also used to reduce the initial small aperture value of the snowy aperture setting value and increase the initial shutter speed value of the snowy shutter speed setting value according to the amount of snow when it is determined to be a snowy day type; the imaging adjustment module is also used to increase the initial small aperture value of the foggy aperture setting value and reduce the initial shutter speed value of the foggy shutter speed setting value according to the amount of fog or rain and fog when it is determined to be a foggy day type.

[0057] By using the global threshold method to perform cloud recognition processing on darker sky images and the red-to-blue ratio threshold method to perform cloud recognition processing on brighter sky images, we can obtain cloud height, cloud amount and cloud type. Combined with local meteorological observation data and basic meteorological knowledge, we can determine the weather type.

[0058] Furthermore, the monitoring cloud platform also includes an image processing module; the image processing module is used to adjust the contrast, sharpness and white balance of the highway image according to the light intensity.

[0059] According to the clarity from the center to the edge of the image, determine whether to adjust the contrast, sharpness and white balance. If adjusted, adjust them linearly within the range of ±10% of the standard contrast. The adjusted contrast is 60 to 80 and the sharpness is 20 to 40.

[0060] Furthermore, the monitoring cloud platform also includes an image stabilization module; the image stabilization module is used to perform image fusion on the multiple highway images collected by the PTZ camera within a set time period according to the target vehicle.

[0061] Specifically, the image fusion adopts an alpha fusion model. For example, when the shutter speed of the PTZ camera is 1 / 600 second, the 10 highway images collected within 1 / 60 second are cropped with the same annotated vehicle as the center and input into the alpha fusion model for image fusion. The alpha fusion model directly weights the image at each corresponding position of the two images participating in the fusion, and alpha controls the proportion of each frame image participating in the fusion.

[0062] The above solution breaks through traditional limitations and has the excellent ability to conduct all-weather detection under various complex environmental conditions such as day, night, rain, snow, etc., demonstrating excellent environmental adaptability.

[0063] Furthermore, sources of the vehicle sample set and the highway sample set include an autonomously annotated picture set, an automatically annotated website picture library set and a public traffic picture set, wherein the automatically annotated website picture library set and the public traffic picture set are pictures acquired from the Internet, and the autonomously annotated picture set is generated by annotating pictures collected by historical PTZ cameras.

[0064] The public traffic picture collection is obtained from the typical traffic accidents, traffic incidents and highway incidents disclosed on the website of the Traffic Management Bureau.

[0065] The self-annotated image set ensures the basic accuracy of the sample set and the detection model. The automatically annotated website image library set uses the GroundingDINO model to automatically annotate and classify the website image library. The OCR text recognition model is used to determine whether the text description of the images published on the website meets the category of the detection model and determine whether it is adopted.

[0066] Furthermore, the sample set updating module is used to determine whether the pictures and texts in the automatic annotation website picture library set and the public traffic picture set are adopted according to the similarity between the pictures and the texts.

[0067] Furthermore, the sample set updating module is also used to determine whether to adopt the sampled review text based on the overall similarity between the sampled review text and all pictures in the corresponding automatic annotation website picture library set and the public traffic picture set.

[0068] Implemented using the CLIP model: Input the image into the image encoder of the CLIP model, and input the corresponding text into the text encoder of the CLIP model to obtain image embedding and text embedding respectively. Calculate the cosine similarity between each image embedding and each text embedding, and the cosine similarity between each text embedding and all image embeddings. The cosine similarity is the text vector of the text embedding multiplied by the image vector of the image embedding, divided by its vector modulus. If the cosine similarities all meet the threshold, it is judged to be adopted.

[0069] When the system runs to the update time, the sample set update module and the plan update module run to update the sample set and the plan set. The update time is preferably 3 days or 5 days, which can be adjusted according to the actual frequency of local traffic events.

[0070] In the above scheme, the self-learning function of the detection model is realized. By continuously accumulating, analyzing and updating the sample set, the detection model can be gradually optimized to intelligently adapt to the complexity of various scenarios, and can dynamically adapt to the characteristics and changes of new scenarios. In the continuous iteration process, an intelligent detection model with relatively powerful capabilities is obtained.

[0071] Furthermore, the alarm management module marks the results of the highway detection model, the vehicle detection model and the spilled object fitting model on the GIS road network monitoring map.

[0072] Furthermore, the PTZ camera also has day and night modes, and the monitoring cloud platform has shadow elimination and field of view correction functions to ensure efficient operation in various harsh environments.

[0073] The AI-driven highway intelligent perception and intelligent safety emergency dispatch system with efficient collaboration, real-time monitoring, intelligent decision-making, and visualization processes video images captured by fixed cameras in real time to achieve a high-accuracy alarm function. The system uses high-definition camera images on the highway for centralized analysis and automatically detects various traffic events and accidents, such as stopped vehicles, slow-moving vehicles in congestion, vehicle accidents, vehicles driving in the wrong direction, highway defects, highway fires, and highway spills. The system can trigger automatic alarms and video recordings in real time and quickly, providing effective support for road traffic safety management and operations.

[0074] The sensing end also includes a fixed type camera, which can adjust the corresponding detection mode. This active adaptability makes the system more intelligent, and effectively improves the real-time, accuracy and adaptability of the system compared to traditional systems that rely on manual configuration or simple static modes.

[0075] Specifically, the system can automatically adjust its working mode according to the changes in weather conditions to adapt to different weather conditions. This active adaptability can greatly improve the system's operating efficiency and real-time performance, ensuring optimal performance during both peak and trough traffic periods. Compared with traditional systems, this innovation not only improves the system's intelligence level, but also provides a more practical and real-time traffic monitoring solution.

[0076] At the same time, the system's automatic perception and adjustment of environmental changes further enhances its flexibility. For example, when bad weather or special events occur, the system can automatically switch to more adaptable exposure parameters and image processing parameters to ensure that the system continues to operate efficiently. This feature makes the system more robust and reliable, thus meeting the actual operational needs in different scenarios.

[0077] Therefore, the multi-mode automatic conversion function is significantly innovative in improving the system's adaptability, flexibility and real-time performance, enabling the system to better cope with traffic monitoring needs in complex and changing environments.

[0078] The system has excellent performance in image processing, introduces advanced image stabilization and shadow elimination functions, and achieves efficient and reliable data output through a standard Ethernet interface. Compared with traditional systems, this technical innovation not only focuses on improving image quality, but also pays more attention to the accuracy and convenience of system output data.

[0079] First, image stabilization technology effectively suppresses the vibration and vibration of the camera image through precise image processing methods, ensuring the stability of the image. This not only helps to improve the clarity and recognizability of the image, but also provides a reliable foundation for subsequent data processing. Compared with traditional systems that rely on the stability of the camera itself, this innovation significantly improves the actual application effect of the system in dynamic traffic scenes.

[0080] Secondly, the shadow removal function of image preprocessing further improves the image quality. Through intelligent algorithms, the system can eliminate shadows caused by lighting changes to ensure that the objects in the image are clearly visible. This not only helps to accurately monitor traffic incidents, but also provides a more reliable basis for subsequent data output.

[0081] Most importantly, the intelligent sensing end outputs data through a standard Ethernet interface. This technical choice makes the system's output data easier to transmit and process, and has wide compatibility. Compared with traditional systems, the use of a standard Ethernet interface not only reduces the complexity of integration and application, but also improves the real-time and reliability of data.

[0082] Therefore, innovations in image processing and data output not only have advantages in improving image quality, but also focus on the reliability and practicality of output data, making the system more efficient and reliable in actual operation.

[0083] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A highway intelligent perception and real-time monitoring system, characterized in that: include: An intelligent sensing terminal, including a PTZ camera and an all-sky imager, wherein the PTZ camera is used to collect road images capable of controlling exposure parameters, and the all-sky imager is used to collect cloud images; A monitoring cloud platform, which communicates with the intelligent sensing terminal in real time, and is equipped with a vehicle detection model of a neural network algorithm, a highway detection model of a neural network algorithm, an imaging adjustment module, a sample set update module, a plan update module, and an alarm management module; The vehicle detection model is used to detect vehicle violations and vehicle accidents in the highway image; The highway detection model is used to detect highway events in the highway image; The imaging adjustment module is used to control the PTZ camera to collect the highway image with exposure parameters corresponding to the weather type and light intensity, wherein the weather type is determined according to the cloud condition information represented by the cloud image; The sample set updating module is used to update the vehicle sample set of the vehicle detection model and the highway sample set of the highway detection model, wherein the vehicle sample set is generated by classifying and automatically annotating pictures of vehicle violations and vehicle accidents obtained from the network according to their text descriptions, and the highway sample set is generated by classifying and automatically annotating pictures of highway events obtained from the network according to their text descriptions; The plan updating module is used to intercept the text description corresponding to the picture obtained by the sample set updating module, and generate a plan set corresponding to the picture through a keyword extraction algorithm; The alarm management module corresponds the detection results of the highway detection model and the vehicle detection model to the plan set, and performs alarm and plan display.

2. The highway intelligent perception and real-time monitoring system according to claim 1 is characterized in that: The exposure parameters of the PTZ camera include an initial aperture value, an initial shutter speed value, an initial sensitivity value, and a target brightness value; The imaging adjustment module is used to set the corresponding initial aperture value, initial shutter speed value and initial sensitivity value according to the weather type, and is also used to set the target brightness value that is in inverse proportion to the light intensity according to the light intensity; When the PTZ camera runs to the adjustment time, the initial aperture value, the initial shutter speed value and the initial sensitivity value are executed to automatically adjust the exposure with the target brightness value as the target.

3. The highway intelligent perception and real-time monitoring system according to claim 2 is characterized in that: The exposure parameters of the PTZ camera also include aperture range constraints and shutter speed range constraints; The imaging adjustment module is further used to set an aperture range constraint according to the aperture initial value and the light intensity, and to set a shutter speed range constraint according to the shutter speed initial value and the light intensity.

4. The highway intelligent perception and real-time monitoring system according to claim 3 is characterized in that: The imaging adjustment module is used to determine the weather type and its corresponding amount of rain, snow, fog or rain and fog according to the cloud condition information; The imaging adjustment module is also used to adjust the initial aperture value and the initial shutter speed value according to the amount of rain / snow / fog / rain and fog when it is determined to be a rainy day type, a snowy day type or a foggy day type.

5. The highway intelligent perception and real-time monitoring system according to any one of claims 1 to 4, characterized in that: The monitoring cloud platform also includes an image processing module; The image processing module is used to adjust the contrast, sharpness and white balance of the highway image according to the light intensity.

6. The highway intelligent perception and real-time monitoring system according to any one of claims 1 to 4, characterized in that: The monitoring cloud platform also includes an image stabilization module; The image stabilization module is used to fuse the multiple highway images collected by the PTZ camera within a set time period according to the target vehicle.

7. The highway intelligent perception and real-time monitoring system according to any one of claims 1 to 4, characterized in that: The sources of the vehicle sample set and the highway sample set include an autonomously annotated picture set, an automatically annotated website picture library set and a public traffic picture set, wherein the automatically annotated website picture library set and the public traffic picture set are pictures acquired from the Internet, and the autonomously annotated picture set is generated by annotating pictures collected by historical PTZ cameras.

8. The highway intelligent perception and real-time monitoring system according to claim 7 is characterized in that: The sample set updating module is used to determine whether the pictures and texts in the automatic annotation website picture library set and the public traffic picture set are adopted according to the similarity between the pictures and the texts.

9. The highway intelligent perception and real-time monitoring system according to claim 7 is characterized in that: The sample set updating module is also used to determine whether to adopt the sampled review text based on the overall similarity between the sampled review text and all the pictures in the corresponding automatic annotation website picture library set and the public traffic picture set.

10. The highway intelligent perception and real-time monitoring system according to any one of claims 1 to 4, characterized in that: The vehicle detection model includes a vehicle detection sub-model, a driving detection sub-model, an accident detection sub-model and a reverse detection sub-module; The vehicle detection sub-model, whose sample set is a normal driving vehicle picture, is used to identify the vehicle in the highway image and generate a labeled vehicle; The driving detection sub-model is used to determine whether a vehicle is stopped or slow-moving due to vehicle congestion based on the labeled vehicles in the plurality of highway images; The accident detection sub-model, whose sample set is a vehicle traffic accident picture, is used to determine whether a vehicle traffic accident occurs in the highway image; The wrong-travel detection submodule has a sample set of wrong-travel vehicle images, which is used to determine whether a vehicle is wrong-travelling in the highway image.

Images