Road safety early warning method based on multi-sensor fusion and computer vision technology

Abstract

The invention relates to the technical field of traffic safety, in particular to a road safety early warning method based on multi-sensor fusion and a computer vision technology. The method comprises the following steps: respectively generating a position initial Boolean value and a speed initial Boolean value based on data of a current target object, and obtaining a weighted Boolean value by the position initial Boolean value and the speed initial Boolean value through a weighting algorithm; object and lane line detection is carried out on the current target object image, obstacles and lane lines are identified, and the driving direction and speed of the current target object at the next moment are predicted; obtaining a final fusion speed of the current target object based on the measurement speed, the simulation speed and the prediction speed at the next moment; and generating a standard driving Boolean value according to the type of the target object, the final fusion speed, the predicted driving direction and the positioning and position information, evaluating the collision danger level through the standard driving Boolean value and the weighted Boolean value, and generating a corresponding early warning signal. The road safety early warning comprehensiveness and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of traffic safety technology, and in particular to a road safety early warning method based on multi-sensor fusion and computer vision technology. Background Art

[0002] With the rapid development of society, transportation has become an important part of people's daily life. However, traffic congestion and traffic accidents not only pose a serious threat to people's life safety, but also impose a certain burden on the social economy. Especially in some old roads in ancient city areas or intersections of complex roads, although reflectors are often installed to assist drivers to peek into blind spots around corners, the effectiveness of this traditional measure is greatly reduced due to the obstruction of vision caused by neglect or improper maintenance. In order to improve road safety and reduce the occurrence of traffic accidents, people have been looking for effective solutions.

[0003] In recent years, intelligent road safety warning technology, as an emerging field, has gradually entered the public eye. However, these intelligent road safety warning technologies still face many challenges in practical applications. Existing intelligent road safety warning technologies predict road safety by measuring the speed and positioning of motor vehicles, non-motor vehicles, and pedestrians in real time. Although this can provide dynamic information of motor vehicles, non-motor vehicles, and pedestrians, the existing methods only use a single data source for road safety warnings. A single data source often cannot fully reflect the actual conditions of the road. For example, the impact of factors such as sudden changes in vehicle speed and changes in traffic lights on traffic safety is often difficult to capture and predict in real time through conventional speed and positioning data, thereby increasing the risk of traffic accidents. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problem that the prior art only uses a single data source for road safety warnings, does not fully consider relevant factors such as sudden changes in vehicle speed and traffic lights, resulting in low accuracy of road safety warnings.

[0005] To solve the above technical problems, the present invention provides a road safety warning method based on multi-sensor fusion and computer vision technology, comprising the following steps:

[0006] Based on the image of the current target object, the position information of the current target object is obtained, including: based on the image of the current target object, using image processing technology to perform coordinate mapping to obtain the position information of the current target object.

[0007] Preferably, the predicting the driving direction and speed of the current target object at the next moment based on the recognition results of the obstacle and the lane line and the measured speed and location of the current target object comprises: predicting the driving direction and speed of the current target object at the next moment through a Kalman filter algorithm based on the measured speed and location of the current target object;

[0008] Based on the recognition results of obstacles and lane lines, the prediction results of the driving direction and speed of the current target object at the next moment are optimized to obtain the optimized driving direction and speed of the current target object at the next moment.

[0009] Preferably, obtaining the final fusion speed of the current target object based on the measured speed, the simulated speed and the predicted speed at the next moment of the current target object includes:

[0010] Based on the positioning of the current target object, a first simulated speed of the current target object is obtained; based on the image of the current target object, a second simulated speed of the current target object is obtained by an optical flow method; based on the predicted driving direction and speed at the next moment, a path specification weight of the current target object is obtained;

[0011] The measured speed of the current target object, the first simulated speed, the second simulated speed, the path specification weight, and the predicted speed at the next moment are weightedly fused to obtain the final fused speed of the current target object;

[0012] The calculation formula for the final fusion speed of the current target object is:

[0013] V final =w1×V1+w2×V2+w3×V3+w4×V S ,

[0014] Among them, V final represents the final fusion speed, V1 represents the measured speed of the current target object, V2 represents the first simulated speed of the current target object, V3 represents the second simulated speed of the current target object, and V S represents the predicted speed of the current target object at the next moment, w1, w2, w3, and w4 represent the weight of the measured speed of the current target object, the weight of the first simulated speed, the weight of the second simulated speed, and the path specification weight of the predicted speed at the next moment, respectively.

[0015] Preferably, it also includes:

[0016] Based on the historical positioning data, a positioning error correction model is established, and based on the historical measurement speed data, a measurement speed error correction model is established;

[0017] Inputting the obtained measured speed and position of the current target object into the positioning error correction model and the measurement speed error correction model respectively, and outputting the positioning error compensation value and the measurement speed error compensation value respectively;

[0018] Based on the positioning error compensation value and the measurement speed error compensation value, the final fusion speed formula after correction is:

[0019] V=w1×(V1-E1)+w2×(V2-E2)+w3×V3+w4×V S ,

[0020] Where V represents the final fusion speed after correction, E1 and E2 represent the positioning error compensation value and the measurement speed error compensation value respectively, V1 represents the measurement speed of the current target object, V2 represents the first simulation speed of the current target object, V3 represents the second simulation speed of the current target object, and V S represents the predicted speed of the current target object at the next moment, w1, w2, w3, and w4 represent the weight of the measured speed of the current target object, the weight of the first simulated speed, the weight of the second simulated speed, and the path specification weight of the predicted speed at the next moment, respectively.

[0021] Preferably, based on the image of the current target object, the second simulated speed of the current target object is obtained by an optical flow method, and the calculation formula is:

[0022]

[0023] Wherein, V3 represents the second simulation speed of the current target object, D represents the moving distance of the current target object between two consecutive frames of images, and Δt represents the time difference between the current target object and two consecutive frames of images.

[0024] Preferably, performing object detection on the image of the current target object to identify obstacles within a preset range centered on the current target object includes: based on the image of the current target object, using any one of the YOLO and SSD target detection algorithms to detect obstacles within a preset range centered on the current target object;

[0025] Lane line detection is performed on the image of the current target object to identify the lane line within a preset range centered on the current target object, including: based on the image of the current target object, any one of Hough transform and deep learning model is used to detect the lane line within the preset range centered on the current target object.

[0026] Preferably, generating the initial Boolean value of the position based on the type, position and position information of the current target object includes: correcting the position of the current target object according to the position information of the current target object to obtain a corrected position;

[0027] Based on the type of the current target object and the corrected positioning, determine whether the current target object is within the preset range corresponding to the type of the current target object. If it is within the corresponding preset range, the initial Boolean value of the generated position is True, otherwise it is False;

[0028] The generating of the initial Boolean value of the speed based on the type of the current target object, the measured speed and the simulated speed includes: comparing the measured speed and the simulated speed of the current target object, and selecting the larger speed between the two as the judgment speed;

[0029] Based on the type of the current target object and the judgment speed, it is determined whether the current target object exceeds the speed limit value corresponding to the type of the current target object. If it exceeds the corresponding speed limit value, the initial Boolean value of the generated speed is False, otherwise it is True.

[0030] Preferably, the generating of the standard driving Boolean value based on the type of the current target object, the final fusion speed, the predicted driving direction at the next moment, the positioning and position information includes: judging whether the final fusion speed of the current target object meets the standard driving standard corresponding to the type of the current target object based on the type of the current target object and the final fusion speed;

[0031] Correcting the positioning of the current target object according to the position information of the current target object to obtain a corrected positioning;

[0032] Based on the type of the current target object, the corrected positioning, and the predicted driving direction at the next moment, respectively determining whether the corrected positioning of the current target object and the predicted driving direction at the next moment meet the standard driving standards corresponding to the type of the current target object;

[0033] If the final fusion speed of the current target object, the corrected positioning of the current target object, and the judgment results of the predicted driving direction at the next moment all meet the standard driving standards corresponding to the type of the current target object, the standard driving Boolean value generated is True, otherwise it is False.

[0034] Preferably, it also includes:

[0035] Acquire a driving violation data set, wherein the driving violation data set includes: measured speed, location, image and collision risk level of multiple driving violation samples;

[0036] Preprocessing the illegal driving data set, dividing the preprocessed illegal driving data set into a training set and a validation set, building a road safety warning model, training the road safety warning model through the training set, outputting the predicted danger level and collision danger level labels according to the road safety warning model, optimizing the loss function, validating the trained road safety warning model through the validation set, and obtaining a target road safety warning model;

[0037] The actual measured speed and location of the current target object are input into the target road safety warning model, and the predicted collision risk level of the current target object is output.

[0038] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0039] The road safety early warning method based on multi-sensor fusion and computer vision technology described in the present invention generates a final fusion speed based on the measured speed of the current target object, the simulated speed obtained according to the positioning and image of the current target object, and the speed of the current target object at the next moment, fuses multi-source data to provide more accurate speed estimation, reduces the error of a single data source, and improves the accuracy and reliability of speed estimation by weighted fusion of different speed information, thereby effectively improving the accuracy of road safety prediction. The initial position Boolean value is generated based on the type, positioning and position information of the current target object, the initial speed Boolean value is generated based on the type, measurement speed and simulated speed of the current target object, and the position and speed of the current target object are classified and judged according to the different types of the current target object, thereby improving the accuracy of the initial position Boolean value and the initial speed Boolean value, and the initial position Boolean value and the initial speed Boolean value are generated by a weighted algorithm, and the collision risk level is evaluated based on the weighted Boolean value and the standard driving Boolean value. Different Boolean values ​​provide multi-dimensional information, and the weighted Boolean value is generated in combination with the weighted algorithm, which helps to more comprehensively evaluate the collision risk level, provide accurate early warning signals, and improve the prevention effect of road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 It is a step flow chart of the road safety warning method based on multi-sensor fusion and computer vision technology of the present invention.

[0042] Figure 2 It is a structural diagram of the road safety warning method based on multi-sensor fusion and computer vision technology of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0044] Reference Figure 1 , 2 As shown, the present invention provides a road safety warning method based on multi-sensor fusion and computer vision technology, comprising the following steps:

[0045] Step S1: obtaining the measured speed, location and image of multiple target objects within the road range to be monitored, identifying the type of the current target object based on the image of the current target object, and obtaining the location information of the current target object, and obtaining the simulated speed of the current target object based on the location and image of the current target object;

[0046] In this embodiment, specifically, the target objects include: pedestrians, motor vehicles, and non-motor vehicles.

[0047] In this embodiment, the measured speed, location and image of multiple target objects are collected in real time by a data acquisition device installed on a column within the road range to be monitored, wherein the data acquisition device includes:

[0048] Millimeter wave radar, used to collect real-time measurement speeds of multiple target objects within the road range to be monitored;

[0049] Beidou positioning module, used to detect the positioning of multiple target objects within the road range to be monitored in real time;

[0050] The camera is used to collect images of multiple target objects within the road range to be monitored in real time.

[0051] In this embodiment, based on the positioning of the current target object, the first simulated speed of the current target object is obtained, including: obtaining the longitude and latitude coordinate data of the current target object at consecutive time points, calculating the difference in longitude and latitude coordinates between two time points, obtaining the position change, recording the time difference between the two time points, and using the position change and time difference, dividing the position change by the time difference to calculate the first simulated speed of the current target object.

[0052] In this embodiment, preferably, obtaining the position information of the current target object based on the image of the current target object includes: performing coordinate mapping based on the image of the current target object using image processing technology to obtain the position information of the current target object.

[0053] Specifically, the image of the current target object is collected in real time by a camera, and based on the image of the current target object collected in real time, a filter (such as a Gaussian filter) is applied to reduce noise in the image, enhance image quality, and adjust the contrast, brightness or other parameters of the image to improve the visibility of the current target object;

[0054] Perform feature extraction on the processed image of the current target object, define a reference coordinate system, and map the pixel coordinates in the image to the actual world coordinate system. For example, select a point in the image as the origin, and define the scales of the horizontal and vertical axes. Obtain the intrinsic and extrinsic parameters of the camera through camera calibration. These parameters include focal length, principal point position, and rotation and translation information of the camera relative to the world coordinate system. Use a geometric transformation algorithm to map the image coordinates to the actual coordinate system. The geometric transformation algorithm can be a perspective transformation. Perspective transformation is often used to process the perspective distortion of the image so that the position of the target object can be accurately reflected in the actual coordinate system.

[0055] Image processing technology can provide accurate location information by analyzing the features in the image and mapping the coordinates, which helps to correct the positioning data of the current target object and reduce positioning deviations caused by sensor errors or environmental factors, thereby improving the overall positioning accuracy.

[0056] Step S2: Generate an initial Boolean value of the position based on the type, location and position information of the current target object; generate an initial Boolean value of the speed based on the type, measured speed and simulated speed of the current target object; and obtain a weighted Boolean value by adding the initial Boolean value of the position and the initial Boolean value of the speed through a weighted algorithm;

[0057] In this embodiment, preferably, generating the initial Boolean value of the position based on the type, position and position information of the current target object includes: correcting the position of the current target object according to the position information of the current target object to obtain a corrected position;

[0058] Based on the type of the current target object and the corrected positioning, determine whether the current target object is within the preset range corresponding to the type of the current target object. If it is within the corresponding preset range, the initial Boolean value of the generated position is True, otherwise it is False;

[0059] The generating of the initial Boolean value of the speed based on the type of the current target object, the measured speed and the simulated speed includes: comparing the measured speed and the simulated speed of the current target object, and selecting the larger speed between the two as the judgment speed;

[0060] Based on the type of the current target object and the judgment speed, it is determined whether the current target object exceeds the speed limit value corresponding to the type of the current target object. If it exceeds the corresponding speed limit value, the initial Boolean value of the generated speed is False, otherwise it is True.

[0061] In this embodiment, preferably, performing object detection on the image of the current target object to identify obstacles within a preset range centered on the current target object includes:

[0062] Based on the image of the current target object, use any of the YOLO and SSD target detection algorithms to detect obstacles within a preset range centered on the current target object. The YOLO and SSD target detection algorithms provide fast and accurate target detection and can identify static and dynamic obstacles in real time.

[0063] Specifically, the SSD target detection algorithm is used to detect obstacles within a preset range centered on the current target object, including: collecting an image of the current target object through a camera, adjusting the image size, normalizing the pixel values, and performing data enhancement, such as rotation and flipping, to improve the robustness of the model.

[0064] The preprocessed image is input into the basic network (VGG16 or ResNet network) in SSD to extract feature maps of different layers. A detection head is applied to each feature map to predict the position and category of each default box. Each detection head outputs a set of adjustment values ​​for the position of the bounding box and the category score of each box. The category score and position adjustment value are used to generate the recognition result of obstacles within a preset range centered on the current target object.

[0065] Specifically, using the YOLO target detection algorithm to detect obstacles within a preset range centered on the current target object includes: collecting an image of the current target object through a camera, adjusting the image of the current target object to a size required by a basic network (CNN or ResNet or DenseNet network) of the YOLO target detection algorithm, and performing a standardization process;

[0066] The processed image of the current target object is input into the basic network of the YOLO target detection algorithm to generate a feature map. The feature map is divided into a network, and multiple bounding boxes are predicted for each grid unit. The actual bounding box is generated according to the predicted bounding box coordinates and adjustment values. The confidence threshold is used to filter high-confidence bounding boxes, and non-maximum suppression is used to remove duplicate bounding boxes. After filtering, the recognition results of obstacles within a preset range centered on the current target object are output.

[0067] Perform lane line detection on the image of the current target object and identify the lane lines within a preset range centered on the current target object, including:

[0068] Based on the image of the current target object, any one of the Hough transform and deep learning models is used to detect the lane lines within a preset range centered on the current target object.

[0069] Specifically, based on the image of the current target object, Hough transform is used to detect lane lines within a preset range centered on the current target object, including: converting the image of the current target object into a grayscale image, applying Gaussian filtering to the grayscale image to reduce noise and details, using the Canny edge detection algorithm to detect edges in the image, generating a region of interest, namely, a Mask, converting the edge detection result into a parameter space of a straight line in the Hough transform space, mapping the detected straight line from the Hough space back to the original image space, and drawing the lane line.

[0070] Hough transform can accurately detect straight lines and lane lines, and can work effectively even in low-contrast situations. Deep learning models can provide higher robustness and adaptability, and can handle a variety of complex lane line scenarios, including curves and non-standard lane lines.

[0071] Step S3: performing object detection and lane line detection on the image of the current target object, identifying obstacles and lane lines within a preset range centered on the current target object, and predicting the driving direction and speed of the current target object at the next moment based on the recognition results of the obstacles and lane lines and the measured speed and positioning of the current target object;

[0072] In this embodiment, preferably, the predicting the driving direction and speed of the current target object at the next moment based on the recognition results of the obstacle and the lane line and the measured speed and positioning of the current target object includes: predicting the driving direction and speed of the current target object at the next moment through a Kalman filter algorithm based on the measured speed and positioning of the current target object;

[0073] Based on the recognition results of obstacles and lane lines, the prediction results of the driving direction and speed of the current target object at the next moment are optimized to obtain the optimized driving direction and speed of the current target object at the next moment.

[0074] Specifically, based on the measured speed and position of the current target object, the Kalman filter algorithm is used to predict the driving direction and speed of the current target object at the next moment. First, the state variables for predicting the trajectory of the current target object are defined, and a system model describing the state change of the current target object is established. According to the relationship between the measured speed and position of the current target object and the state variables, an observation model is established, and the filter is initialized to provide an initial state estimate and a covariance matrix. According to the system model, the Kalman filter algorithm is used to predict the current state, and the state estimate and the covariance matrix are updated. Using the observation model, the measured speed and position of the current target object are compared with the state prediction, and the Kalman filter algorithm is used to update the state estimate. The prediction and update steps are repeated. According to the real-time measured speed and position data of the current target object, the prediction and update steps are cyclically performed to continuously optimize the prediction of the vehicle trajectory.

[0075] The Kalman filter algorithm can provide smooth and accurate prediction results. Combined with obstacle recognition, it can optimize predictions in real time, avoid collisions, and intelligently plan paths. Lane line recognition information further ensures that the target object travels within the lane range, thereby enhancing the safety and stability of the system. This comprehensive optimization method improves the accuracy of predicting the driving direction and speed of the current target object at the next moment.

[0076] Step S4: based on the measured speed, simulated speed and predicted speed of the current target object at the next moment, obtain the final fusion speed of the current target object;

[0077] In this embodiment, preferably, the obtaining of the final fusion speed of the current target object based on the measured speed, the simulated speed and the predicted speed at the next moment of the current target object includes:

[0078] Based on the positioning of the current target object, a first simulated speed of the current target object is obtained; based on the image of the current target object, a second simulated speed of the current target object is obtained by an optical flow method; based on the predicted driving direction and speed at the next moment, a path specification weight of the current target object is obtained;

[0079] The measured speed of the current target object, the first simulated speed, the second simulated speed, the path specification weight, and the predicted speed at the next moment are weightedly fused to obtain the final fused speed of the current target object;

[0080] The calculation formula for the final fusion speed of the current target object is:

[0081] V final =w1×V1+w2×V2+w3×V3+w4×V S ,

[0082] Among them, V finalrepresents the final fusion speed, V1 represents the measured speed of the current target object, V2 represents the first simulated speed of the current target object, V3 represents the second simulated speed of the current target object, and V S represents the predicted speed of the current target object at the next moment, w1, w2, w3, and w4 represent the weight of the measured speed of the current target object, the weight of the first simulated speed, the weight of the second simulated speed, and the path specification weight of the predicted speed at the next moment, respectively.

[0083] In this embodiment, preferably, based on the image of the current target object, the second simulated speed of the current target object is obtained by an optical flow method, and the calculation formula is:

[0084]

[0085] Wherein, V3 represents the second simulation speed of the current target object, D represents the moving distance of the current target object between two consecutive frames of images, and Δt represents the time difference between the current target object and two consecutive frames of images.

[0086] Based on the image of the current target object, the second simulated speed of the current target object is obtained by the optical flow method, which can provide additional speed information and can calculate and adapt to different environmental conditions in real time.

[0087] In this embodiment, preferably, it also includes:

[0088] Based on the historical positioning data, a positioning error correction model is established, and based on the historical measurement speed data, a measurement speed error correction model is established;

[0089] Inputting the obtained measured speed and position of the current target object into the positioning error correction model and the measurement speed error correction model respectively, and outputting the positioning error compensation value and the measurement speed error compensation value respectively;

[0090] Based on the positioning error compensation value and the measurement speed error compensation value, the final fusion speed formula after correction is:

[0091] V=w1×(V1-E1)+w2×(V2-E2)+w3×V3+w4×V S ,

[0092] Where V represents the final fusion speed after correction, E1 and E2 represent the positioning error compensation value and the measurement speed error compensation value respectively, V1 represents the measurement speed of the current target object, V2 represents the first simulation speed of the current target object, V3 represents the second simulation speed of the current target object, and V Srepresents the predicted speed of the current target object at the next moment, w1, w2, w3, and w4 represent the weight of the measured speed of the current target object, the weight of the first simulated speed, the weight of the second simulated speed, and the path specification weight of the predicted speed at the next moment, respectively.

[0093] It also includes a feedback mechanism, which adjusts the parameters of the positioning error correction model according to the error between the positioning error compensation value and the actual positioning error compensation value, and adjusts the parameters of the measurement speed error correction model according to the error between the measurement speed error compensation value and the actual measurement speed error compensation value, so as to improve the reliability of the final fusion speed and thereby increase the accuracy of road safety warnings.

[0094] The final fused speed is generated based on the measured speed of the current target object, the positioning of the current target object, the simulated speed obtained by the image and the speed of the current target object at the next moment. The fusion of multi-source data provides a more accurate speed estimation and reduces the error of a single data source. By weighted fusion of different speed information, the accuracy and reliability of speed estimation are improved, effectively improving the accuracy of road safety predictions.

[0095] Step S5: Generate a standard driving Boolean value based on the type of the current target object, the final fusion speed, the predicted driving direction at the next moment, and the positioning and location information;

[0096] In this embodiment, preferably, the generating of the standard driving Boolean value based on the type of the current target object, the final fusion speed, the predicted driving direction at the next moment, the positioning and position information includes:

[0097] Based on the type and final fusion speed of the current target object, determine whether the final fusion speed of the current target object meets the standard driving standards corresponding to the type of the current target object; if the type of the current target object is a motor vehicle, determine whether the final fusion speed of the current target object exceeds the speed limit, and if so, it is considered irregular driving.

[0098] Correcting the positioning of the current target object according to the position information of the current target object to obtain a corrected positioning;

[0099] Based on the type of the current target object, the corrected position, and the predicted driving direction at the next moment, determine whether the corrected position of the current target object and the predicted driving direction at the next moment meet the standard driving standards corresponding to the type of the current target object; for example, if the current target object is a motor vehicle, determine whether the corrected position of the target object and the predicted driving direction at the next moment are in the motor vehicle lane, and whether there is any retrograde behavior. If so, it is irregular driving.

[0100] If the final fusion speed of the current target object, the corrected positioning of the current target object, and the judgment results of the predicted driving direction at the next moment all meet the standard driving standards corresponding to the type of the current target object, the standard driving Boolean value generated is True, otherwise it is False.

[0101] Step S6: evaluating the collision risk level according to the standard driving Boolean value and the weighted Boolean value, and generating a corresponding graded warning signal according to the collision risk level;

[0102] An initial position Boolean value is generated based on the type, positioning and position information of the current target object. An initial speed Boolean value is generated based on the type, measured speed and simulated speed of the current target object. The position and speed of the current target object are classified and judged according to the type of the current target object, thereby improving the accuracy of the initial position Boolean value and the initial speed Boolean value. The initial position Boolean value and the initial speed Boolean value are used to generate a weighted Boolean value through a weighted algorithm. The collision risk level is evaluated based on the weighted Boolean value and the standard driving Boolean value. Different Boolean values ​​provide multi-dimensional information. Combining the weighted algorithm to generate a weighted Boolean value helps to more comprehensively evaluate the collision risk level, provide accurate warning signals, and improve the prevention effect of road safety.

[0103] In this embodiment, a graded warning is issued to the current target by an alarm device installed or carried on the current target, including:

[0104] For non-motor vehicles, when the distance to the target object is 50 meters, a voice reminder will be issued through the wearable device or the embedded voice playback equipment on the non-motor vehicle. When the distance to the target object is 20 meters, a vibration or buzzer reminder will be added. When the distance to the target object is 5 meters, the voice reminder volume will be increased to the preset value and accompanied by vibration to further attract the attention of the non-motor vehicle driver.

[0105] For motor vehicles, when the distance to the target object is 50 meters, visual and voice warnings are issued through the on-board screen. When the distance to the target object is 20 meters, vibration or buzzer prompts are added. When the distance to the target object is 5 meters, the audio volume is increased to the preset value, accompanied by steering wheel vibration, to further attract the driver's attention.

[0106] By designing different warning strategies for non-motor vehicles and motor vehicles, safety and alertness can be significantly improved. When the distance to the target object is 50 meters, non-motor vehicles use voice reminders, and motor vehicles use visual and voice warnings through the on-board screen to help drivers identify potential dangers early. When the distance to the target object is 20 meters, vibration or buzzer prompts are added to further attract attention. When the distance to the target object is shortened to 5 meters, the audio sound is enhanced to the preset value and accompanied by vibration or steering wheel vibration, providing a strong multi-sensory reminder to reduce the driving burden, improve reaction time and effectively reduce the risk of collision.

[0107] In this embodiment, preferably, a driving violation data set is obtained, and the driving violation data set includes: measured speed, location, image and collision risk level of multiple driving violation samples;

[0108] Preprocessing the illegal driving data set, dividing the preprocessed illegal driving data set into a training set and a validation set, building a road safety warning model, training the road safety warning model through the training set, outputting the predicted danger level and collision danger level labels according to the road safety warning model, optimizing the loss function, validating the trained road safety warning model through the validation set, and obtaining a target road safety warning model;

[0109] The actual measured speed and location of the current target object are input into the target road safety warning model, and the predicted collision risk level of the current target object is output. The road safety warning model will continue to learn and optimize, and provide personalized safety suggestions to improve the accuracy and practicality of the warning, thereby enhancing driving safety and the driver's driving experience. Through this dynamic adjustment method, the road safety warning model can significantly reduce the accident rate, increase the driver's sensitivity to road risks, and promote safer and more efficient driving behavior.

[0110] This embodiment also includes saving the non-standard state to the cloud server. For example, a motor vehicle speed>80km / h, a non-motor vehicle speed>50km / h, or driving against the norm is defined as a serious non-standard state.

[0111] calculate:

[0112] Road management departments can use the big data of cloud servers to conduct formal normative analysis to determine the correlation between serious traffic accidents in downtown streets and driving regulations. This can be used to improve road management and quickly handle traffic accidents.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A road safety warning method based on multi-sensor fusion and computer vision technology, characterized in that: The following steps are involved: Obtain the measured speed, location and image of multiple target objects within the road range to be monitored, identify the type of the current target object based on the image of the current target object, and obtain the location information of the current target object, and obtain the simulated speed of the current target object based on the location and image of the current target object; Generate an initial position Boolean value based on the type, positioning and position information of the current target object; generate an initial speed Boolean value based on the type, measured speed and simulated speed of the current target object; and obtain a weighted Boolean value by adding the initial position Boolean value and the initial speed Boolean value through a weighted algorithm; Perform object detection and lane line detection on the image of the current target object, identify obstacles and lane lines within a preset range centered on the current target object, and predict the driving direction and speed of the current target object at the next moment based on the recognition results of obstacles and lane lines and the measured speed and positioning of the current target object; Based on the measured speed, simulated speed and predicted speed of the current target object at the next moment, the final fusion speed of the current target object is obtained; Generate a standard driving Boolean value based on the type of the current target object, the final fusion speed, the predicted driving direction at the next moment, and the positioning and location information; According to the standard driving Boolean value and the weighted Boolean value, the collision risk level is evaluated, and according to the collision risk level, a corresponding graded warning signal is generated.

2. A road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1, characterized in that: Based on the image of the current target object, the position information of the current target object is obtained, including: based on the image of the current target object, using image processing technology to perform coordinate mapping to obtain the position information of the current target object.

3. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1 is characterized in that: The method of predicting the driving direction and speed of the current target object at the next moment based on the recognition results of the obstacle and the lane line and the measured speed and location of the current target object includes: predicting the driving direction and speed of the current target object at the next moment through a Kalman filter algorithm based on the measured speed and location of the current target object; Based on the recognition results of obstacles and lane lines, the prediction results of the driving direction and speed of the current target object at the next moment are optimized to obtain the optimized driving direction and speed of the current target object at the next moment.

4. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1 is characterized in that: The method of obtaining the final fusion speed of the current target object based on the measured speed, the simulated speed and the predicted speed at the next moment of the current target object includes: Based on the positioning of the current target object, a first simulated speed of the current target object is obtained; based on the image of the current target object, a second simulated speed of the current target object is obtained by an optical flow method; based on the predicted driving direction and speed at the next moment, a path specification weight of the current target object is obtained; The measured speed of the current target object, the first simulated speed, the second simulated speed, the path specification weight, and the predicted speed at the next moment are weightedly fused to obtain the final fused speed of the current target object; The calculation formula for the final fusion speed of the current target object is: In final =w1×V1+w2×V2+w3×V3+w4×V S , Among them, V final represents the final fusion speed, V1 represents the measured speed of the current target object, V2 represents the first simulated speed of the current target object, V3 represents the second simulated speed of the current target object, and V S represents the predicted speed of the current target object at the next moment, w1, w2, w3, and w4 represent the weight of the measured speed of the current target object, the weight of the first simulated speed, the weight of the second simulated speed, and the path specification weight of the predicted speed at the next moment, respectively.

5. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 4 is characterized in that: Also includes: Based on the historical positioning data, a positioning error correction model is established, and based on the historical measurement speed data, a measurement speed error correction model is established; Inputting the obtained measured speed and position of the current target object into the positioning error correction model and the measurement speed error correction model respectively, and outputting the positioning error compensation value and the measurement speed error compensation value respectively; Based on the positioning error compensation value and the measurement speed error compensation value, the final fusion speed formula after correction is: <h2 style=";text-align:left;direction:ltr">V = w1×(V1-E1)+w2×(V2-E2)+w3×V3+w4×V<h2 style=";text-align:left;direction:ltr"> S <h2 style=";text-align:left;direction:ltr"> , Where V represents the final fusion speed after correction, E1 and E2 represent the positioning error compensation value and the measurement speed error compensation value respectively, V1 represents the measurement speed of the current target object, V2 represents the first simulation speed of the current target object, V3 represents the second simulation speed of the current target object, and V S represents the predicted speed of the current target object at the next moment, w1, w2, w3, and w4 represent the weight of the measured speed of the current target object, the weight of the first simulated speed, the weight of the second simulated speed, and the path specification weight of the predicted speed at the next moment, respectively.

6. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 4 is characterized by: Based on the image of the current target object, the second simulated speed of the current target object is obtained by the optical flow method. The calculation formula is: Wherein, V3 represents the second simulation speed of the current target object, D represents the moving distance of the current target object between two consecutive frames of images, and Δt represents the time difference between the current target object and two consecutive frames of images.

7. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1 is characterized by: Performing object detection on the image of the current target object to identify obstacles within a preset range centered on the current target object, including: based on the image of the current target object, using any one of the YOLO and SSD target detection algorithms to detect obstacles within a preset range centered on the current target object; Lane line detection is performed on the image of the current target object to identify the lane line within a preset range centered on the current target object, including: based on the image of the current target object, using any one of Hough transform and deep learning model to detect the lane line within a preset range centered on the current target object.

8. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1 is characterized by: The generating of the initial Boolean value of the position based on the type, position and position information of the current target object includes: correcting the position of the current target object according to the position information of the current target object to obtain a corrected position; Based on the type of the current target object and the corrected positioning, determine whether the current target object is within the preset range corresponding to the type of the current target object. If it is within the corresponding preset range, the initial Boolean value of the generated position is True, otherwise it is False; The generating of the initial Boolean value of the speed based on the type of the current target object, the measured speed and the simulated speed includes: comparing the measured speed and the simulated speed of the current target object, and selecting the larger speed between the two as the judgment speed; Based on the type of the current target object and the judgment speed, it is determined whether the current target object exceeds the speed limit value corresponding to the type of the current target object. If it exceeds the corresponding speed limit value, the initial Boolean value of the generated speed is False, otherwise it is True.

9. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1, characterized in that: The generating of the standard driving Boolean value based on the type of the current target object, the final fusion speed, the predicted driving direction at the next moment, the positioning and position information includes: judging whether the final fusion speed of the current target object meets the standard driving standard corresponding to the type of the current target object based on the type of the current target object and the final fusion speed; Correcting the positioning of the current target object according to the position information of the current target object to obtain a corrected positioning; Based on the type of the current target object, the corrected positioning, and the predicted driving direction at the next moment, respectively determining whether the corrected positioning of the current target object and the predicted driving direction at the next moment meet the standard driving standards corresponding to the type of the current target object; If the final fusion speed of the current target object, the corrected positioning of the current target object, and the judgment results of the predicted driving direction at the next moment all meet the standard driving standards corresponding to the type of the current target object, the standard driving Boolean value generated is True, otherwise it is False.

10. The road safety warning method based on multi-sensor fusion and computer vision technology according to claim 1, characterized in that: Also includes: Acquire a driving violation data set, wherein the driving violation data set includes: measured speed, location, image and collision risk level of multiple driving violation samples; Preprocessing the illegal driving data set, dividing the preprocessed illegal driving data set into a training set and a validation set, building a road safety warning model, training the road safety warning model through the training set, outputting the predicted danger level and collision danger level labels according to the road safety warning model, optimizing the loss function, validating the trained road safety warning model through the validation set, and obtaining a target road safety warning model; The actual measured speed and location of the current target object are input into the target road safety warning model, and the predicted collision risk level of the current target object is output.

Images