Road anomaly prediction and early warning method, device and equipment and computer readable storage medium

Abstract

The invention discloses a road anomaly prediction and early warning method, device and equipment and a computer readable storage medium, and relates to the technical field of intelligent driving safety, and the method comprises the steps: obtaining panoramic image data, an image semantic segmentation algorithm and an image anomaly detection algorithm; determining road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm; and predicting a motion track based on the panoramic image data, the road object information and the image abnormal information, and generating a corresponding early warning signal for road early warning. According to the method, the panoramic image data is acquired through the vehicle roof camera, and the road object information and the image abnormal information are accurately identified and the motion track state is predicted in combination with the image semantic segmentation and abnormal detection algorithm, so that the potential danger is judged, the real-time early warning signal is generated for early warning, the accuracy and the real-time performance of road early warning are effectively improved, and the road early warning efficiency is improved. And the driving safety is obviously enhanced, and traffic accidents are reduced.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving safety technology, and in particular to road anomaly prediction and warning methods, devices, equipment, and computer-readable storage media. Background Art

[0002] With the development of intelligent transportation systems and autonomous driving technology, people's demand for road safety is growing. Road anomaly prediction and warning can effectively reduce traffic accidents and improve driving safety. In a complex traffic environment, timely identification and warning of potential road anomalies can greatly avoid possible dangers.

[0003] Currently, existing practices mainly rely on on-board radar, laser sensors and forward-facing cameras to detect road conditions. These systems use data collected by sensors to monitor the environment around the vehicle, identify potential obstacles and abnormal conditions, and issue warnings to drivers in a timely manner to improve road safety.

[0004] However, existing approaches can only provide detection with a limited viewing angle, and have limitations in blind spot monitoring on the sides and rear of the vehicle. They are unable to perform detection when dealing with complex environments and blind spots. For example, the forward-facing camera may not be able to detect pedestrians or obstacles on the side of the vehicle, or radar and laser sensors may be affected by severe weather conditions, resulting in reduced detection accuracy and inability to effectively identify and warn of road anomalies caused by road construction, traffic accidents or other emergencies. Therefore, how to provide a panoramic view and efficiently and accurately provide real-time warnings of road anomalies has become an urgent problem to be solved.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a road anomaly prediction and warning method, device, equipment and computer-readable storage medium, aiming to solve the technical problem of how to provide a panoramic perspective and efficiently and accurately provide real-time warning of road anomalies.

[0007] To achieve the above objectives, the present application proposes a road anomaly prediction and warning method, the method comprising:

[0008] Obtain panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm;

[0009] Determine road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm;

[0010] Based on the panoramic image data, the road object information and the image abnormality information, a motion trajectory is predicted to generate a corresponding warning signal for road warning.

[0011] In one embodiment, the steps of acquiring panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm include:

[0012] Obtain road image data, alien vehicle image data, pedestrian image data, obstacle image data, image segmentation algorithm, loss function, isolation forest anomaly detection algorithm, speed deviation statistical anomaly detection algorithm and model speed distribution anomaly detection algorithm;

[0013] Determine panoramic image data based on the road image data, the other vehicle image data, the pedestrian image data and the obstacle image data;

[0014] Determine an image semantic segmentation algorithm based on the image segmentation algorithm and the loss function;

[0015] An image anomaly detection algorithm is determined based on the isolation forest anomaly detection algorithm, the speed deviation statistical anomaly detection algorithm and the model speed distribution anomaly detection algorithm.

[0016] In one embodiment, the step of determining the road object information and the image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm comprises:

[0017] Annotating the panoramic image data to determine the object category and the object image position;

[0018] Determine an object detection model training set, an object detection model verification set, and an object detection model test set based on the panoramic image data, the object category, and the object image position;

[0019] Acquire an object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data;

[0020] Inputting the panoramic image data into the object detection optimization model to perform image object detection to determine and annotate the panoramic image data;

[0021] Based on the object detection model training set, the object detection model test set and the object detection optimization model, the image semantic segmentation algorithm is applied to perform regional segmentation on the annotated panoramic image data to determine road object information, where the road object information includes lane line information and traffic sign information;

[0022] The anomaly information of the image is determined by identifying the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm.

[0023] In one embodiment, the step of acquiring the object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data includes:

[0024] Obtaining an object detection model and a model performance adjustment method, wherein the object detection model includes a region classification detection model, a grid division detection model, and a priori box detection model, and the model performance adjustment method includes a data augmentation operation, an operation to increase model complexity, and an introduction of a regularization operation;

[0025] Inputting the object detection model training set into the object detection model for training, adjusting the object detection model hyperparameters to determine an improved object detection model, wherein the object detection model hyperparameters include a learning rate and a number of iterations;

[0026] Determining evaluation indicators based on the object detection model verification set and the object detection improved model, wherein the evaluation indicators include accuracy, recall rate and average precision mean;

[0027] An object detection optimization model is determined based on the evaluation index and the model performance adjustment method.

[0028] In one embodiment, the step of applying the image semantic segmentation algorithm to perform region segmentation on the annotated panoramic image data based on the object detection model training set, the object detection model test set and the object detection optimization model to determine the road object information includes:

[0029] Based on the image semantic segmentation algorithm and the object detection model training set, the object detection optimization model is trained to obtain model road information;

[0030] Determining an abnormality indicator based on the object detection model test set and the object detection optimization model;

[0031] Based on the abnormal index, the object detection optimization model is visualized and analyzed to determine the model abnormal segmentation result, and based on the model abnormal segmentation result, the object segmentation optimization model is determined to obtain the road object information.

[0032] In one embodiment, the step of identifying and determining image anomaly information of the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm includes:

[0033] Acquiring target vehicle operating condition characteristics, wherein the target vehicle operating condition characteristics include normal vehicle driving characteristics and abnormal vehicle driving characteristics;

[0034] Performing data preprocessing on the annotated panoramic image data to determine cleaned panoramic image data;

[0035] Determine an abnormal deviation value based on the image abnormality detection algorithm, the cleaning panoramic image data and the target vehicle working condition characteristics, wherein the abnormal deviation value includes a false alarm rate and a false alarm rate;

[0036] The object detection optimization model is adjusted based on the object detection model verification set and the abnormal deviation value to obtain image abnormality information.

[0037] In one embodiment, the step of predicting the motion trajectory based on the panoramic image data, the road object information and the image abnormality information to generate a corresponding warning signal for road warning includes:

[0038] Obtain target sensor fusion data and target tracking algorithm;

[0039] Selecting a prediction model based on the target sensor fusion data, the target tracking algorithm and the panoramic image data to determine a predicted motion trajectory state;

[0040] Determine a road crisis state based on the road object information, the image abnormality information and the predicted motion trajectory state;

[0041] A warning signal is generated based on the road crisis state to perform road warning, and the warning signal includes an audible alarm and a visual alarm.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a road abnormality prediction and warning device, the road abnormality prediction and warning device comprising:

[0043] An acquisition module, used to acquire panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm;

[0044] A processing module, configured to determine road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm;

[0045] An execution module is used to predict the motion trajectory based on the panoramic image data, the road object information and the image abnormality information to generate a corresponding warning signal for road warning.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a road abnormality prediction and warning device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the road abnormality prediction and warning method as described above.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the road abnormality prediction and warning method as described above are implemented.

[0048] One or more technical solutions proposed in this application have at least the following technical effects:

[0049] The present application proposes a method for predicting and warning road anomalies, which obtains panoramic image data, an image semantic segmentation algorithm, and an image anomaly detection algorithm; determines road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm, and the image anomaly detection algorithm; and predicts the motion trajectory based on the panoramic image data, the road object information, and the image anomaly information to generate a corresponding warning signal for road warning. The present application obtains panoramic image data through a roof camera, and combines image semantic segmentation with anomaly detection algorithms to accurately identify road object information and image anomaly information, and predict the state of the motion trajectory, thereby judging potential dangers and generating real-time warning signals for warning, which effectively improves the accuracy and real-time performance of road warnings, and significantly enhances driving safety and reduces the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the road anomaly prediction and warning method of the present application;

[0053] Figure 2 A flowchart diagram of the second embodiment of the road anomaly prediction and warning method of the present application;

[0054] Figure 3 This is a schematic diagram of the module structure of the road anomaly prediction and warning device according to an embodiment of the present application;

[0055] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the road anomaly prediction and warning method in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solutions of the embodiments of the present application are: obtaining panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm; determining road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm; predicting motion trajectories based on the panoramic image data, the road object information and the image anomaly information to generate corresponding warning signals for road warnings.

[0060] In this embodiment, for ease of description, the following description is made with the road anomaly prediction and warning device being the execution subject.

[0061] As existing technologies can only provide detection with a limited viewing angle, there are limitations in monitoring blind spots on the side and rear of the vehicle, and detection cannot be performed when dealing with complex environments and blind spots. For example, the forward-facing camera may not be able to detect pedestrians or obstacles on the side of the vehicle, or radar and laser sensors may be affected by severe weather conditions, resulting in reduced detection accuracy and inability to effectively identify and warn of road anomalies caused by road construction, traffic accidents or other emergencies.

[0062] The present application provides a solution, which includes acquiring panoramic image data, an image semantic segmentation algorithm, and an image anomaly detection algorithm; determining road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm, and the image anomaly detection algorithm; and predicting motion trajectories based on the panoramic image data, the road object information, and the image anomaly information to generate corresponding warning signals for road warnings.

[0063] It can be seen from the above embodiments that the present application obtains panoramic image data through a roof camera, and combines image semantic segmentation and anomaly detection algorithms to accurately identify road object information and image anomaly information, and predict the motion trajectory state, thereby judging potential dangers and generating real-time warning signals for warning, effectively improving the accuracy and real-time performance of road warnings, and significantly enhancing driving safety and reducing the occurrence of traffic accidents.

[0064] Based on this, the present application embodiment provides a road abnormality prediction and warning method, referring to Figure 1 , Figure 1This is a flow chart of the first embodiment of the road anomaly prediction and warning method of the present application.

[0065] In this embodiment, there is a preferred implementation mode, in which the road warning system is provided with a roof camera, which is installed on the top of the vehicle and is used to provide panoramic image data. By installing the roof camera on the top of the vehicle, more comprehensive road information can be provided, thereby collecting abnormal road status information in complex environments and blind spots.

[0066] In this embodiment, there is also a preferred implementation mode, the road warning system is further provided with an image processing unit, and the image processing unit is used to receive image data transmitted by the roof camera and perform image analysis and processing tasks to more efficiently identify image anomalies.

[0067] In this embodiment, there is also a preferred implementation mode, in which the road warning system is further provided with a data fusion module, and the data fusion module is used to combine the image data of the roof camera and other sensor data, such as the forward camera and radar, so as to facilitate comprehensive analysis and more accurate assessment of road abnormalities.

[0068] In this embodiment, there is also a preferred implementation mode, the road warning system is further provided with a warning module, and the warning module is used to issue a warning signal according to the analysis result of the data fusion module, so as to more conveniently realize the road abnormality warning.

[0069] In this embodiment, there is also a preferred implementation mode, the road warning system is also provided with a display interface, and the display interface is used to display road information and warning prompts to the driver, so as to provide the user with more comprehensive road abnormality prompts more clearly.

[0070] In this embodiment, the road abnormality prediction and warning method includes steps S10 to S30:

[0071] Step S10, obtaining panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm;

[0072] It should be noted that the panoramic image data reflects the collected all-round environmental information around the vehicle, the image semantic segmentation algorithm reflects the characteristics of the algorithm for identifying and segmenting different objects and road signs in the image, and the image anomaly detection algorithm reflects the characteristics of identifying abnormal patterns or unusual behaviors in the image.

[0073] It can be understood that the panoramic image data represents the need for comprehensive perception of the vehicle's surrounding environment. The panoramic image data can be obtained using public data sets, such as COCO and PASCAL VOC, or by using a camera to collect the panoramic image data in an actual road environment. The image information captured by the roof camera provides a wide viewing angle that can cover the vehicle's surroundings, including the front, side and rear states, thereby capturing more environmental details, such as road signs, lane lines, pedestrians, vehicles and other obstacles. The image semantic segmentation algorithm can identify different objects in the image and classify and mark them, such as distinguishing between vehicles, pedestrians and road signs. The image anomaly detection algorithm can analyze image data and identify abnormal patterns or unusual behaviors in the image, such as emergencies, abnormal speed changes or other potential dangerous factors.

[0074] In addition, it should be noted that obtaining the panoramic image data can enable the system to monitor the road environment from a wider perspective, reduce blind spots, and improve the coverage of the warning. The use of image semantic segmentation algorithms can provide more accurate object location and category information. At the same time, the use of image anomaly detection algorithms can detect abnormal events in a timely manner, so that the warning system can more accurately identify potential dangerous situations, quickly analyze image data and respond, and provide drivers with real-time warning information, thereby significantly improving driving safety.

[0075] For ease of understanding, the acquisition of panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm are taken as examples for explanation, wherein the information acquisition device is an information acquisition module and the storage device is a memory.

[0076] The roof camera continuously collects road image data, image data of other vehicles, image data of pedestrians, and image data of obstacles around the vehicle, determines panoramic image data based on the road image data, image data of other vehicles, image data of pedestrians, and image data of obstacles, selects a suitable image segmentation algorithm, such as Adam and RMSprop, selects a suitable loss function, such as cross entropy loss and Dice loss, determines an image semantic segmentation algorithm based on the image segmentation algorithm and the loss function, selects a suitable image anomaly detection algorithm, such as an isolation forest anomaly detection algorithm, a speed deviation statistical anomaly detection algorithm, and the model speed distribution anomaly detection algorithm, the isolation forest anomaly detection algorithm constructs multiple isolated trees by randomly selecting features and segmentation points, and the anomalies will be isolated at relatively shallow positions. The speed deviation statistical anomaly detection algorithm can calculate the mean and standard deviation of the vehicle speed, and mark the speed value beyond a certain standard deviation range as abnormal. The model speed distribution anomaly detection algorithm establishes a probability distribution model of the speed, such as a normal distribution, in which points deviating from the distribution are considered to be abnormal. An image anomaly detection algorithm is determined based on the isolation forest anomaly detection algorithm, the speed deviation statistical anomaly detection algorithm and the model speed distribution anomaly detection algorithm, the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm are stored in a memory, and subsequent processing is performed based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm.

[0077] In a feasible implementation, step S10 may include steps A11 to A14:

[0078] Step A11, obtaining road image data, foreign vehicle image data, pedestrian image data, obstacle image data, image segmentation algorithm, loss function, isolation forest anomaly detection algorithm, speed deviation statistical anomaly detection algorithm and model speed distribution anomaly detection algorithm;

[0079] It should be noted that the road image data reflects the characteristics of the basic environment of the road, including lane lines, road surface markings and road boundaries, the other vehicle image data reflects the characteristics of the driving status of other vehicles, the pedestrian image data reflects the characteristics of the position and movement trajectory of pedestrians on the road, the obstacle image data reflects the characteristics of fixed or mobile obstacles on the road, such as roadblocks and accident vehicles, the image segmentation algorithm reflects the characteristics of segmenting and analyzing image content, the loss function reflects the characteristics of the metric for measuring the difference between model prediction and actual annotation when training the image segmentation model, the isolated forest anomaly detection algorithm reflects the characteristics of identifying abnormal patterns in image data, the speed deviation statistical anomaly detection algorithm reflects the characteristics of statistical analysis of vehicle speed to identify abnormal driving behavior, and the model speed distribution anomaly detection algorithm reflects the characteristics of learning and predicting vehicle speed distribution and identifying abnormal speed patterns that deviate from the normal distribution.

[0080] Step A12, determining panoramic image data based on the road image data, the other vehicle image data, the pedestrian image data and the obstacle image data;

[0081] It can be understood that by combining road image data, other vehicle image data, pedestrian image data and obstacle image data to determine panoramic image data, the comprehensiveness of the road warning system can be improved, thereby monitoring the road environment from a broader perspective, reducing blind spots, and significantly improving driving safety.

[0082] Step A13, determining an image semantic segmentation algorithm based on the image segmentation algorithm and the loss function;

[0083] It can be understood that by applying the image semantic segmentation algorithm, it is possible to perform regional segmentation on the image, identify lane lines and traffic signs, quickly analyze image data and respond, and enhance the real-time response capability of the system.

[0084] Step A14, determining an image anomaly detection algorithm based on the isolation forest anomaly detection algorithm, the speed deviation statistical anomaly detection algorithm and the model speed distribution anomaly detection algorithm.

[0085] It can be understood that the isolation forest anomaly detection algorithm provides the system with a fast and effective way to identify abnormal patterns in images, while the speed deviation statistical anomaly detection algorithm and the model speed distribution anomaly detection algorithm identify speed-related abnormal behaviors, thereby detecting and identifying abnormal conditions in images, improving the accuracy and reliability of anomaly detection, more accurately identifying abnormal patterns and speed deviations in images, effectively reducing traffic accidents, and improving driving safety.

[0086] Step S20, determining road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm;

[0087] It should be noted that the road object information reflects the characteristics of the specific state of the object in the road environment, including position, size and type, and the image abnormality information reflects the characteristics of abnormal behavior detected in the image that is significantly different from the normal situation.

[0088] It can be understood that by combining road object information and image anomaly information, different types of objects and abnormal events can be identified and distinguished, so as to more accurately predict potential dangers, identify abnormal situations in a timely manner, and quickly issue early warning signals.

[0089] For ease of understanding, the determination of road object information and image abnormality information is taken as an example for explanation, wherein the information acquisition device is an information acquisition module, the storage device is a memory, and the processing device is a processing module.

[0090] The information acquisition module acquires panoramic image data, such as vehicle image data, pedestrian image data, and obstacle image data, including weather conditions, lighting conditions, road types, and geographic locations, annotates the panoramic image data to determine object categories, such as vehicle categories, pedestrian categories, and obstacle categories, and uses an annotation tool, such as LabelImg, to determine the object image position, such as using a bounding box to annotate the object image position with a bounding box around it, and marking its category as "vehicle", determining an object detection model training set, an object detection model verification set, and an object detection model test set based on the panoramic image data, the object category, and the object image position, and acquiring an object detection model based on the object detection model training set, the object detection model verification set, and the panoramic image data. An object detection optimization model is obtained, the panoramic image data is input into the object detection optimization model for image object detection to determine the annotated panoramic image data, based on the object detection model training set, the object detection model test set and the object detection optimization model, the image semantic segmentation algorithm is applied to perform regional segmentation on the annotated panoramic image data to determine road object information, the road object information includes lane line information and traffic sign information, based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm, the annotated panoramic image data is identified to determine image anomaly information, the road object information and the image anomaly information are stored in a memory, and subsequent processing is performed based on the road object information and the image anomaly information.

[0091] Step S30: predicting a motion trajectory based on the panoramic image data, the road object information and the image abnormality information to generate a corresponding warning signal for road warning.

[0092] It should be noted that the warning signal reflects the characteristics of responding to the identified abnormal road conditions, including sound alarms and visual warnings.

[0093] It is understandable that if the abnormal behaviors and paths of multiple objects affect each other, the complexity and danger level will increase. When object detection finds a large obstacle at a close distance, and anomaly detection finds that the obstacle is where it should not be, and path prediction shows that it is about to be encountered, it can be judged as a high potential danger. If the object detected by the object is far away, but its speed is abnormal and it is heading in the direction of its own vehicle, it may be approaching in combination with path prediction, and it should also be judged as a high potential danger. As new data is continuously input, the above detection and prediction results will be updated in real time, re-evaluated, and then sent to the early warning module.

[0094] For ease of understanding, the generation of corresponding warning signals for road warning is taken as an example for explanation, wherein the information collection device is an information collection module, the storage device is a memory, and the execution device is an execution module.

[0095] The information acquisition module obtains the target sensor fusion data, fuses the image data of the roof camera with the data of the forward camera, radar and other sensors, obtains the target tracking algorithm, such as SORT and DeepSORT, selects the prediction model based on the target sensor fusion data, the target tracking algorithm and the panoramic image data to determine the predicted motion trajectory state, the prediction model includes Kalman filtering, particle filtering, recurrent neural network or long short-term memory network in deep learning, and trains the prediction model, uses the trained model to predict the motion trajectory in the future within a short period of time to determine the predicted motion trajectory, determines the road crisis state based on the road object information, the image abnormality information and the predicted motion trajectory state, obtains the road object information, including the type, position and size of the detected object, if the distance between the detected object and the own vehicle is less than the target value, for example, 20 meters, that is, the road crisis state determines that there is potential danger, Image abnormality information is obtained from abnormality detection, such as abnormal vehicle speed, sudden turn or abnormal object appearance. If the speed is abnormal, a target proportion exceeding the normal speed range is defined, such as 50%, that is, the road crisis state determines that there is a potential danger, and the predicted motion trajectory state is obtained, including the estimated driving path of the own vehicle and other related objects. If the predicted motion trajectory state crosses or approaches, and the time interval is short, such as 5 seconds, that is, the road crisis state determines that there is a potential danger, and a warning signal is generated based on the road crisis state for road warning. The warning signal includes sound alarm and visual warning. For example, on a highway, the object detects that multiple vehicles in front of the road suddenly slow down, and the abnormality detection finds that the deceleration amplitude exceeds the normal range. The path prediction shows that there may be a rear-end collision. At this time, the road crisis state determines that there is a potential danger, and a danger alarm is immediately issued. At the same time, abnormal conditions of vehicles in other lanes can also be detected, and suggestions are provided on whether to change lanes.

[0096] In a feasible implementation, step S30 may include steps B11 to B14:

[0097] Step B11, obtaining target sensor fusion data and target tracking algorithm;

[0098] It should be noted that the target sensor fusion data reflects the characteristics of integrating multi-source sensor information, and the target tracking algorithm reflects the characteristics of continuous tracking and analysis of multiple target objects.

[0099] It is understandable that the image data from the roof camera can be fused with the data from sensors such as the forward-facing camera and radar to provide more comprehensive road information. Through target tracking algorithms such as SORT and DeepSORT, the detected targets can be tracked in continuous image frames to obtain their position, speed and acceleration information. Then, a suitable prediction model is selected, such as Kalman filtering, particle filtering, recurrent neural networks in deep learning, or long short-term memory networks for training, and the trained model is used to predict its motion trajectory in the short term in the future.

[0100] Step B12, selecting a prediction model based on the target sensor fusion data, the target tracking algorithm and the panoramic image data to determine a predicted motion trajectory state;

[0101] It can be understood that by obtaining the predicted motion trajectory state, not only can the current dangerous situation be identified, but also possible risks in the future can be predicted and prevented, and the future state of the target can be predicted in real time, so that the system can send early warning signals to the driver in time and accurately predict potential collision risks.

[0102] Step B13, determining a road crisis state based on the road object information, the image abnormality information and the predicted motion trajectory state;

[0103] It should be noted that the road crisis state reflects the characteristics of the immediate road abnormal state.

[0104] It is understandable that once the road crisis state determines that there is potential danger, a warning signal will be triggered, such as an audible alarm or a visual warning, to remind the driver to take action, thereby taking preventive measures and reducing the occurrence of traffic accidents.

[0105] Step B14, generating a warning signal based on the road crisis state to perform road warning, wherein the warning signal includes a sound alarm and a visual alarm.

[0106] It can be understood that by generating a warning signal based on the road crisis status, a warning can be issued to the driver in time, thereby reducing the occurrence of traffic accidents, improving driving safety, and more accurately identifying and predicting potential risks, improving the reliability of the warning signal. At the same time, a large amount of data can be quickly processed and a warning signal can be generated, so that the driver can respond in time and reduce the occurrence of unexpected situations.

[0107] The road anomaly prediction and warning method proposed in this embodiment obtains panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm; determines road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm; predicts the motion trajectory based on the panoramic image data, the road object information and the image anomaly information to generate a corresponding warning signal for road warning. The technical problem of how to more effectively monitor the blind spots of vehicles for road warning is solved. Compared with the prior art, the present application uses a roof camera to obtain panoramic image data, covers the blind spots that are difficult to monitor by forward cameras and radars, and combines the image semantic segmentation algorithm and the image anomaly detection algorithm. Through the image semantic segmentation algorithm, various objects in the image are accurately identified and classified, and the image anomaly detection algorithm can capture abnormal patterns or unusual behaviors in the image in a timely manner, thereby predicting potential dangerous situations and generating real-time warning signals, providing drivers with more comprehensive and accurate road safety information, improving the accuracy and real-time performance of road warnings, and significantly enhancing driving safety, effectively reducing the occurrence of traffic accidents.

[0108] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0109] In this embodiment, refer to Figure 2 , Figure 2 The flowchart of the second embodiment of the road abnormality prediction and warning method of the present application is provided, and step S20 specifically includes steps S21 to S26:

[0110] Step S21, annotating the panoramic image data to determine the object category and the object image position;

[0111] It should be noted that the object category reflects the characteristics of the attributes of the object in the panoramic image data, and the object image position reflects the characteristics of the geometric position of the object in the panoramic image.

[0112] It can be understood that by accurately identifying the object category and determining the object image position, the accuracy of object detection is improved, and abnormal road conditions can be distinguished and identified more reliably.

[0113] For ease of understanding, an explanation is given by taking determining the object category and the object image position as an example, wherein the information collection device is an information collection module, the storage device is a memory, and the processing device is a processing module.

[0114] The information acquisition module acquires panoramic image data, annotates the panoramic image data to determine object categories, such as vehicle categories, pedestrian categories, and obstacle categories, and uses annotation tools, such as LabelImg, to determine object image positions, such as using a bounding box to annotate the object image position with a bounding box around it, and marking its category as "vehicle", storing the object category and object image position in a memory, and performing subsequent processing based on the object category and object image position.

[0115] Step S22, determining an object detection model training set, an object detection model verification set, and an object detection model test set based on the panoramic image data, the object category, and the object image position;

[0116] It should be noted that the object detection model training set reflects the characteristics of the data set used to train the object detection model, the object detection model verification set reflects the characteristics of the data set used for model verification, and the object detection model test set reflects the characteristics of the data set used to finally test the model performance.

[0117] It can be understood that the object detection model training set is the basis of model learning and provides basic samples of objects that the model needs to recognize. The object detection model verification set plays a monitoring role in the training process, so as to understand the learning progress of the model and adjust the model parameters accordingly. The object detection model test set is the final evaluation of the model performance, so as to ensure the stability and reliability of the model, and more realistically predict the performance of the model in practical applications, so as to adjust the model in time to ensure that the model does not over-adapt to the training data and can maintain good performance on new data.

[0118] For ease of understanding, an object detection model training set, an object detection model verification set, and an object detection model test set are taken as an example for explanation, wherein the information collection device is an information collection module, the storage device is a memory, and the processing device is a processing module.

[0119] The information acquisition module acquires panoramic image data, annotates the panoramic image data to determine object categories, such as vehicle categories, pedestrian categories, and obstacle categories, and uses annotation tools, such as LabelImg, to determine the object image position, such as using a bounding box to annotate the object image position with a bounding box around it, and marking its category as "vehicle", selecting a suitable deep learning model, such as Faster R-CNN, YOLO series, and SSD, dividing the annotated data into an object detection model training set, an object detection model verification set, and an object detection model test set, and performing subsequent processing based on the object detection model training set, the object detection model verification set, and the object detection model test set.

[0120] Step S23, acquiring an object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data;

[0121] It should be noted that the object detection optimization model reflects the characteristics of the optimized machine learning model obtained after adjusting the hyperparameters.

[0122] It can be understood that the object detection optimization model can more accurately identify and locate objects in images, reduce false detections and missed detections, and can adapt to different environmental conditions and image changes.

[0123] For ease of understanding, an example of obtaining an object detection optimization model is used for explanation, wherein the information acquisition device is an information acquisition module, the storage device is a memory, and the processing device is a processing module.

[0124] The information acquisition module obtains panoramic image data, annotates the panoramic image data to determine the object category, such as vehicle category, pedestrian category and obstacle category, and uses annotation tools, such as LabelImg, to determine the object image position, such as using a bounding box to annotate the bounding box around the object image position, and marks its category as "vehicle", selects a suitable deep learning model, such as Faster R-CNN, YOLO series and SSD, divides the annotated data into an object detection model training set, an object detection model verification set and an object detection model test set, and can use the object detection model training set data to train the selected deep learning model. During the training process, the deep learning model learns the relationship between the features in the image and the object category and position. By adjusting the hyperparameters of the deep learning model, such as the learning rate and the number of iterations, the performance of the model is optimized to determine the object detection optimization model. For example, gradually reducing the learning rate can make the model converge more stably in the later stage of training, and subsequent processing is performed based on the object detection optimization model.

[0125] In a feasible implementation, step S23 may include steps C11 to C14:

[0126] Step C11, obtaining an object detection model and a model performance adjustment method, wherein the object detection model includes a region classification detection model, a grid division detection model, and a priori box detection model, and the model performance adjustment method includes a data augmentation operation, an operation to increase model complexity, and an introduction of a regularization operation;

[0127] It should be noted that the object detection model reflects the feature that a suitable deep learning model needs to be selected before model training, and the model performance adjustment method reflects the feature of optimizing when the model performance is not ideal.

[0128] It is understandable that before model training, it is necessary to select a suitable deep learning model, such as Faster R-CNN, YOLO series and SSD. Faster R-CNN is a deep learning model for target detection tasks. It is an end-to-end object detection framework proposed by Microsoft Research. Faster R-CNN first uses a region proposal network to generate candidate regions that may contain objects. Then, it uses a convolutional neural network to extract features and a fully connected layer to classify and regress bounding boxes for these regions. The advantage of Faster R-CNN is its high detection accuracy, but its calculation speed is relatively slow because it needs to extract features and classify each candidate region separately. YOLO is a very popular real-time object detection system. Its design concept is to solve the object detection task as a regression problem. YOLO divides the image into multiple grids, and each grid is responsible for predicting the bounding box and category probability of the object in the region. YOLOv1 is the initial version, followed by YOLOv2 (Darknet), YOLOv3 and YOLOv4. Each new version has improved performance and speed. The advantage of YOLO is its fast speed and suitability for real-time object detection, but its accuracy is slightly lower than FasterR-CNN. SSD is a single-shot detection model that detects objects at different scales, which enables it to detect objects of different sizes. SSD uses a default box to predict the bounding box and category probability of an object. It makes predictions on feature maps at different levels, so that objects of different scales can be captured. The advantages of SSD are fast speed and high accuracy. It has a fast detection speed while maintaining high detection accuracy.

[0129] In addition, it should be noted that if the model performance is not ideal, you can add data augmentation operations such as flipping, rotation, scaling, and color transformation to increase the diversity of the data, or use a more complex model structure or a larger model scale, or introduce regularization techniques such as L1 and L2 regularization to prevent overfitting.

[0130] Step C12, inputting the object detection model training set into the object detection model for training, adjusting the object detection model hyperparameters to determine an improved object detection model, wherein the object detection model hyperparameters include a learning rate and a number of iterations;

[0131] It should be noted that the object detection model hyperparameters reflect the characteristics of parameters that need to be adjusted during the training of the object detection model.

[0132] It is understandable that the object detection model hyperparameters are not learned directly from the data, but are determined by manual selection or using certain strategies. The learning rate determines the step size of the model weight update, and the number of iterations determines the number of updates during the training process. Reasonable settings of the learning rate and number of iterations can speed up the convergence of the model and reduce the training time.

[0133] Step C13, determining evaluation indicators based on the object detection model verification set and the object detection improved model, wherein the evaluation indicators include accuracy, recall rate and average precision mean;

[0134] It can be understood that the trained model is evaluated using the object detection model validation set data to calculate evaluation indicators, which include accuracy, recall and average precision. The accuracy reflects the characteristics of the overall predictive ability of the model, and is used to measure the proportion of samples correctly predicted by the model to the total number of samples. The recall reflects the characteristics of the model's ability to capture positive samples, and is used to measure the proportion of positive samples successfully identified by the model to all actual positive samples. The average precision is a comprehensive indicator that measures the average performance of the model in each category.

[0135] In addition, it should be noted that based on the evaluation results, the deficiencies of the model should be analyzed and whether further adjustments to the model structure or training parameters are needed should be considered.

[0136] Step C14: determining an object detection optimization model based on the evaluation index and the model performance adjustment method.

[0137] It is understandable that the object detection optimization model is deployed in the vehicle-mounted application system to perform real-time image object detection on the collected images. During the deployment process, the computational efficiency and real-time requirements of the model need to be considered, and the model can be compressed and optimized.

[0138] Step S24, inputting the panoramic image data into the object detection optimization model to perform image object detection to determine and annotate the panoramic image data;

[0139] It should be noted that the annotated panoramic image data reflects the characteristics of the annotated image data set.

[0140] It is understandable that by annotating panoramic image data, the detection accuracy of the model can be improved, enabling the model to learn more precise object features and location information, thereby improving the accuracy of detection. By training on a variety of annotated panoramic image data, it can better adapt to different road environments and conditions, and reduce the time for model self-exploration and learning, thereby speeding up data processing and analysis.

[0141] For ease of understanding, the following description is given by taking the determination of annotated panoramic image data as an example, wherein the information acquisition device is an information acquisition module, the storage device is a memory, and the processing device is a processing module.

[0142] The information acquisition module acquires panoramic image data, annotates the panoramic image data to determine object categories, such as vehicle categories, pedestrian categories, and obstacle categories, and uses annotation tools, such as LabelImg, to determine the object image position, such as using a bounding box to annotate the object image position with a bounding box around it, and marking its category as "vehicle", selecting a suitable deep learning model, such as Faster R-CNN, YOLO series, and SSD, inputting the panoramic image data into the object detection optimization model for image object detection to determine the annotated panoramic image data, and performing subsequent processing based on the annotated panoramic image data.

[0143] Step S25, applying the image semantic segmentation algorithm to perform regional segmentation on the annotated panoramic image data based on the object detection model training set, the object detection model test set and the object detection optimization model to determine road object information, wherein the road object information includes lane line information and traffic sign information;

[0144] It can be understood that by using object detection optimization models and image semantic segmentation algorithms to perform regional segmentation on panoramic image data to determine road object information, objects on the road can be more accurately identified and located, image data can be quickly processed and analyzed, and it can adapt to different road conditions and environmental changes, such as different weather, lighting conditions and road types, to ensure that reliable warning information can be provided in various situations.

[0145] For ease of understanding, the determination of road object information is taken as an example for explanation, wherein the information collection device is an information collection module, the storage device is a memory, and the processing device is a processing module.

[0146] The information acquisition module obtains an object detection model training set, an object detection model test set and an object detection optimization model, applies the image semantic segmentation algorithm to perform regional segmentation on the annotated panoramic image data based on the object detection model training set, the object detection model test set and the object detection optimization model to determine road object information, and performs subsequent processing based on the road object information.

[0147] In a feasible implementation, step S25 may include steps D11 to D13:

[0148] Step D11, training the object detection optimization model based on the image semantic segmentation algorithm and the object detection model training set to obtain model road information;

[0149] It is understandable that the object detection model training set can be used to train the model to obtain model road information, so as to more accurately judge the abnormal state of the road and identify abnormal situations in time. At the same time, the object detection model verification set can be used to monitor the performance of the model.

[0150] Step D12, determining an abnormality index based on the object detection model test set and the object detection optimization model;

[0151] It is understandable that the object detection model test set can be used to evaluate the object detection optimization model, calculate the accuracy, recall rate, and IoU indicators, and thus determine the abnormal indicators.

[0152] Step D13: Visually analyze the object detection optimization model based on the abnormality index to determine the model abnormality segmentation result, and determine the object segmentation optimization model based on the model abnormality segmentation result to obtain road object information.

[0153] It can be understood that by visually analyzing the model based on abnormal indicators, checking the types and patterns of errors in the segmentation results, and making targeted improvements, after sufficient optimization, it can be used to segment and recognize road driving images collected in real time, and prepare to identify data information such as vehicles, lanes and traffic signs.

[0154] Step S26: identifying the anomaly information of the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm.

[0155] It can be understood that the performance of the model can be evaluated based on the object detection model validation set, and the model can be adjusted according to the evaluation results to improve the accuracy of detection, thereby determining the abnormal information of the image, so as to identify unusual situations in the image, such as abnormal speed changes or other potential dangerous factors, improve the accuracy of the warning signal, and adapt to different road environments and conditions, thereby enhancing the adaptability of the system.

[0156] For ease of understanding, the following is explained by taking determining abnormal image information as an example, wherein the information acquisition device is an information acquisition module, the storage device is a memory, and the processing device is a processing module.

[0157] The information acquisition module acquires panoramic image data, such as vehicle image data, pedestrian image data and obstacle image data, including weather conditions, lighting conditions, road types and geographical locations, annotates the panoramic image data to determine object categories, such as vehicle categories, pedestrian categories and obstacle categories, and uses annotation tools, such as LabelImg, to determine the object image position, such as using a bounding box to annotate the bounding box around the object image position and marking its category as "vehicle", determines an object detection model training set, an object detection model verification set and an object detection model test set based on the panoramic image data, the object category and the object image position, acquires an object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data, inputs the panoramic image data into the object detection optimization model to perform image object detection to determine the annotated panoramic image data, identifies the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm to determine image anomaly information, and performs subsequent processing based on the image anomaly information.

[0158] In a feasible implementation, step S26 may include steps E11 to E14:

[0159] Step E11, obtaining target vehicle operating condition characteristics, wherein the target vehicle operating condition characteristics include normal vehicle driving characteristics and abnormal vehicle driving characteristics;

[0160] It should be noted that the target vehicle operating condition characteristics reflect the characteristics of the vehicle's driving state.

[0161] It is understandable that meaningful features are extracted to determine the target vehicle operating condition features, such as extracting the vehicle's sudden braking, sharp turns, and sudden shortening of the distance to the vehicle in front, and standardizing or normalizing the features to make different features comparable.

[0162] Step E12, performing data preprocessing on the annotated panoramic image data to determine cleaned panoramic image data;

[0163] It can be understood that by acquiring the annotated panoramic image data, cleaning and preprocessing the annotated panoramic image data, and removing noise, missing values ​​and abnormal values, the cleaned panoramic image data can be obtained.

[0164] Step E13, determining an abnormal deviation value based on the image abnormality detection algorithm, the cleaning panoramic image data and the target vehicle operating condition characteristics, wherein the abnormal deviation value includes a false alarm rate and a false alarm rate;

[0165] It can be understood that the selected anomaly detection algorithm is trained using the training data. For the isolation forest anomaly detection algorithm, a sufficient number of isolated trees are constructed to improve the accuracy of detection. For the speed deviation statistical anomaly detection algorithm, a suitable statistical threshold is determined. For the model speed distribution anomaly detection algorithm, the model is evaluated and adjusted through the validation set, and then the false alarm rate and false negative rate are checked, and the parameters of the model are adjusted according to the results.

[0166] Step E14: adjusting the object detection optimization model based on the object detection model verification set and the abnormal deviation value to obtain image abnormality information.

[0167] It is understandable that by using the object detection model validation set to test the optimized object detection model, specific data about the model performance can be obtained, including abnormal deviation values ​​such as false alarm rate and false negative rate, which characterize the errors that may occur in the model in actual applications, so as to adjust the model parameters, reduce false alarms and false negatives, and ensure more accurate and reliable identification of road anomalies, effectively improving road driving safety.

[0168] The road anomaly prediction and warning method proposed in this embodiment annotates the panoramic image data to determine the object category and the object image position; determines the object detection model training set, the object detection model verification set and the object detection model test set based on the panoramic image data, the object category and the object image position; obtains the object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data; inputs the panoramic image data into the object detection optimization model to perform image object detection to determine the annotated panoramic image data; applies the image semantic segmentation algorithm to perform regional segmentation on the annotated panoramic image data based on the object detection model training set, the object detection model test set and the object detection optimization model to determine road object information, wherein the road object information includes lane line information and traffic sign information; identifies the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm to determine image anomaly information. The technical problem of how to more accurately identify road anomalies and provide road warnings is solved. Compared with the existing technology, the present application identifies object categories and locations through precise annotation of panoramic image data, thereby constructing training sets, verification sets and test sets, and then training and optimizing object detection models. The panoramic image data is input into the optimization model to perform accurate image object detection and semantic segmentation, identify key road information such as lane lines and traffic signs, and at the same time, combine with the image anomaly detection algorithm to identify abnormal situations in the image and issue warnings to the driver in a timely manner, effectively improving the safety of road driving and the accuracy of the warning system.

[0169] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the road abnormality prediction and warning method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0170] This application also provides a road abnormality prediction and warning device, please refer to Figure 3 , the road abnormality prediction and warning device comprises:

[0171] An acquisition module 10 is used to acquire panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm;

[0172] A processing module 20, configured to determine road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm;

[0173] The execution module 30 is used to predict the motion trajectory based on the panoramic image data, the road object information and the image abnormality information to generate a corresponding warning signal for road warning.

[0174] The acquisition module 10 is also used to acquire road image data, foreign vehicle image data, pedestrian image data, obstacle image data, image segmentation algorithm, loss function, isolation forest anomaly detection algorithm, speed deviation statistical anomaly detection algorithm and model speed distribution anomaly detection algorithm;

[0175] Determine panoramic image data based on the road image data, the other vehicle image data, the pedestrian image data and the obstacle image data;

[0176] Determine an image semantic segmentation algorithm based on the image segmentation algorithm and the loss function;

[0177] An image anomaly detection algorithm is determined based on the isolation forest anomaly detection algorithm, the speed deviation statistical anomaly detection algorithm and the model speed distribution anomaly detection algorithm.

[0178] The processing module 20 is further used to mark the panoramic image data to determine the object category and the object image position;

[0179] Determine an object detection model training set, an object detection model verification set, and an object detection model test set based on the panoramic image data, the object category, and the object image position;

[0180] Acquire an object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data;

[0181] Inputting the panoramic image data into the object detection optimization model to perform image object detection to determine and annotate the panoramic image data;

[0182] Based on the object detection model training set, the object detection model test set and the object detection optimization model, the image semantic segmentation algorithm is applied to perform regional segmentation on the annotated panoramic image data to determine road object information, where the road object information includes lane line information and traffic sign information;

[0183] The anomaly information of the image is determined by identifying the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm.

[0184] The processing module 20 is further used to obtain an object detection model and a model performance adjustment method, wherein the object detection model includes a region classification detection model, a grid division detection model and a priori box detection model, and the model performance adjustment method includes a data augmentation operation, an operation to increase model complexity, and an introduction of a regularization operation;

[0185] Inputting the object detection model training set into the object detection model for training, adjusting the object detection model hyperparameters to determine an improved object detection model, wherein the object detection model hyperparameters include a learning rate and a number of iterations;

[0186] Determining evaluation indicators based on the object detection model verification set and the object detection improved model, wherein the evaluation indicators include accuracy, recall rate and average precision mean;

[0187] An object detection optimization model is determined based on the evaluation index and the model performance adjustment method.

[0188] The processing module 20 is further used to train the object detection optimization model based on the image semantic segmentation algorithm and the object detection model training set to obtain model road information;

[0189] Determining an abnormality indicator based on the object detection model test set and the object detection optimization model;

[0190] Based on the abnormal index, the object detection optimization model is visualized and analyzed to determine the model abnormal segmentation result, and based on the model abnormal segmentation result, the object segmentation optimization model is determined to obtain the road object information.

[0191] The processing module 20 is further used to obtain the target vehicle operating condition characteristics, wherein the target vehicle operating condition characteristics include the vehicle normal driving characteristics and the vehicle abnormal driving characteristics;

[0192] Performing data preprocessing on the annotated panoramic image data to determine cleaned panoramic image data;

[0193] Determine an abnormal deviation value based on the image abnormality detection algorithm, the cleaning panoramic image data and the target vehicle working condition characteristics, wherein the abnormal deviation value includes a false alarm rate and a false alarm rate;

[0194] The object detection optimization model is adjusted based on the object detection model verification set and the abnormal deviation value to obtain image abnormality information.

[0195] The execution module 30 is also used to obtain target sensor fusion data and target tracking algorithm;

[0196] Selecting a prediction model based on the target sensor fusion data, the target tracking algorithm and the panoramic image data to determine a predicted motion trajectory state;

[0197] Determine a road crisis state based on the road object information, the image abnormality information and the predicted motion trajectory state;

[0198] A warning signal is generated based on the road crisis state to perform road warning, and the warning signal includes an audible alarm and a visual alarm.

[0199] The road anomaly prediction and warning device provided by the present application adopts the road anomaly prediction and warning method in the above embodiment, which can solve the technical problem of how to provide a panoramic perspective and efficiently and accurately provide real-time warning of road anomalies. Compared with the prior art, the beneficial effects of the road anomaly prediction and warning device provided by the present application are the same as the beneficial effects of the road anomaly prediction and warning method provided by the above embodiment, and the other technical features of the road anomaly prediction and warning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0200] The present application provides a road abnormality prediction and warning device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the road abnormality prediction and warning method in the above-mentioned embodiment one.

[0201] Reference below Figure 4, which shows a schematic diagram of the structure of a road abnormality prediction and warning device suitable for implementing the embodiment of the present application. The road abnormality prediction and warning device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The road abnormality prediction and warning device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0202] like Figure 4 As shown, the road abnormality prediction and warning device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the road abnormality prediction and warning device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the road abnormality prediction and warning device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a road abnormality prediction and warning device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0203] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0204] The road anomaly prediction and warning device provided by the present application adopts the road anomaly prediction and warning method in the above embodiment, which can solve the technical problem of how to provide a panoramic perspective and efficiently and accurately provide real-time warning of road anomalies. Compared with the prior art, the beneficial effects of the road anomaly prediction and warning device provided by the present application are the same as the beneficial effects of the road anomaly prediction and warning method provided by the above embodiment, and the other technical features in the road anomaly prediction and warning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0205] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0206] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0207] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the road abnormality prediction and warning method in the above-mentioned embodiment.

[0208] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0209] The computer-readable storage medium may be included in the road abnormality prediction and warning device; or may exist independently without being assembled into the road abnormality prediction and warning device.

[0210] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the road abnormality prediction and warning device, the road abnormality prediction and warning device: obtains panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm; determines road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm; predicts the motion trajectory based on the panoramic image data, the road object information and the image anomaly information to generate a corresponding warning signal for road warning.

[0211] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0212] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0213] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0214] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned road anomaly prediction and early warning method, and can solve the technical problem of how to provide a panoramic perspective and provide real-time early warning of road anomalies efficiently and accurately. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the road anomaly prediction and early warning method provided by the above-mentioned embodiment, and will not be repeated here.

[0215] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A road anomaly prediction and warning method, characterized in that: The method includes: Obtain panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm; Determine road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm; Based on the panoramic image data, the road object information and the image abnormality information, a motion trajectory is predicted to generate a corresponding warning signal for road warning.

2. The method according to claim 1, characterized in that The steps of acquiring panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm include: Obtain road image data, alien vehicle image data, pedestrian image data, obstacle image data, image segmentation algorithm, loss function, isolation forest anomaly detection algorithm, speed deviation statistical anomaly detection algorithm and model speed distribution anomaly detection algorithm; Determine panoramic image data based on the road image data, the other vehicle image data, the pedestrian image data and the obstacle image data; Determine an image semantic segmentation algorithm based on the image segmentation algorithm and the loss function; An image anomaly detection algorithm is determined based on the isolation forest anomaly detection algorithm, the speed deviation statistical anomaly detection algorithm and the model speed distribution anomaly detection algorithm.

3. The method according to claim 1, characterized in that The step of determining road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm comprises: Annotating the panoramic image data to determine the object category and the object image position; Determine an object detection model training set, an object detection model verification set, and an object detection model test set based on the panoramic image data, the object category, and the object image position; Acquire an object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data; Inputting the panoramic image data into the object detection optimization model to perform image object detection to determine and annotate the panoramic image data; Based on the object detection model training set, the object detection model test set and the object detection optimization model, the image semantic segmentation algorithm is applied to perform regional segmentation on the annotated panoramic image data to determine road object information, where the road object information includes lane line information and traffic sign information; The anomaly information of the image is determined by identifying the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm.

4. The method according to claim 3, characterized in that The step of acquiring the object detection optimization model based on the object detection model training set, the object detection model verification set and the panoramic image data comprises: Obtaining an object detection model and a model performance adjustment method, wherein the object detection model includes a region classification detection model, a grid division detection model, and a priori box detection model, and the model performance adjustment method includes a data augmentation operation, an operation to increase model complexity, and an introduction of a regularization operation; Inputting the object detection model training set into the object detection model for training, adjusting the object detection model hyperparameters to determine an improved object detection model, wherein the object detection model hyperparameters include a learning rate and a number of iterations; Determining evaluation indicators based on the object detection model verification set and the object detection improved model, wherein the evaluation indicators include accuracy, recall rate and average precision mean; An object detection optimization model is determined based on the evaluation index and the model performance adjustment method.

5. The method according to claim 3, characterized in that The step of applying the image semantic segmentation algorithm to perform regional segmentation on the annotated panoramic image data based on the object detection model training set, the object detection model test set and the object detection optimization model to determine the road object information comprises: Based on the image semantic segmentation algorithm and the object detection model training set, the object detection optimization model is trained to obtain model road information; Determining an abnormality indicator based on the object detection model test set and the object detection optimization model; Based on the abnormal index, the object detection optimization model is visualized and analyzed to determine the model abnormal segmentation result, and based on the model abnormal segmentation result, the object segmentation optimization model is determined to obtain the road object information.

6. The method according to claim 3, characterized in that The step of identifying and determining image anomaly information of the annotated panoramic image data based on the object detection model verification set, the object detection optimization model and the image anomaly detection algorithm comprises: Acquiring target vehicle operating condition characteristics, wherein the target vehicle operating condition characteristics include normal vehicle driving characteristics and abnormal vehicle driving characteristics; Performing data preprocessing on the annotated panoramic image data to determine cleaned panoramic image data; Determine an abnormal deviation value based on the image abnormality detection algorithm, the cleaning panoramic image data and the target vehicle working condition characteristics, wherein the abnormal deviation value includes a false alarm rate and a false alarm rate; The object detection optimization model is adjusted based on the object detection model verification set and the abnormal deviation value to obtain image abnormality information.

7. The method according to claim 1, characterized in that The step of predicting the motion trajectory based on the panoramic image data, the road object information and the image abnormality information to generate a corresponding warning signal for road warning includes: Obtain target sensor fusion data and target tracking algorithm; Selecting a prediction model based on the target sensor fusion data, the target tracking algorithm and the panoramic image data to determine a predicted motion trajectory state; Determine a road crisis state based on the road object information, the image abnormality information and the predicted motion trajectory state; A warning signal is generated based on the road crisis state to perform road warning, and the warning signal includes an audible alarm and a visual alarm.

8. A road abnormality prediction and warning device, characterized in that: The device comprises: An acquisition module, used to acquire panoramic image data, image semantic segmentation algorithm and image anomaly detection algorithm; A processing module, configured to determine road object information and image anomaly information based on the panoramic image data, the image semantic segmentation algorithm and the image anomaly detection algorithm; An execution module is used to predict the motion trajectory based on the panoramic image data, the road object information and the image abnormality information to generate a corresponding warning signal for road warning.

9. A road abnormality prediction and warning device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the road abnormality prediction and warning method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the road abnormality prediction and warning method according to any one of claims 1 to 7 are implemented.

Images