Road abnormity identification method and system applied to safe driving and storage medium

Abstract

The invention provides a road anomaly recognition method and system applied to safe driving and a storage medium, and the method comprises the steps: S1, carrying out the data training of a detection network model and a classification network model through deep learning, and enabling the detection network model to be used for recognizing the anomaly position and class of a current picture, the classification network model is used for identifying an abnormal hazard degree; s2, deploying the detection network model and the classification network model trained in the step S1 at a vehicle end, and carrying out vehicle end off-line anomaly detection and manual review so as to fuse abnormal information identified by the models and abnormal information recorded by a driver into a map; and S3, sharing the map fused in the step S2. According to the invention, most of abnormal conditions influencing driving on the road can be detected and identified, abnormal positions can be accurately positioned, and operation suggestions can be given; the system can be really deployed in a production environment through the combination of a deep learning technology and manual actual experience; and large-area urban road abnormal distribution information is obtained at low cost.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving safety technology, and in particular to a road anomaly recognition method and system and a storage medium for safe driving. Background Art

[0003] At present, in order to identify road anomalies, existing technology 1 ("Road Anomaly Intelligent Identification Model Training, Road Anomaly Identification Method and System", Shandong University of Technology, Chen Jianda) trains the model based on the correlation of the impact of abnormal characteristics on vehicle driving position characteristics, and can identify the specific situation of road anomalies based on the detailed difference representation of vehicle density distribution information and vehicle driving direction information.

[0004] This prior art has the following disadvantages: it only targets relatively obvious road anomalies, such as road congestion, road construction, traffic accidents, etc.; it uses vehicle density distribution to identify anomalies, the feature correlation is not high, and it is easily limited by the different distribution of cities and roads; the identification and positioning range is large.

[0005] Prior art 2 (“Road abnormality detection device, road abnormality detection method and computer-readable medium”, Toyota Motor Corporation, Kazuaki Takemura, Masato Matsushita, Tae Sugimura, Hideo Hasegawa, Tomomitsu Umeki) determines whether there is an abnormality in the road facilities by comparing all road facilities included in the current road image with pre-prepared correctness information.

[0006] The second prior art has the following disadvantages: it can only recognize larger scene changes; it only applies traditional technology, and the recognition rate is limited by many conditions, among which it is severely limited by the road scene and the scene of correct information during shooting; and the positioning range is large.

[0007] Existing technology three ("Sustainable road anomaly detection method and system based on multi-source sensor fusion", Zhejiang University, Hu Jun, Xin Hanyu, Wu Chao, Hu Huan, Wang Gaoang, Li Dehong) inputs the preprocessed data into the classification network to predict the abnormal threshold, and then integrates the data according to the threshold and inputs it into different networks to obtain the single-vehicle abnormal road surface detection result. Finally, the single-vehicle abnormal road surface detection result is fused with multi-vehicle perception results based on the OPTICS algorithm to obtain the abnormal road surface detection result.

[0008] The third existing technology has the following disadvantages: the number of recognition categories is small, only road cracks and potholes; the data preprocessing and model recognition process is cumbersome and difficult to apply in an actual production environment; the recognition and positioning range is large, and it requires the coordination and integration of multiple vehicles, which is difficult to apply to a single vehicle. Summary of the invention

[0009] The present invention proposes a corresponding solution to the driving safety hazards caused by current road anomalies:

[0010] (1) Use deep learning technology to detect various abnormalities on the current road that affect driving, including road damage (potholes, bulges, damage, deep holes, etc.), obstacles (fixed and mobile obstacles), abnormal manhole covers, abnormal road facilities (speed bumps, artificially added facilities, etc.), basically covering all abnormalities that affect driving;

[0011] (2) Using deep learning technology to classify the network, the network can identify the severity of road anomalies and directly guide the driverless vehicle to slow down or avoid obstacles;

[0012] (3) The process of offline detection, manual review, and adding map information is combined with machine intelligence and artificial intelligence to greatly improve the credibility of the detection results. It can also save the time-consuming vehicle-side reasoning and achieve real-time application of detection results, ultimately realizing the applicability of the production environment.

[0013] (4) The coordinate system conversion technology is used to convert the image 2D information into 3D position, which can accurately locate the position of the anomaly and achieve centimeter-level error;

[0014] (5) By utilizing the networking function of map information, vehicle-to-vehicle interconnection can be achieved. If one vehicle passes an abnormality, all vehicles will have the distribution information of the road abnormality.

[0015] To achieve the above-mentioned purpose, the technical solution of the present invention provides a road anomaly identification method for safe driving, which includes the following steps: S1: using deep learning to perform data training of a detection network model and a classification network model, the detection network model is used to identify the abnormal position and category of the current image, and the classification network model is used to identify the degree of harm of the anomaly; S2: deploying the detection network model and the classification network model trained in step S1 on the vehicle side, and performing offline vehicle-side anomaly detection and manual review to merge the anomaly information identified by the model with the anomaly information recorded by the driver into a map; S3: sharing the fused map in step S2.

[0016] Furthermore, step S1 specifically includes: S11: using the on-board camera to collect road image data and upload it to the server; S12: screening and labeling the collected data to create training data sets and test sets; S13: using deep learning to train the detection network model data to identify the abnormal location and category of the current image; S14: using deep learning to train the classification network model data to identify the degree of harm caused by the anomaly; S15: selecting the detection network model and classification network model that perform best on the test set as the vehicle-side deployment model.

[0017] Furthermore, step S2 specifically includes: S21: deploying the detection network model and the classification network model on the vehicle side; S22: recording abnormal information through the driver during the operation of the vehicle; S23: after the normal operation of the vehicle is completed every day, obtaining pictures of abnormal roads through the deployed model and visualizing the pictures; S24: manually reviewing the visualized pictures and the abnormal information recorded by the driver, comparing the two, fusing the abnormalities correctly identified by the model and the abnormalities recorded by the driver but not recognized and missed by the model, and removing the abnormalities redundantly identified by the model; S25: converting the 2D position information of the abnormality obtained by fusion in step S24 into a 3D world coordinate system; S26: fusing the abnormality category, the vehicle recommended operation and the abnormal 3D coordinates into the map.

[0018] Furthermore, in step S22, the vehicle collects real-time image data and position information from the on-board camera during operation, and the driver performs corresponding operations after observing road abnormalities, and records the camera image and vehicle position at that time through buttons, and saves them in another fixed memory.

[0019] Furthermore, in step S23, after the normal daily operation of the vehicle is completed, the deployed model infers the pictures collected during driving to obtain whether there is any abnormality in the current picture, the category and 2D position of the abnormality, and vehicle operation suggestions, and visualizes the abnormal pictures and then saves them in a specific memory.

[0020] Furthermore, step S2 further includes the following steps after step S24: S241: collecting road anomaly pictures recorded by the driver but not recognized by the model, for subsequent model iteration and updating.

[0021] Furthermore, in step S25, the 2D position information is converted into a 3D world coordinate system according to the following steps:

[0022] S251: The formula for converting the coordinates of the 3D world coordinate system into the 2D pixel coordinate system is shown in (1). The formula for converting the pixel coordinate system into the world coordinate system is obtained through matrix transformation (2).

[0023]

[0024] In the above formula, Zc is the depth in the camera coordinate system, K is the camera intrinsic parameter matrix, R is the camera extrinsic parameter rotation matrix, and T is the camera extrinsic parameter translation matrix. is the pixel matrix, is the world coordinate matrix corresponding to the object, where K, R, T, is a known value;

[0025] S252: in actual calculation, the actual height value of the object, i.e., Zw, is obtained by estimation, and then the Zc value is obtained by calculation of the third dimension;

[0026] S253: Obtain the 3D world coordinates of the object using formula (2).

[0027] Furthermore, step S3 specifically includes: S31: sharing the map with road anomaly information added through vehicle-side interconnection; S32: reading the map information when all vehicles pass by an abnormal road passed by a certain vehicle to obtain the abnormal category and location of the current road and the recommended vehicle operation.

[0028] The technical solution of the present invention also provides a road anomaly recognition system for safe driving, which includes the following modules: a model training module, which is used to use deep learning to perform data training on a detection network model and a classification network model, the detection network model is used to identify the abnormal position and category of the current image, and the classification network model is used to identify the degree of harm of the anomaly; an offline anomaly detection and manual review module, which is used to deploy the detection network model and the classification network model obtained by the model training module on the vehicle side, and perform offline anomaly detection and manual review on the vehicle side, so as to merge the anomaly information recognized by the model with the anomaly information recorded by the driver into a map; a map sharing module, which is used to share the map merged by the offline anomaly detection and manual review module.

[0029] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program, which is used to implement the method as described above when the computer program is executed by one or more processors. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is an overall flow chart of the road anomaly identification method for safe driving of the present invention;

[0031] Figure 2 It is a flowchart of the deep learning recognition model training of the present invention;

[0032] Figure 3 It is a flow chart of vehicle-side offline abnormality detection and manual review of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these embodiments.

[0034] The present invention is mainly based on deep learning technology and provides a solution for detecting abnormal information of the current road and providing reasonable operations.

[0035] In an embodiment of the present invention, a road anomaly identification method for safe driving includes the following steps: S1: using deep learning to perform data training on a detection network model and a classification network model, the detection network model is used to identify the abnormal position and category of the current image, and the classification network model is used to identify the degree of harm of the anomaly; S2: deploying the detection network model and the classification network model trained in step S1 on the vehicle side, and performing offline vehicle-side anomaly detection and manual review to merge the anomaly information identified by the model with the anomaly information recorded by the driver into a map; S3: sharing the fused map in step S2.

[0036] Furthermore, step S1 specifically includes: S11: using the on-board camera to collect road image data and upload it to the server; S12: screening and labeling the collected data to create training data sets and test sets; S13: using deep learning to train the detection network model data to identify the abnormal location and category of the current image; S14: using deep learning to train the classification network model data to identify the degree of harm caused by the anomaly; S15: selecting the detection network model and classification network model that perform best on the test set as the vehicle-side deployment model.

[0037] Furthermore, step S2 specifically includes: S21: deploying the detection network model and the classification network model on the vehicle side; S22: recording abnormal information through the driver during the operation of the vehicle; S23: after the normal operation of the vehicle is completed every day, obtaining pictures of abnormal roads through the deployed model and visualizing the pictures; S24: manually reviewing the visualized pictures and the abnormal information recorded by the driver, comparing the two, fusing the abnormalities correctly identified by the model and the abnormalities recorded by the driver but not recognized and missed by the model, and removing the abnormalities redundantly identified by the model; S25: converting the 2D position information of the abnormality obtained by fusion in step S24 into a 3D world coordinate system; S26: fusing the abnormality category, the vehicle recommended operation and the abnormal 3D coordinates into the map.

[0038] Furthermore, in step S22, the vehicle collects real-time image data and position information from the on-board camera during operation, and the driver performs corresponding operations after observing road abnormalities, and records the camera image and vehicle position at that time through buttons, and saves them in another fixed memory.

[0039] Furthermore, in step S23, after the normal daily operation of the vehicle is completed, the deployed model infers the pictures collected during driving to obtain whether there is any abnormality in the current picture, the category and 2D position of the abnormality, and vehicle operation suggestions, and visualizes the abnormal pictures and then saves them in a specific memory.

[0040] Furthermore, step S2 further includes the following steps after step S24: S241: collecting road anomaly pictures recorded by the driver but not recognized by the model, for subsequent model iteration and updating.

[0041] Furthermore, in step S25, the 2D position information is converted into a 3D world coordinate system according to the following steps:

[0042] S251: The formula for converting the coordinates of the 3D world coordinate system into the 2D pixel coordinate system is shown in (1). The formula for converting the pixel coordinate system into the world coordinate system is obtained through matrix transformation (2).

[0043]

[0044] In the above formula, Zc is the depth in the camera coordinate system, K is the camera intrinsic parameter matrix, R is the camera extrinsic parameter rotation matrix, and T is the camera extrinsic parameter translation matrix. is the pixel matrix, is the world coordinate matrix corresponding to the object, where K, R, T, is a known value;

[0045] S252: in actual calculation, the actual height value of the object, i.e., Zw, is obtained by estimation, and then the Zc value is obtained by calculation of the third dimension;

[0046] S253: Obtain the 3D world coordinates of the object using formula (2).

[0047] Furthermore, step S3 specifically includes: S31: sharing the map with road anomaly information added through vehicle-side interconnection; S32: reading the map information when all vehicles pass by an abnormal road passed by a certain vehicle to obtain the abnormal category and location of the current road and the recommended vehicle operation.

[0048] In another embodiment of the present invention, a road anomaly recognition system for safe driving includes the following modules: a model training module, which is used to use deep learning to perform data training on a detection network model and a classification network model, the detection network model is used to identify the abnormal position and category of the current image, and the classification network model is used to identify the degree of hazard of the anomaly; an offline anomaly detection and manual review module, which is used to deploy the detection network model and the classification network model obtained by the model training module on the vehicle side, and perform offline anomaly detection and manual review on the vehicle side, so as to integrate the anomaly information recognized by the model with the anomaly information recorded by the driver into a map; a map sharing module, which is used to share the map integrated by the offline anomaly detection and manual review module.

[0049] In other embodiments of the present invention, a computer-readable storage medium containing a computer program is used to implement the method described above when the computer program is executed by one or more processors. Specific embodiments

[0051] See also Figure 1-Figure 3 The specific implementation steps of the present invention are as follows:

[0052] Part 1: Deep Learning Recognition Model Training

[0053] A. Collect various road image data using on-board cameras and upload them to the server;

[0054] B. Screen and label the collected data to create training data sets and test sets;

[0055] C. Use deep learning technology to detect the network for data training and accurately identify the abnormal location and category of the current image;

[0056] D. Use deep learning technology to classify the network for abnormal data training and accurately identify the degree of harm caused by abnormalities;

[0057] E. Select the detection network model and classification network model that perform best on the test set as the vehicle-side deployment model.

[0058] Part 2: Offline detection anomalies and manual review on the vehicle side

[0059] F. Deploy the above model on the vehicle side;

[0060] G. During the operation of the vehicle, the real-time image data and location information of the on-board camera are collected. When the driver observes road abnormalities, he / she performs corresponding operations and records the camera image and vehicle location at that time by pressing buttons, and saves them in another fixed memory;

[0061] H. After the normal daily operation of the vehicle is completed, the deployed model infers the images collected during driving to obtain information such as whether the current image is abnormal, the category and 2D location of the abnormality, and vehicle operation suggestions (obstacle avoidance or deceleration). It also visualizes the abnormal images and saves them in a specific storage;

[0062] I. Manually review the visual images and the anomaly information recorded by the driver, compare the two, merge the anomalies correctly identified by the model and the anomalies recorded by the driver but not identified by the model, and remove the anomalies redundantly identified by the model;

[0063] J. Collect road anomaly pictures recorded by drivers but not identified by the model for subsequent model iteration updates;

[0064] K. The abnormal 2D position information obtained by the fusion of the above steps is converted into a 3D world coordinate system through the coordinate system conversion technology. The specific operations are as follows:

[0065] a. The 3D world coordinate system is converted into the 2D pixel coordinate system as shown in formula (1), and the pixel coordinate system is converted into the world coordinate system by matrix transformation (2);

[0066]

[0067] In the above formula, Zc is the depth in the camera coordinate system, K is the camera intrinsic parameter matrix, R is the camera extrinsic parameter rotation matrix, and T is the camera extrinsic parameter translation matrix. is the pixel matrix, is the world coordinate matrix corresponding to the object, where K, R, T, is a known value;

[0068] b. In actual calculation, the actual height value of the object, i.e., Zw, can be estimated, and then the Zc value can be obtained by calculating the third dimension;

[0069] c. The 3D world coordinates of the object can be obtained through formula (2).

[0070] L. Integrate the anomaly category, vehicle recommended operation, and anomaly 3D coordinates obtained in the above steps into the map;

[0071] Part 3: Map Information Sharing

[0072] M. Share the map with road anomaly information on the vehicle side;

[0073] N. When all vehicles pass by an abnormal road that a certain vehicle has passed, read the map information to obtain the abnormal category and location of the current road and the recommended vehicle operation.

[0074] The key points of the present invention are:

[0075] (1) Use deep learning technology to learn the category and location information of road anomalies and learn the vehicle operation recommendations that should be taken;

[0076] (2) Combine offline detection and manual review to integrate all road anomaly information;

[0077] (3) Convert the 2D information detected by the model into 3D information and save it for vehicle-side interconnection.

[0078] Beneficial technical effects of the present invention:

[0079] The deep learning model trained by the present invention has a wide recognition range and can detect and identify most of the abnormal conditions on the road that affect driving;

[0080] The present invention can accurately locate abnormal positions and directly give operational suggestions, solving the problems of safe driving and route planning for unmanned driving;

[0081] The present invention combines deep learning technology with actual human experience to solve the problem of false detection and missed detection caused by deep learning technology, and can be truly deployed in a production environment;

[0082] The present invention realizes vehicle-side interconnection and can obtain abnormal distribution information of urban roads over a large area at a very low cost.

[0083] Those skilled in the art can understand that the conversion method of the present invention for converting a 2D pixel coordinate system into a 3D world coordinate system can be replaced by other conversion methods with the same function; finally, the carrier map of the abnormal information can be replaced by other methods, such as text files, etc.; the category of the deep learning detection network can also be replaced by other similar categories.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A road anomaly recognition method for safe driving, characterized in that: The steps include: S1: Use deep learning to train data for the detection network model and the classification network model. The detection network model is used to identify the abnormal location and category of the current image, and the classification network model is used to identify the degree of harm caused by the abnormality. S2: deploy the detection network model and classification network model trained in step S1 on the vehicle side, and perform off-line abnormality detection and manual review on the vehicle side to integrate the abnormal information identified by the model with the abnormal information recorded by the driver into the map; S3: Share the map fused in step S2.

2. The method according to claim 1, characterized in that Step S1 specifically includes: S11: Collect road image data using the vehicle-mounted camera and upload it to the server; S12: Screen and label the collected data to create training data sets and test sets; S13: Use deep learning to train the detection network model data to identify the abnormal location and category of the current image; S14: Use deep learning to train classification network model data to identify the degree of harm caused by anomalies; S15: Select the detection network model and classification network model that perform best on the test set as the vehicle-side deployment model.

3. The method according to claim 2, characterized in that Step S2 specifically includes: S21: deploy the detection network model and the classification network model on the vehicle side; S22: During the operation of the vehicle, the driver records abnormal information; S23: After the normal operation of the vehicle is completed every day, the pictures of abnormal roads are obtained through the deployed model and the pictures are visualized; S24: Manually review the visual images and the abnormal information recorded by the driver, compare the two, merge the abnormalities correctly identified by the model and the abnormalities recorded by the driver but not identified by the model, and remove the abnormalities redundantly identified by the model; S25: converting the abnormal 2D position information obtained by fusion in step S24 into a 3D world coordinate system; S26: Integrate the anomaly category, vehicle recommended operation, and anomaly 3D coordinates into the map.

4. The method according to claim 3, characterized in that In step S22, the vehicle collects real-time image data and location information from the on-board camera during operation, and the driver performs corresponding operations after observing road abnormalities, and records the camera image and vehicle position at the time by pressing buttons, and saves them in another fixed memory.

5. The method according to claim 4, characterized in that In step S23, after the normal daily operation of the vehicle is completed, the deployed model infers the pictures collected during driving to determine whether there is any abnormality in the current picture, the category and 2D location of the abnormality, and vehicle operation suggestions, and visualizes the abnormal pictures and saves them in a specific memory.

6. The method according to claim 3, characterized in that Step S2 further includes the following steps after step S24: S241: Collect road anomaly pictures recorded by the driver but not recognized by the model, for subsequent model iteration and update.

7. The method according to claim 3, characterized in that In step S25, the 2D position information is converted into a 3D world coordinate system according to the following steps: S251: The formula for converting the coordinates of the 3D world coordinate system into the 2D pixel coordinate system is shown in (1). The formula for converting the pixel coordinate system into the world coordinate system is obtained through matrix transformation (2). In the above formula, Zc is the depth in the camera coordinate system, K is the camera intrinsic parameter matrix, R is the camera extrinsic parameter rotation matrix, and T is the camera extrinsic parameter translation matrix. is the pixel matrix, is the world coordinate matrix corresponding to the object, where K, R, T, is a known value; S252: in actual calculation, the actual height value of the object, i.e., Zw, is obtained by estimation, and then the Zc value is obtained by calculation of the third dimension; S253: Obtain the 3D world coordinates of the object using formula (2).

8. The method according to claim 3, characterized in that Step S3 specifically includes: S31: sharing the map with road anomaly information through vehicle-side interconnection; S32: When all vehicles pass by an abnormal road that a certain vehicle has passed, map information is read to obtain the abnormal category and location of the current road and the vehicle recommended operation.

9. A road anomaly recognition system for safe driving, characterized in that: Includes the following modules: Model training module, used to use deep learning to perform data training on detection network model and classification network model. The detection network model is used to identify the abnormal location and category of the current image, and the classification network model is used to identify the degree of harm caused by the abnormality. The offline anomaly detection and manual review module is used to deploy the detection network model and the classification network model obtained by the model training module on the vehicle side, and perform offline anomaly detection and manual review on the vehicle side, so as to integrate the anomaly information identified by the model with the anomaly information recorded by the driver into the map; The map sharing module is used to share the map merged by the offline anomaly detection and manual review modules.

10. A computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by one or more processors, it is used to implement the method according to any one of claims 1 to 8.

Images