Automatic driving system based on 5G Internet of Things

Abstract

The invention relates to an automatic driving system based on 5G Internet of Things, which belongs to the technical field of unmanned driving and comprises an image acquisition device, a cloud server platform and a master control center. The image acquisition device comprises sensor devices such as a laser radar camera and the like and is used for acquiring road images; the cloud server platform is used for processing the image information and carrying out intelligent identification to obtain a road identification result; the master control center performs decision control on the driving behavior of the automatic driving vehicle according to the data processing result; through an image processing technology of a cloud server platform, collected road condition information, especially pothole road surfaces and ice and snow roads, is intelligently identified, and decisions are made for automatic driving according to the road conditions. According to the method, the bumpy road and the speed control on the ice and snow road can be accurately identified, the accurate control of the automatic driving vehicle under the complex road condition is realized, and the identification accuracy of the poor road condition and the unmanned driving control accuracy are improved.

Description

Technical Field

[0001] The present invention relates to an autonomous driving system based on 5G Internet of Things, belonging to the field of driverless technology. Background Art

[0002] The current autonomous driving technology mainly faces the urban road environment, which is characterized by a simple ground environment and generally flat road surfaces. Compared with the urban environment, the rural road environment is usually more complex, such as potholed roads, icy roads, etc.; especially some rural roads even do not have high-precision maps. Whether autonomous driving vehicles can autonomously identify different road surface environments, thus realizing steering planning and speed planning, ensuring that the speed of autonomous driving vehicles changes smoothly under severe road bumps, and then improving the severe bumps of the vehicle body, which poses higher requirements for driverless technology.

[0003] Most of the current driverless technologies are studied with the urban road as the background. Facing the road characteristics such as potholed roads and icy roads in rural roads and even remote mountainous areas, there is no good way to extract the characteristics of road conditions and then make autonomous driving decisions. Summary of the Invention

[0004] The purpose of the present invention is to provide an autonomous driving system based on 5G Internet of Things to solve the deficiencies in the background art.

[0005] To achieve the above purpose, the present invention provides the following technical solutions: An autonomous driving system based on 5G Internet of Things according to the present invention is characterized in that: it includes an image acquisition device, a cloud server platform, a general control center, and an autonomous driving vehicle; The image acquisition device includes a number of lidars and camera devices, which are used to collect images of the road to obtain road image information; The cloud server platform is used to perform real-time processing and analysis on the road image information, conduct intelligent recognition, and obtain a road recognition result; The general control center makes driving decision control for the autonomous driving vehicle according to the road recognition result; The autonomous driving vehicle is used for autonomous driving and executes driving decisions.

[0006] In a preferred embodiment, the acquisition of road information by the image acquisition device is as follows: Because the lidar can collect images within a maximum distance of 100 feet in front, in autonomous driving, the lidar in the acquisition device real-time collects images and videos of the road conditions within 30 meters in front of the autonomous driving vehicle and transmits them back to the cloud server platform. The cloud server platform processes and analyzes the image information to obtain a road recognition result.

[0007] The process of the cloud server platform processing and analyzing image information to obtain road recognition results is as follows: Step 1: Obtain road picture data according to the collected image information, or download road pictures from the Internet such as Baidu and 360 search engines, and classify the road picture data to construct a road category data set; the road categories include flat roads, potholed roads, and icy roads; Step 2: Based on the road category data set, train an image recognition autoencoder model to obtain a road category detection model; Step 3: Input the road data to be detected into the trained road category detection model to obtain a road category detection result.

[0008] In a preferred embodiment, the image recognition autoencoder model includes: an image feature extraction network model and an image feature fusion network model; The image feature extraction network model is composed of several layers of feature extraction encoders, which are used to extract road feature information and reduce the image resolution and the number of features; The image feature fusion network model is composed of several layers of decoders, and each decoder layer is set with weights, which are used to obtain a road category recognition result according to the extracted road feature information.

[0009] The image feature extraction network is actually a process of encoding the input image, and the image feature fusion network model is exactly the process of decoding it.

[0010] In a preferred embodiment, the training method of the image feature extraction network model is as follows: Step 1.1: Collect road picture data according to the road category; Step 1.2: Uniformly make the road picture data into pictures with a size of 128*96 pixels to obtain a training data set; Step 1.3: Input the training data set into the first layer of feature extraction encoder for encoding, and obtain a first-level road feature map by reducing the image resolution and the number of features; Step 1.4: Input the first-level road feature map into the second layer of feature extraction encoder for encoding to obtain a second-level road feature map; at the same time, input it into the third layer of feature extraction encoder for encoding, and so on, until the lowest-level road feature map is finally obtained.

[0011] In a preferred embodiment, the training method of the image feature fusion network model is as follows: Step 1.5: Input the road feature maps of each level into the image feature fusion network model for decoding in turn, compare the decoded image information with the original input value data, and obtain the prediction result of each layer; Step 1.6: Calculate the loss function based on the prediction result and the true road category. The loss function is a scalar value that calculates the difference between the prediction result and the true road feature map. Then, adjust the weights of the decoder according to the scalar value, and repeat Step 1.5. Finally, obtain the loss function with the smallest scalar value; the smaller the loss function value, the better, preferably approaching 0. However, in some cases, due to factors such as data noise or model complexity, the optimal loss function value may not be 0, so only the smallest loss function can be obtained.

[0012] Step 1.7: Use the loss function with the smallest scalar value to train the image feature fusion network to obtain the finally trained road category prediction model. Step 1.8: Input the real-time collected front road image information during the autonomous driving process into the trained road category prediction model to obtain the road category recognition result.

[0013] In a preferred embodiment, when the road category recognition result is a potholed road and an icy road, the processing process of the cloud server platform is as follows: For the front road identified as a potholed road by the image recognition technology in the autonomous driving system, it is necessary to further calculate the size and depth of the pothole.

[0014] Using the image information obtained by the camera, through technologies such as feature extraction and object detection, find the potholed area on the road and obtain its position, size, shape and other information. At the same time, the image information obtained by the camera can also be fused with the depth information and ground height information obtained by the lidar sensor to obtain more accurate pothole size and depth information.

[0015] Obtain the road surface height information: Use sensors such as lidar and cameras to obtain the height information of the road surface and record the pothole positions.

[0016] Calculate the relative height of the pothole bottom: Use sensors such as lidar to measure the distance of the laser beam from the lidar to the pothole bottom, and the depth information of the pothole relative to the vehicle chassis can be obtained.

[0017] Calculate the absolute height of the pothole: By subtracting the relative height of the pothole bottom from the road surface height information, the depth information of the pothole bottom relative to the road surface, that is, the absolute depth information of the pothole, can be obtained.

[0018] Based on the calculated pothole information, the potholes can be classified and judged. For example, according to the feature information such as the size and depth of the pothole, it can be classified into types such as shallow potholes, deep potholes or protrusions, and then further judged whether to drive directly through, detour or stop driving.

[0019] Set a threshold for the pothole depth according to the performance of the self-driving vehicle itself; If the pothole depth is less than or equal to the threshold, the autonomous vehicle can pass over the pothole; If the pothole depth is greater than the threshold, the autonomous vehicle cannot pass, and at this time, the cloud server platform sends an alarm message to the general control center; When the road category recognition result is an ice and snow road, calculate the friction coefficient between the wheels and the road surface: In the case of icy and slippery roads, due to the uncertainty of the change in the road surface friction coefficient, traditional control algorithms may not be adaptable. Therefore, a database comparison method can be used to control the vehicle speed. By predicting and estimating the road surface friction coefficient, more accurate and stable vehicle speed control can be achieved.

[0020] Collect in advance the friction coefficients between the autonomous vehicle and the road surface on the ice and snow road to form a friction coefficient set; According to different friction coefficients, formulate corresponding braking speeds; form a braking speed database; Map the friction coefficient between the wheels and the road surface on the ice and snow road obtained in real time to the braking speed database to obtain the current braking vehicle speed. At this time, the cloud server platform sends speed information to the general control center.

[0021] When the general control center receives the alarm message sent by the cloud server platform, the control process is as follows: During driving, data such as the current speed, acceleration, and attitude information of the vehicle can be obtained through sensors, and combined with the distance between the vehicle's own width and the lane edge, calculate whether the conditions for detouring can be met. If the conditions are met, detour; if the conditions are not met, take preventive measures such as sending an alarm.

[0022] When the pothole depth is greater than the threshold: Obtain the current vehicle position through lidar positioning and mark it in the actual ground coordinate system to obtain the vehicle position coordinates. Through the position coordinates, calculate the distance between the current vehicle position and the lane. When the distance is greater than the vehicle's own width, turn to bypass the pothole; when the distance is less than or equal to the vehicle's own width, stop immediately and re-plan the path.

[0023] When the general control center receives the speed information sent by the cloud server platform, it moves forward at the braking speed corresponding to the current friction coefficient.

[0024] In the above technical solution, the technical effects and advantages provided by the present invention are as follows: 1. The present invention first classifies the collected road pictures by road category, and then trains the image recognition neural network model. Through training, the road category recognition result is obtained, which effectively identifies pothole roads and ice and snow roads, and based on the recognition result, further makes judgments on speed and path planning, improving the recognition accuracy of complex road conditions and the control accuracy of driverless driving.

[0025] 2. The present invention provides depth calculation on potholed roads and braking speed judgment on icy roads, and respectively provides control methods according to the calculation results and speed judgment results, so as to make more appropriate decisions for autonomous driving and realize the smooth driving of autonomous vehicles on complex road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

[0027] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] Embodiment 1. Please refer to Figure 1 As shown, an autonomous driving system based on 5G Internet of Things in this embodiment includes an image acquisition device, a cloud server platform, a general control center, and an autonomous driving vehicle; The image acquisition device includes a lidar, a camera, a front vehicle radar, a rear vehicle radar, and ultrasonic waves; it is used for collecting road images and the surrounding environment.

[0030] According to experience, we have predefined road categories in advance, namely flat roads, potholed roads, and icy roads. According to these categories, we purposefully collect road picture data containing these categories. We can also download these pictures from the Internet such as Baidu and 360 search engines, and classify and organize these pictures according to road categories.

[0031] The cloud server platform is used to perform real-time and efficient processing on a large amount of image acquisition information and perform intelligent recognition to obtain road recognition results; The general control center makes decision control on all driving behaviors of the autonomous driving vehicle using algorithms; The autonomous driving vehicle is used for autonomous driving and executing decisions.

[0032] The acquisition of road information by the image acquisition device is as follows: Since lidar can collect images of the front up to a distance of 100 feet, in autonomous driving, the lidar in the acquisition device continuously collects images and videos of the road conditions within 30 meters in front of the autonomous vehicle and transmits them back to the cloud server platform for calculation. The process by which the cloud server platform processes and analyzes the image information to obtain the road recognition result is as follows: Step 1: Obtain road picture data based on the collected image information. Road pictures can also be downloaded from the Internet, such as Baidu and 360 search engines, and the road picture data is classified to construct a road category dataset. The road categories include flat roads, potholed roads, and icy roads. Step 2: Based on the road category dataset, train an image recognition autoencoder model to obtain a road category detection model. Step 3: Input the road data to be detected into the trained road category detection model to obtain the road category detection result.

[0033] The image recognition autoencoder model is a commonly used result when using a convolutional neural network for image feature extraction, namely the encoder-decoder structure. Among them, the encoder is responsible for downsampling the input image layer by layer to extract low-level feature information; the decoder restores the low-resolution feature map to a high-resolution image layer by layer through upsampling and transposed convolution operations, while retaining high-level semantic information.

[0034] The image feature extraction network model consists of several layers of feature extraction encoders, which are used to extract road feature information and reduce the image resolution and the number of features. The image feature fusion network model consists of several layers of decoders. Each decoder layer is set with a weight, which is used to calculate based on the extracted road feature information and perform weighted calculation to obtain the road category recognition result. The image feature extraction network is actually a process of encoding the input image, and the image feature fusion network model is exactly the process of decoding it.

[0035] During the training process, first, the input picture is downsampled by the encoder to obtain low-level feature information, and then these feature information are input into the decoder for upsampling operations. In each decoder layer, the loss function is calculated based on the prediction result and the true label to measure the difference between the prediction result and the true label, and the decoder weight value is adjusted using the difference value. During the entire training process, by continuously adjusting the decoder weight value, the value of the loss function is reduced, and finally, an accurate prediction model is obtained. The process is as follows: The training method of the image feature extraction network model, that is, the downsampling process is: Step 1.1: Collect road picture data according to the road category. Step 1.2: Uniformly produce road picture data into pictures with a size of 128*96 pixels to obtain a training data set; Step 1.3: Input the training data set into the first-level feature extraction encoder for encoding. By reducing the image resolution and the number of features, this can greatly reduce computational redundancy. Through calculation, a first-level road feature map is obtained; Step 1.4: Input the first-level road feature map into the second-level feature extraction encoder for encoding to obtain a second-level road feature map; at the same time, input it into the third-level feature extraction encoder for encoding. This operation is performed N times, and finally the lowest-level road feature map is obtained.

[0036] The training method of the image feature fusion network model, that is, the upsampling process is as follows: Step 1.5: Sequentially input the road feature maps of each level into the image feature fusion network model for decoding to obtain the decoded image feature information. Compare the decoded image feature information with the original input value data to obtain the prediction result of each layer; Step 1.6: Calculate the loss function according to the prediction result and the real road category. The loss function is a scalar value that calculates the difference between the prediction result and the real road feature map. Then adjust the weights of the decoder according to the scalar value, and repeat Step 1.5. Finally, obtain the loss function with the smallest scalar value; during the continuous calculation process, finally obtain the loss function with a gradually decreasing value.

[0037] Step 1.7: Use the loss function with the smallest scalar value to train the image feature fusion network to obtain the finally trained road category prediction model; Step 1.8: Input the real-time collected front road image information during the automatic driving process into the trained road category prediction model to obtain the road category recognition result.

[0038] When the road category recognition result is a potholed road and an icy road, the processing process of the cloud server platform is as follows: For the front road identified as a potholed road by the image recognition technology in the automatic driving system, it is necessary to further calculate the size and depth of the potholes.

[0039] Using the image information obtained by the camera, through technologies such as feature extraction and object detection, find the potholed areas on the road and obtain their position, size, shape and other information. At the same time, the image information obtained by the camera can also be fused with the depth information and ground height information obtained by the lidar sensor to obtain more accurate pothole size and depth information.

[0040] LiDAR can calculate the distance information between an object and the sensor by measuring the time difference between the emission and reception of a laser beam. In ground height detection, this principle can be used to obtain ground height information. Specifically, LiDAR emits a laser beam and records the time when the reflected light is received. Since the speed of light is known, the distance from the LiDAR to the object surface can be calculated based on the recorded time.

[0041] Obtain road surface height information: Use sensors such as LiDAR and cameras to obtain the height information of the road surface and record the pothole positions.

[0042] Calculate the relative height of the pothole bottom: Use sensors such as LiDAR to measure the distance from the LiDAR to the pothole bottom, and the depth information of the pothole relative to the vehicle chassis can be obtained.

[0043] Calculate the absolute height of the pothole: By subtracting the relative height of the pothole bottom from the road surface height information, the depth information of the pothole bottom relative to the road surface can be obtained, which is the absolute depth information of the pothole.

[0044] For example: Obtain the height value H1 from the LiDAR to the pothole surface on the road and record the pothole position; obtain the height value H2 from the LiDAR to the pothole bottom through the LiDAR; the difference between H2 and H1 is the absolute depth value of the pothole; both H1 and H2 are relative heights relative to the LiDAR position, so the difference between H2 and H1 is the absolute height.

[0045] Based on the calculated pothole information, the potholes can be classified and judged. For example, according to the characteristic information such as the size and depth of the pothole, it can be classified into shallow potholes, deep potholes or protrusions, etc., and then further judged whether to pass directly, detour or stop driving.

[0046] Set a threshold for the pothole depth according to the performance of the autonomous vehicle itself; If the pothole depth is less than or equal to the threshold, the autonomous vehicle can pass over the pothole; If the pothole depth is greater than the threshold, the autonomous vehicle cannot pass, and at this time the cloud server platform sends an alarm message to the total control center; When the road category recognition result is an ice and snow road, calculate the friction coefficient between the wheels and the road surface: Collect in advance the friction coefficients between the autonomous vehicle and the road surface on the ice and snow road to form a friction coefficient set; According to different friction coefficients, formulate corresponding braking speeds; form a braking speed database; Map the real-time obtained friction coefficient between the wheels and the road surface on the ice and snow road to the braking speed database to obtain the current braking vehicle speed, and at this time the cloud server platform sends speed information to the total control center.

[0047] When the total control center receives the alarm information sent by the cloud server platform, the control process is as follows: When the pothole depth is greater than the threshold: The position of the current vehicle is obtained through lidar positioning and marked in the actual ground coordinate system to obtain the vehicle position coordinates. Based on the position coordinates, the distance between the current vehicle position and the lane is calculated. When the distance is greater than the width of the vehicle itself, the vehicle steers around the pothole; when the distance is less than or equal to the width of the vehicle itself, the vehicle stops immediately and makes a new path plan.

[0048] When the total control center receives the speed information sent by the cloud server platform, it moves forward at the braking speed corresponding to the current friction coefficient.

[0049] The braking speed can also be 0. When the current friction coefficient is very small, the corresponding braking speed is 0. When the braking speed is 0, the vehicle stops or makes another path plan.

[0050] As described above, this is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An autonomous driving system based on 5G Internet of Things, characterized by: It includes image acquisition device, cloud server platform, general control center and autonomous driving vehicle; The image acquisition device includes a plurality of laser radars and camera devices, which are used to acquire images of the road and obtain road image information; The cloud server platform is used to process and analyze road image information in real time, perform intelligent recognition, and obtain road recognition results; The general control center makes driving decision control for the autonomous driving vehicle according to the road recognition result; The autonomous driving vehicle is used for autonomous driving and executing driving decisions.

2. The 5G Internet of Things-based autonomous driving system according to claim 1, characterized in that: The process of the image acquisition device acquiring images of the road is as follows: The camera device in the image acquisition device collects image information of the road conditions within 30 meters in front of the autonomous driving vehicle in real time and transmits it back to the cloud server platform. The cloud server platform processes and analyzes the image information to obtain a road recognition result.

3. The 5G Internet of Things-based autonomous driving system according to claim 2, characterized in that: The cloud server platform processes and analyzes the image information to obtain the road recognition result in the following process: Step 1: Obtain road image data based on the collected image information, and classify the road image data to construct a road category data set; the road categories include flat roads, bumpy roads, and icy and snowy roads; Step 2: Based on the road category dataset, train the image recognition autoencoder model to obtain the road category detection model; Step 3: Input the road data to be detected into the trained road category detection model to obtain the road category detection result.

4. The 5G Internet of Things-based autonomous driving system according to claim 3 is characterized in that: The image recognition autoencoder model includes: image feature extraction network model and image feature fusion network model; The image feature extraction network model is composed of several layers of feature extraction encoders for extracting road feature information; The image feature fusion network model is composed of several layers of feature decoders, each layer of decoders is set with a weight, which is used to extract road feature information, fuse it according to its importance to road category detection, and perform regression prediction to obtain a road category detection model.

5. The 5G Internet of Things-based autonomous driving system according to claim 4 is characterized in that: The training method of the image feature extraction network model is: Step 1.1, collect road image data according to road category; Step 1.2: The road image data are uniformly made into images of 128*96 pixels in size to obtain a training data set; Step 1.3, input the training data set into the first-layer feature extraction encoder for encoding to obtain a first-level road feature map; Step 1.4: Input the first-level road feature map into the second-layer feature extraction encoder for encoding to obtain the second-level road feature map; at the same time, input it into the third-layer feature extraction encoder for encoding, and so on, and finally obtain the lowest-level road feature map.

6. The 5G Internet of Things-based autonomous driving system according to claim 5, characterized in that: The training method of the image feature fusion network model is: Step 1.5: Input each level of road feature map into the image feature fusion network model for decoding, compare the decoded image information with the original input value data, and obtain the prediction result of each layer; Step 1.6, calculate the loss function according to the prediction result and the real road category, the loss function is a scalar value of the difference between the prediction result and the real road feature map, and then adjust the weight of the decoder according to the scalar value, repeat step 1.5, and finally obtain the loss function with the minimum scalar value; Step 1.7, use the loss function with the smallest scalar value to train the image feature fusion network to obtain the final trained road category prediction model; Step 1.8: Input the front road image information collected in real time during the autonomous driving process into the trained road category prediction model to obtain the road category recognition result.

7. The 5G Internet of Things-based autonomous driving system according to claim 6, characterized in that: When the road category recognition result is pothole road and icy road, the processing process of the cloud server platform is as follows: When the road type recognition result is a pothole road: The laser radar obtains the height value H1 from itself to the surface of the pothole on the road and records the location of the pothole; the laser radar obtains the height value H2 from the laser radar to the bottom of the pothole; the difference between H2 and H1 is the absolute depth value of the pothole; According to the performance of the self-driving vehicle, a threshold value is set for the depth of the pothole; If the depth of the pothole is less than or equal to the threshold, the self-driving vehicle can pass through the pothole; If the depth of the pothole is greater than the threshold, the self-driving vehicle cannot pass through, and the cloud server platform sends an alarm message to the main control center; When the road type recognition result is ice and snow road, calculate the friction coefficient between the wheel and the road surface: Collect the friction coefficient between the autonomous driving vehicle and the road surface on the icy and snowy road in advance to form a friction coefficient set; According to different friction coefficients, the corresponding braking speed is formulated to form a braking speed database; The friction coefficient between the wheels and the road surface on the icy and snowy roads obtained in real time is mapped to the braking speed database to obtain the current braking speed. At this time, the cloud server platform sends the speed information to the main control center.

8. The 5G Internet of Things-based autonomous driving system according to claim 7, characterized in that: When the general control center receives the alarm information sent by the cloud server platform, the control process is: When the pothole depth is greater than the threshold: The current position of the vehicle is obtained through laser radar positioning, and marked in the actual ground coordinate system to obtain the vehicle position coordinates. The distance between the current vehicle position and the lane is calculated through the position coordinates. When the distance is greater than the width of the vehicle itself, the vehicle turns to bypass the pothole; when the distance is less than or equal to the width of the vehicle itself, the vehicle stops immediately and re-plans the path.

9. The 5G Internet of Things-based autonomous driving system according to claim 7, characterized in that: When the general control center receives the speed information sent by the cloud server platform, it moves forward according to the braking speed corresponding to the current friction coefficient.

Images