A sensor data screening method and device for unmanned mine car based on cloud data

Abstract

The application uploads the data obtained by the vehicle-mounted camera sensor of the unmanned mine car to the cloud server, carries out feature recognition on the cloud data through the cloud server, then selects the road curvature, the slope, whether there is an obstacle in the front road, whether there is a fork in the front road as the characteristic parameters according to the characteristics of the mine area to realize the data screening of the vehicle-mounted camera, so that the data input into the decision model of the unmanned mine car is reduced, and at the same time, the speed information of the unmanned mine car is referred to in the vehicle-mounted camera data screening process, and the screening weight is dynamically determined, so that the decision time of the decision model is reduced, and according to the characteristics of the mine area, the sensor data screening and the dynamic determination of the screening weight are considered, most of the interference garbage data is considered, the decision time is reduced, and the decision accuracy of the decision layer is improved.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method and device for filtering sensor data of unmanned mining trucks based on cloud data. Background Technology

[0002] Autonomous driving technology utilizes various technologies, including radar, laser, ultrasound, GPS, odometer, and computer vision, to perceive its surrounding environment. Through an advanced computer control system, it identifies obstacles and various signs, and plans appropriate paths to control the movement of mining trucks. With the rapid development and widespread application of automotive intelligence technology, autonomous vehicles have become the future development trend of the automotive industry. Due to its innovativeness, practicality, complexity, and multidisciplinary nature, autonomous vehicles are also a very popular research field, with many international and domestic companies conducting research on autonomous driving technology.

[0003] With the gradual improvement of communication methods, in order to improve the real-time performance of autonomous driving path and driving parameter prediction, uploading data acquired by sensors during autonomous driving to a cloud server and then using this data for path and driving parameter prediction has become a major research focus. Generally, autonomous vehicles integrate 4-6 onboard cameras for decision support. Such a large number of onboard cameras generate a large amount of data. However, not all data is useful for decision-making. Moreover, due to the unmanned nature of autonomous vehicles, it is necessary to make near real-time and accurate predictions of subsequent paths and driving parameters. If the data generated by the sensors is not filtered before being input into the prediction model for calculation, not only will the latency increase significantly, but the prediction accuracy will also decrease.

[0004] Meanwhile, mines, due to their fewer sudden disturbances and simpler road conditions, are gradually becoming a hot research area for autonomous driving. However, existing technologies do not have a solution specifically designed for the characteristics of mines to filter the onboard camera data of autonomous mining trucks, thereby reducing latency and improving the accuracy of predicting driving parameters. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the above-mentioned technical solutions by providing a method and device for filtering sensor data of unmanned mining trucks based on cloud data. For the application scenario of mining, the method filters the data collected by the sensors of unmanned mining trucks, thereby reducing the amount of data input to the prediction model, and achieving the purpose of reducing latency and improving accuracy.

[0006] The concepts involved in this application will first be explained with reference to the accompanying drawings. It should be noted that the following explanation of each concept is only to make the content of this application easier to understand and does not imply any limitation on the scope of protection of this application.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for filtering sensor data of an unmanned mining truck based on cloud data includes:

[0008] Step 1: Acquire video data collected by multiple onboard camera sensors while the driverless mining truck is in motion;

[0009] Step 2: Convert the video data collected by the multiple vehicle cameras into image format frame by frame, add timestamps, and upload them to the cloud server;

[0010] Step 3: Obtain the feature information of each frame of the image through image recognition technology as cloud data. The cloud data includes: road slope information ahead, curvature of the road ahead, whether there are obstacles ahead, and whether there is a fork in the road ahead.

[0011] Step 4: Filter each frame of image based on the cloud data obtained in Step 3 and the driving speed.

[0012] Furthermore, the camera sensor is used to collect environmental information around the mining truck.

[0013] Furthermore, the environmental information surrounding the mining truck includes the situation ahead of the mining truck and road conditions.

[0014] Furthermore, in step 2, data can be uploaded using 5G communication technology. Alternatively, data transmission can be achieved by establishing an edge network communication node in the mining area, using an unmanned vehicle-mounted computer to transmit the data stream to the edge node, and then the edge network node transmits the data to the cloud computer.

[0015] Furthermore, step 4 specifically includes:

[0016] Step 4.1: Generate classification rules using machine learning;

[0017] Step 4.2: Classify the cloud data in Step 3 using the classification rules established in Step 4.1. The first category is junk data; the second category is data that may be uploaded; and the third category is data that is confirmed to be uploaded.

[0018] Step 4.3: Determine the data filtering and uploading strategy based on the classification rules and driving speed.

[0019] Furthermore, step 4.1 specifically includes:

[0020] Step 4.1.1: Use a manned mining cart to drive a vehicle on this task route, record the data information during the driving process through an on-vehicle camera sensor, and record the driving parameters of the vehicle through an on-vehicle computer. The driving parameters include vehicle speed. Add Step 4.1.2: Upload the above information to the cloud server and perform feature recognition processing on the above information; extract the road curvature, slope, whether there are obstacles on the front road, and whether there is a fork on the front road of each frame of image as the feature parameter information for data screening to generate a feature information dataset. Assign initial weights to the four feature parameters of road curvature, slope, whether there are obstacles on the front road, and whether there is a fork on the front road, and establish a classification model as shown in formula (1):

[0021] Ti = a * Ti_c + b * Ti_s + c * Ti_o + d * Ti_a (1)

[0022] Where, Ti is the image feature value, a, b, c, d are the weights of each feature parameter, Ti_c is the road curvature size recognized from the i-th frame of image, Ti_s is the road slope size recognized from the i-th frame of image, Ti_o is the information of the size of obstacles on the front road recognized from the i-th frame of image, and Ti_a is whether there is a road fork;

[0023] Use the feature information dataset of each frame of image and the driving parameters at this moment as training data to train the classification model to obtain the model parameters a, b, c, d;

[0024] Step 4.1.3: Perform feature recognition on the collected real-time driving data, determine the road curvature size, road slope size, information on the size of road obstacles, and whether there is a fork on the road, input them into the classification model, and calculate the image feature value Ti. The specific classification rules based on the image feature value are as follows:

[0025] If Ti < T1, determine it as the third type of data; if T1 ≤ Ti ≤ T2, determine it as the second type of data; if T2 < Ti, determine it as the first type of data, where T1 < T2.

[0026] Furthermore, the classification model is a convolutional network model.

[0027] Furthermore, the said Step 4.3 further includes:

[0028] If the driving speed v < 35 km / h, only upload the third type of data;

[0029] If the driving speed 120 > v ≥ 35 km / h, calculate the upload ratio k according to formula (2):

[0030] k = (lnV - 3.55) / 1.24 (2)

[0031] Where V is the speed of the unmanned mining truck, in km / h; based on the calculation results, data with a proportion of k in the second type of data and data in the third type of data are selected as the uploaded data;

[0032] If the driving speed is ≥120km / h, select all data from the second category and the third category as the data to be uploaded.

[0033] According to another aspect of the present invention, a method for filtering sensor data of an unmanned mining truck based on cloud data is provided, comprising:

[0034] Vehicle-mounted cameras: There are multiple vehicle-mounted cameras used to acquire video data collected by multiple vehicle-mounted cameras while the unmanned mining truck is in motion;

[0035] Data conversion and upload module: used to convert the video data collected by the multiple vehicle cameras into image format in frames, add timestamps, and upload it to the cloud server;

[0036] Data filtering module: Used to run the data classification and filtering methods described above.

[0037] Based on the above technical solution, this application provides a method for filtering sensor data of unmanned mining trucks based on cloud data, which has the following technical effects:

[0038] This application uploads data acquired by the onboard camera sensors of an unmanned mining truck to a cloud server. The cloud server then performs feature recognition on the cloud data and selects road curvature, slope, the presence of obstacles ahead, and the presence of forks in the road ahead as feature parameters based on the characteristics of the mining area. This reduces the amount of data input to the unmanned mining truck's decision-making model. Furthermore, the speed information of the unmanned mining truck is considered during the data filtering process, and the filtering weights are dynamically determined, thereby reducing the decision-making time of the decision-making layer. Moreover, this application filters out most of the irrelevant data that causes interference by filtering sensor data according to the characteristics of the mining area and dynamically determining the filtering weights, thus improving the decision-making accuracy of the decision-making layer while reducing decision-making time. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1A flowchart illustrating the cloud-based sensor data filtering method for unmanned mining trucks provided in this application embodiment;

[0041] Figure 2 A comparison chart of the data filtered using the method of this application and the original data provided for an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0043] The concepts involved in this application will first be explained with reference to the accompanying drawings. It should be noted that the following explanation of each concept is only to make the content of this application easier to understand and does not imply any limitation on the scope of protection of this application.

[0044] like Figure 1 As shown, a method for filtering sensor data from unmanned mining trucks based on cloud data includes:

[0045] Step 1: Acquire video data collected by multiple onboard camera sensors while the driverless mining truck is in motion;

[0046] The camera sensor is mainly used to collect environmental information around the mining truck, including but not limited to the situation in front of the mining truck and road conditions.

[0047] Step 2: Convert the video data collected by the multiple vehicle cameras into image format frame by frame, add timestamps, and upload them to the cloud server;

[0048] Specifically, data can be uploaded using 5G communication technology, or data can be transmitted by establishing edge network communication nodes in the mining area, using unmanned vehicle-mounted computers to transmit data streams to the edge nodes, and then the edge network nodes transmit the data to the cloud computers.

[0049] Step 3: Obtain the feature information of each frame of the image through image recognition technology as cloud data. The cloud data includes: information on the slope of the road ahead, the curvature of the road ahead, whether there are obstacles on the road ahead, whether there is a fork in the road ahead, etc.

[0050] Cloud servers offer superior performance compared to in-vehicle computers, enabling them to run more complex models. Therefore, this application employs image recognition technology on the cloud server to obtain feature information from each frame of the image, which is then used as cloud data to participate in the decision-making layer's computation.

[0051] Step 4: Filter each frame of the image based on the cloud data obtained in Step 3 and the driving speed;

[0052] Specifically, step 4 includes:

[0053] Step 4.1: Generate classification rules using machine learning;

[0054] Due to the presence of multiple onboard cameras in the driverless mining truck, the truck generates a large amount of data during its operation. Faced with this massive amount of data, efficiently acquiring knowledge through machine learning has gradually become a major driving force for the development of machine learning technology. Specifically, step 4.1 includes: Step 4.1.1: Using a manned mining truck to drive the vehicle along the task route, and recording the data information of the driving process through the onboard camera sensors, and recording the driving parameters of the vehicle through the onboard computer, including vehicle speed, acceleration, and whether steering is involved; and recording the data information and driving parameters of this driving process through the onboard camera sensors.

[0055] Step 4.1.2: Upload the above information to the cloud server and perform feature recognition processing on the above information; extract the road curvature, slope, whether there are obstacles in the road ahead, and whether there are forks in the road ahead as feature parameters for data filtering to generate a feature information dataset; assign initial weights to the four feature parameters of road curvature, slope, whether there are obstacles in the road ahead, and whether there are forks in the road ahead, and establish a classification model as shown in formula (1):

[0056] Ti=a*Ti_c+b*Ti_s+c*Ti_o+d*Ti_a (1)

[0057] Among them, Ti is the image feature value, a, b, c, and d are the weights of each feature parameter, Ti_c is the road curvature size recognized in the i-th frame image, Ti_s is the road slope size recognized in the i-th frame image, Ti_o is the size information of the road obstacle in front recognized in the i-th frame image, and Ti_a is whether there is a road fork; specifically, the present application uses a convolutional network model as the classification model; the convolutional network model is trained with training data to establish classification rules; during training, the Adam optimizer is used, and the initial learning rate is 0.01. Affected by the software and hardware environment, the size of the feature information data set input into the learning network model is 4x4xC, where C is the number of parameters in each feature parameter data set input into the network model. It is worth mentioning that the present application uses four parameters, namely road curvature, slope, whether there is an obstacle in the front road, and whether there is a fork in the front road, as feature parameters, so C takes the value of 4;示例性地,采用某作业路线为例,具体地,首先针对道路曲率,坡度,前方道路是否存在障碍物以及前方道路是否存在岔路等四个特征参数赋值初始权重,然后依据公式(1)计算出每帧图像的特征值,通过训练数据对分类模型进行训练,主要是对特征参数的权重进行训练,从而得到最优参数。

[0058] Step 4.1.3: Perform feature recognition on the collected real-time driving data to determine the road curvature size, road slope size, road obstacle size information, and whether there is a road fork. After inputting them into the classification model, calculate the image feature value Ti, and its specific classification rules based on the image feature value are as follows:

[0059] If Ti < T1, determine it as the third type of data; if T1 ≤ Ti ≤ T2, determine it as the second type of data; if T2 < Ti, determine it as the first type of data, where T1 < T2.

[0060] Step 4.2: Calculate the feature value of each frame of picture using the feature value calculation method in Step 4.1. Then, classify the cloud data in Step 3 using the classification rules established in Step 4.1. The first type is garbage data; the second type is possible upload data; the third type is determined upload data. Among them, for garbage data, since it has little impact on the decision-making of the driverless mining truck, this type of data is not input into the decision-making model to participate in the decision-making. For possible upload data, the upload ratio is judged by the following steps. For determined upload data, since it is essential data for participating in the decision-making, all this type of data participates in the decision-making in the decision-making model.

[0061] Step 4.3: Determine the data filtering and uploading strategy based on the classification rules and driving speed; Since the speed of the unmanned vehicle plays a crucial role in the correct decision-making of the vehicle, when the speed is high, more information needs to be uploaded in order to improve the decision-making accuracy and reduce the risk of accidents, while when the speed is low, less information is uploaded to reduce the decision-making time; Therefore, this application focuses on the impact of the driving speed of the unmanned mining truck on the decision-making of the mining truck to make a decision on the filtering of sensor data. Specifically, the driving speed of the unmanned vehicle is obtained in real time. If the speed is <35km / h, only the third type of data is uploaded; if the driving speed is 120>v≥35km / h, the upload ratio k is calculated according to formula (2):

[0062] k=(lnV-3.55) / 1.24 (2)

[0063] Where V is the driving speed of the unmanned mining truck, in km / h. Based on the calculation results, data in the second category with a proportion of k and data in the third category are selected as uploaded data; if the driving speed is ≥120km / h, all data in the second category and data in the third category are selected as uploaded data.

[0064] For example, when the speed is 45km / h, k is 0.21, then 21% of the data in the second type of information will be randomly selected and uploaded;

[0065] This completes the filtering of vehicle-mounted camera sensor data, uploading useful information and reducing the amount of data input to the decision-making model, thereby reducing the decision-making time of the decision-making layer. Furthermore, based on the characteristics of unmanned mining areas, this application selects road curvature, slope, and the presence of obstacles in the road ahead as feature parameters to filter vehicle-mounted camera data, increasing the amount of useful information in the uploaded data, filtering out most of the irrelevant data that causes interference, and improving the decision-making accuracy of the decision-making layer.

[0066] According to another aspect of the present invention, a method for filtering sensor data of an unmanned mining truck based on cloud data is provided, comprising:

[0067] Vehicle-mounted cameras: There are multiple vehicle-mounted cameras used to acquire video data collected by multiple vehicle-mounted cameras while the unmanned mining truck is in motion;

[0068] Data conversion and upload module: used to convert the video data collected by the multiple vehicle cameras into image format in frames, add timestamps, and upload it to the cloud server;

[0069] Data filtering module: Used to run the data classification and filtering methods described above.

[0070] This application uses a specific work route in a mining area as the specific task and conducts an experiment using the above-mentioned scheme. First, a driver with many years of driving experience is selected to drive the route. During the driving process, the driver uses relatively accurate driving parameters based on road condition information. For example, maintaining a higher driving speed on flat sections, adjusting the speed according to the gradient when a steep slope is detected ahead, controlling the speed based on the gradient when the road ahead is curved, and adjusting the speed based on the presence of vehicles at forks in the road. The data is then transmitted via an onboard camera. The sensors record data and driving parameters for each driving process; a total of 10,580 frames of images and corresponding driving data for each frame are acquired. This information is then uploaded to a cloud server. The cloud server uses the sensor data filtering method described in this application to classify the 36,580 frames of images. The first category consists of 17,683 frames, the second category 8,519 frames, and the third category 10,378 frames. It is evident that the filtering method of this application can filter out a significant proportion of irrelevant data. Then, based on the driving speed of the unmanned mining truck, the prediction delay of the unmanned driving parameters for this operating route is statistically analyzed. Figure 2 As shown, the vertical axis represents the decision time of the model after the data is filtered using the screening method of this application and then input into the decision model. The solid line represents the decision time without data screening, and the dashed line represents the decision time of the input data after using the data screening method of this application. The decision time was reduced by an average of 21%, and the decision accuracy was also significantly improved upon verification.

[0071] The embodiments and / or implementation methods described above are merely preferred embodiments and / or implementation methods for implementing the technology of the present invention, and are not intended to limit the implementation methods of the technology of the present invention in any way. Any person skilled in the art can make some modifications or alterations to other equivalent embodiments without departing from the scope of the technical means disclosed in the content of the present invention, but they should still be regarded as the technology or embodiments that are substantially the same as the present invention.

Claims

1. A method for filtering sensor data from unmanned mining trucks based on cloud-based data, characterized in that, It includes the following steps: Step 1: Obtain video data collected by multiple on-vehicle camera sensors during the driving of an unmanned mining truck; Step 2: Convert the video data collected by the multiple on-vehicle cameras into an image format in units of frames, stamp with timestamps and upload to the cloud server; Step 3: Obtain the feature information of each frame of image through image recognition technology as cloud data. The cloud data includes: the slope information of the road surface ahead during driving, the curvature magnitude of the road ahead, whether there are obstacles on the road ahead, and whether it is a fork in the road ahead; Step 4: Screen each frame of image according to the cloud data obtained in Step 3 in combination with the driving speed. Step 4 specifically includes: Step 4.1: Generate classification rules through machine learning. Step 4.1 specifically includes: Step 4.1.1: Drive a vehicle on the task route by a manned mining truck, record the data information during the driving process through an on-vehicle camera sensor, and record the driving parameters of the vehicle through an on-vehicle computer. The driving parameters include vehicle speed, acceleration, and whether to turn; and record the data information and driving parameters of this driving process through an on-vehicle camera sensor; Step 4.1.2: Upload the above information to the cloud server, and perform feature recognition processing on the above information; extract the road curvature, slope, whether there are obstacles on the road ahead, and whether there is a fork in the road ahead of each frame of picture as the feature parameter information for data screening to generate a feature information dataset, assign initial weights to the four feature parameters of road curvature, slope, whether there are obstacles on the road ahead, and whether there is a fork in the road ahead, and establish a classification model as shown in formula (1): Ti = a * Ti_c + b * Ti_s + c * Ti_o + d * Ti_a (1) Where, Ti is the image feature value, a, b, c, d are the weights of each feature parameter, Ti_c is the road curvature magnitude recognized from the i-th frame of image, Ti_s is the road slope magnitude recognized from the i-th frame of image, Ti_o is the information of the size of obstacles on the road ahead recognized from the i-th frame of image, and Ti_a is whether there is a fork in the road: Use the feature information dataset of each frame of picture and the driving parameters at this moment as training data to train the classification model to obtain the model parameters a, b, c, d; Step 4.1.3: Perform feature recognition on the collected real-time driving data, determine the road curvature magnitude, road slope magnitude, information on the size of road obstacles, and whether there is a fork in the road, input them into the classification model and calculate the image feature value Ti. The specific classification rule based on the image feature value is: If Ti < T1, determine it as the third type of data; if T1 ≤ Ti ≤ T2, determine it as the second type of data; if T2 < Ti, determine it as the first type of data, where T1 < T2 Step 4.2: Classify the cloud data in Step 3 using the classification rules established in Step 4.

1. The first type of data is garbage data; the second type of data is possible upload data; the third type of data is determined upload data; Step 4.3: Determine the data filtering and uploading strategy based on the classification rules and driving speed.

2. The method for filtering sensor data of unmanned mining trucks based on cloud data according to claim 1, characterized in that, The camera sensor is used to collect environmental information around the mining truck.

3. The method for filtering sensor data of unmanned mining trucks based on cloud data according to claim 2, characterized in that, The environmental information surrounding the mining truck includes the conditions ahead of the truck and road conditions.

4. The method for filtering sensor data of unmanned mining trucks based on cloud data according to claim 1, characterized in that, In step 2, data is uploaded using 5G communication technology, or data is transmitted by establishing an edge network communication node in the mining area, using an unmanned vehicle-mounted computer to transmit the data stream to the edge node, and then the edge network node transmits the data to the cloud computer.

5. The method for filtering sensor data of unmanned mining trucks based on cloud data according to claim 1, characterized in that, The classification model is a convolutional network model.

6. The method for filtering sensor data of unmanned mining trucks based on cloud data according to claim 1, characterized in that, Step 4.3 further includes: If the driving speed v < 35km / h, only the third type of data will be uploaded; If the driving speed is 120 > v ≥ 35 km / h, then the upload ratio k is calculated according to formula (2): k = (lnV - 3.55) / 1.24 (2) Where V is the speed of the unmanned mining truck, in km / h; based on the calculation results, data with a proportion of k in the second type of data and data in the third type of data are selected as the uploaded data; If the driving speed is ≥120 km / h, select all data from the second category and the third category as the data to be uploaded.

7. A cloud-based unmanned mining truck sensor data filtering system, comprising: Vehicle-mounted cameras: There are multiple vehicle-mounted cameras used to acquire video data collected by multiple vehicle-mounted cameras while the unmanned mining truck is in motion; Data conversion and upload module: used to convert the video data collected by the multiple vehicle cameras into image format in frames, add timestamps, and upload it to the cloud server; Data filtering module: used to run the sensor data filtering method according to any one of claims 1-6.

Images