Intelligent recognition method of bulldozer operation based on vehicle vision and multi-task learning

Abstract

The present invention belongs to the technical field related to intelligent identification of earth-moving engineering machinery, and discloses a method for intelligent identification of bulldozer operation based on vehicle-mounted vision and multi-task learning. The method comprises the following steps: S1 sets multiple cameras on the bulldozer to collect video data of the bulldozer during operation; for the collected video data, annotates its shoveling state and shoveling action according to the content of soil in front of the bulldozer blade, thereby forming a database corresponding to the video data and the shoveling state and shoveling action; S2 constructs a multi-task learning network, trains the multi-task learning network with the database in S1, thereby obtaining a recognition model, collects real-time video data of the bulldozer in real time, and inputs the real-time video data into the recognition model, thereby obtaining the shoveling state and shoveling action of the bulldozer. Through the present invention, the problems of high labor cost and inability to intelligently identify bulldozer operations are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to intelligent identification of earth-moving engineering machinery, and more specifically, relates to an intelligent identification method for bulldozer operation based on vehicle-mounted vision and multi-task learning. Background Art

[0002] With the acceleration of urbanization in my country, a large number of infrastructure projects for people's livelihood have been carried out. Earthwork projects are included in highways, railways, large-scale water conservancy projects and other projects related to the national economy and people's livelihood. Earthwork projects are large in scale, long in construction period and huge in investment. They are one of the main sub-projects in construction projects and provide basic construction site conditions for other sub-projects.

[0003] Bulldozers are suitable for many important processes of earthwork projects due to their distinctive characteristics and diverse functions. Therefore, the production efficiency of bulldozer construction operations greatly affects the completion time and construction effect of earthwork leveling work. At present, the operation supervision of bulldozers is generally carried out by assigning construction workers to arrive at the earthwork site and conduct manual supervision next to the bulldozer. The bulldozer construction operations are mainly supervised by relying on the five senses and subjective experience. It is a labor-intensive work that is time-consuming, labor-intensive and ineffective. However, with the development of society, labor costs are constantly increasing, and the requirements for bulldozer operations are also getting higher and higher. Therefore, it is necessary to study a set of intelligent recognition methods for bulldozer operations. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method for intelligent identification of bulldozer operation to solve the problems of high manpower cost and inability to intelligently identify bulldozer operation supervision.

[0005] To achieve the above object, according to one aspect of the present invention, a method for intelligently identifying bulldozer operations based on vehicle-mounted vision and multi-task learning is provided, wherein the method comprises the following steps:

[0006] S1: multiple cameras are set on the bulldozer to be identified to collect video data of the bulldozer when it is in operation; for the collected video data, its shoveling state is marked according to the content of soil in front of the bulldozer blade, and the shoveling action of the bulldozer is marked according to the position of the bulldozer in the video data, so as to form a database with one-to-one correspondence between the video data, the shoveling state and the shoveling action;

[0007] S2 constructs a multi-task learning network, trains the multi-task learning network with the database in step S1 to obtain a recognition model, collects real-time video data of the bulldozer in real time, and inputs the real-time video data into the recognition model to obtain the shoveling state and shoveling action of the bulldozer.

[0008] Further preferably, in step S1, the shoveling state includes full load, empty load and half load.

[0009] Further preferably, in step S1, the shoveling action includes moving forward, moving backward and turning.

[0010] Further preferably, in step S1, the fully loaded means that the soil in front of the blade overflows the edge of the blade, the unloaded means that there is only the blade but no soil in the image, and the half loaded means that there is soil in front of the blade but it does not overflow the edge of the blade.

[0011] Further preferably, in step S1, the judgment of the shoveling action includes two steps: first, for the captured video data, identifying the position of the bulldozer in each frame image; second, judging the shoveling action of the bulldozer based on the timing information of multiple consecutive frames of images and the corresponding bulldozer position.

[0012] Further preferably, in step S1, for the collected video data, when pre-processing the video data, a translation flip and Gaussian noise method is used to add disturbance and jitter to the original data, so as to perform artificial noise enhancement.

[0013] Further preferably, in step S2, the multi-task network is built based on a backbone network of a Transformer structure, and the backbone network is used to extract image features and temporal features of video frames, wherein the backbone network has two output branches, including a video target detection branch and a video action recognition branch.

[0014] Further preferably, in step S2, when training the multi-task learning network, the loss functions used are the cross entropy loss function CEloss and the smooth L1loss loss function.

[0015] Further preferably, in step S2, when training the multi-task learning network, the evaluation index used for shoveling state recognition is the mAP value, and the evaluation index used for shoveling action recognition is the Accuracy value.

[0016] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0017] 1. The present invention proposes a complete video data collection method based on vehicle-mounted vision that can be used for bulldozers. When working outdoors, accompanied by long-term shoveling work, changes in temperature differences may cause the camera glass to be covered with a layer of water mist, etc.; during night construction, a concentrated light source is required to provide lighting conditions, and the irradiation of the concentrated light source may cause the camera mirror to reflect, resulting in poor video imaging quality; when shoveling and transporting soil materials, there will be serious dust pollution, and a large amount of dust will fall on the camera lens glass, blocking the camera's line of sight. The video data collection method based on vehicle-mounted vision designed by the present invention can overcome the interference of adverse factors such as lighting, water mist, and dust on the construction site, and use six cameras with different installation positions, angles, and monitoring ranges to work synchronously, and cooperate with storage devices to record the construction environment and bulldozer shoveling operations in their respective monitoring ranges;

[0018] 2. The present invention proposes a data enhancement method that combines the robustness requirements of the engineering recognition model. Considering the complexity of the construction site environment, the construction site environment is often disturbed by visual noise such as dust and sand. Therefore, the robustness of the model used in engineering recognition is very important in practical applications. The present invention takes this situation into consideration. During data preprocessing, the translation flip and Gaussian noise methods are used to add disturbance and jitter to the original data, and the data is artificially denoised, thereby moderately increasing the complexity of the samples. When the processed samples are used for training, the generalization ability of the model can be further improved;

[0019] 3. The present invention proposes a new video multi-task learning network model based on the transformer architecture. Taking into account the complexity of the construction site environment, the traditional convolutional backbone network can only calculate the pixels within a convolution kernel, and the receptive field is very small. It cannot handle the interference problems such as lighting and shadows in engineering scenes. Although the convolutional backbone network is well-known in the field of computer vision, its role is greatly limited for complex engineering application scenarios. However, the transformer architecture, as an emerging visual architecture, uses the self-attention mechanism and has a large receptive field. It is very suitable for dealing with many noise interference problems in engineering applications caused by the complex construction site environment. The recognition model built based on the transformer architecture is more suitable for practical applications in engineering scenarios.

[0020] 4. The present invention proposes an automated intelligent bulldozer operation recognition method based on vehicle-mounted vision and multi-task learning, which uses a vehicle-mounted camera to replace human eyes to monitor the operation of the bulldozer. For the video data collected by the camera, computer vision technology is used to extract information related to the bulldozer operation status, that is, a computer vision method based on multi-task learning is used to perform the recognition of the bulldozer's shoveling status (empty, half-loaded, fully loaded) and shoveling action (forward, backward, turning) tasks, thereby realizing automated supervision of bulldozer operations and providing a data basis for improving bulldozer production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning constructed according to a preferred embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of a vehicle-mounted vision-based video data collection method constructed according to a preferred embodiment of the present invention;

[0023] Figure 3 It is a flow chart of shoveling state identification and shoveling action identification constructed according to the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] like Figure 1 As shown in the figure, the intelligent recognition method of bulldozer operation based on vehicle-mounted vision technology and multi-task learning includes:

[0026] Offline training phase:

[0027] (S1) Figure 2 As shown in the figure, a vehicle-mounted camera is installed on the bulldozer to collect video data when the bulldozer is operating.

[0028] (S2) Combined with the robustness requirements of the engineering recognition model, the data preprocessing method of data enhancement is adopted. The translation flip and Gaussian noise methods are used to add disturbance and jitter to the original data, thereby improving the generalization ability of the model.

[0029] (S3) Based on the collected video data, two tasks, namely, the shoveling state and the shoveling motion, are labeled respectively.

[0030] (S4) Taking into account the complexity of the construction site environment, a multi-task learning network for videos is built based on the transformer architecture. The large receptive field of the transformer architecture is used to adaptively solve many noise interference problems in engineering applications caused by the complex construction site environment.

[0031] (S5) The obtained two types of labeled data are input into the multi-task network for training until the loss function and the evaluation index curve converge, indicating that the training is completed.

[0032] Real-time recognition stage:

[0033] (S6) Figure 3 As shown in the figure, the trained multi-task network model is combined with the vehicle-mounted camera to form an intelligent bulldozer operation recognition system. The images collected by the vehicle-mounted camera in real time are transmitted to the input port of the multi-task network model. After processing, the model can automatically output the recognized bulldozer shoveling status and shoveling action.

[0034] Preferably, the video data collection method based on vehicle-mounted vision described in S1 includes:

[0035] (S1.1) The camera used to collect video stream data is selected considering the environmental conditions, feasibility and economy of the earthwork construction site.

[0036] (S1.2) Cameras with different positions, angles and monitoring ranges are installed on the bulldozer body to comprehensively record the construction process and external environment information of the bulldozer.

[0037] (S1.3) A storage device of sufficient size is selected to store video data. The storage device can be connected to multiple camera data sources and can perform real-time viewing and backtracking of data collected by each data source.

[0038] Preferably, the data enhancement method described in S2 that combines the robustness requirements of the engineering recognition model is to use translation flipping and Gaussian noise methods to add disturbance and jitter to the original data during data preprocessing, thereby improving the generalization ability of the model.

[0039] Preferably, the video data labeling method described in S3 adopts separate labeling: wherein the shoveling state labeling is based on video target detection labeling, and the labeling is mainly performed on randomly extracted video frames, and the extracted video frames (i.e., images) are labeled using labelme, and the shovel blade is marked in the bounding box of the image and labeled. The label categories include full load, half load, and empty load. Finally, a labeling file is generated, and the file content includes the path, label, and bounding box coordinate information of each video frame; the shoveling action labeling is based on video action recognition labeling, and is performed on video frames extracted at equal intervals. The shoveling action of each video frame is labeled separately, and the label categories include forward, backward, and turn. The labeling process can be directly performed by making a txt file, and the file content of the generated txt labeling file includes the path and label of each video frame.

[0040] Preferably, the method for establishing a multi-task network model described in S4 is built based on a backbone network of a Transformer structure, and uses the backbone network to extract image features and temporal features of video frames. There are two output branches in the backbone network, and the front-end partial network layers of the two output branches are shared based on multi-task learning, and the back-end network layers form a video target detection branch and a video action recognition branch, respectively, and output two outputs. The multi-task learning network has Input1 and Input2, which correspond to the data streams of the video target detection branch and the video action recognition branch, respectively. During training, the two branches input their own training samples for training; when used after the training is completed, Input1 and Input2 can share a data input, and simultaneously output two detection results of shoveling status and shoveling action.

[0041] Preferably, the transformer-based architecture used in S4 takes into account the complexity of the construction site environment. The receptive field of the traditional convolutional backbone network is very small and cannot handle interference problems such as lighting and shadows well. Although it is well-known in the field of computer vision, its role is greatly limited for complex engineering application scenarios; however, the transformer architecture, as an emerging visual architecture, has a large receptive field and is very suitable for handling many noise interference problems in engineering applications caused by the complex construction site environment.

[0042] Preferably, the transformer-based architecture described in S4 can calculate every pixel in the data based on the self-attention mechanism, but the traditional convolution architecture can only calculate the pixels within a convolution kernel, and its receptive field is much smaller. Therefore, the transformer-based architecture can obtain better global information and can adaptively handle the adverse interference caused by the complex construction site environment in the data.

[0043] Preferably, the loss function in the model training described in S5 is a cross entropy loss function CEloss:

[0044]

[0045] Among them, M represents the number of categories; N represents the total number of samples; y ic is a sign function (0 or 1), if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; p ic is the predicted probability that the observed sample i belongs to category c.

[0046] The recognition of the shoveling state is a target recognition problem, which also involves regression loss and the loss function smooth L1loss:

[0047] smooth L1 loss = 0.5x 2 (|x|<1)

[0048] smooth L1 loss=|x|-0.5(otherwise)

[0049] Where x is the difference between the predicted box and the groundtruth

[0050] Preferably, the evaluation index in the model training described in S5, the shovel state recognition adopts mAP as the evaluation index:

[0051] mAP = mean(∫p(R)dR)

[0052] Where p(R) is the precision value corresponding to a certain recall value in the PR curve after training is completed.

[0053] The accuracy value is used as the evaluation index for shoveling action recognition:

[0054] Accuracy=(TP+TN) / (P+N)

[0055] Where TP is the number of samples that are actually positive examples and are identified as positive examples; TN is the number of samples that are actually negative examples and are identified as negative examples; P is the actual number of positive samples; N is the actual number of negative samples.

[0056] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Intelligent recognition method for bulldozer operation based on vehicle vision and multi-task learning, characterized in that: The method comprises the following steps: S1: multiple cameras are set on the bulldozer to be identified to collect video data of the bulldozer when it is in operation; for the collected video data, its shoveling state is marked according to the content of soil in front of the bulldozer blade, and the shoveling action of the bulldozer is marked according to the position of the bulldozer in the video data, so as to form a database with one-to-one correspondence between the video data, the shoveling state and the shoveling action; For the collected video data, when preprocessing the video data, the translation flip and Gaussian noise methods are used to add disturbance and jitter to the original data to perform artificial noise increase, thereby moderately increasing the complexity of the sample. When the processed samples are used for training, the generalization ability of the model can be further improved; S2 constructs a multi-task learning network, trains the multi-task learning network with the database in step S1 to obtain a recognition model, collects real-time video data of the bulldozer in real time, and inputs the real-time video data into the recognition model to obtain the shoveling state and shoveling action of the bulldozer.

2. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 1, characterized in that: In step S1, the shoveling status includes full load, empty load and half load.

3. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 2, characterized in that: In step S1, the shoveling action includes forward, backward and turning.

4. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 3, characterized in that: In step S1, the fully loaded state is when the soil in front of the blade overflows the edge of the blade, the unloaded state is when there is only the blade but no soil in the image, and the half loaded state is when there is soil in front of the blade but it does not overflow the edge of the blade.

5. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 4, characterized in that: In step S1, the determination of the bulldozer action includes two steps: first, for the captured video data, the position of the bulldozer in each frame image is identified; second, the bulldozer action is determined based on the timing information of the continuous multiple frames of images and the corresponding bulldozer position.

6. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 1 or 2, characterized in that: In step S2, the multi-task network is built based on a backbone network of a Transformer structure, and the backbone network is used to extract image features and temporal features of video frames, wherein the backbone network has two output branches, including a video target detection branch and a video action recognition branch.

7. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 1 or 2, characterized in that: In step S2, when training the multi-task learning network, the loss functions used are the cross entropy loss function CEloss and the smooth L1loss loss function.

8. The method for intelligently identifying bulldozer operation based on vehicle-mounted vision and multi-task learning as claimed in claim 1 or 2, characterized in that: In step S2, when training the multi-task learning network, the evaluation index used for shoveling state recognition is the mAP value, and the evaluation index used for shoveling action recognition is the Accuracy value.

Images