Excavator working state recognition method based on deep learning

Abstract

The application discloses a kind of based on deep learning's excavator working condition identification method, comprising: S1.Collection overhead view site excavator image;S2.Establish target detection network, train the target detection network;S3.Recognize excavator body Body and machine arm Hand in each frame image, return the boundary box parameter of excavator body Body and machine arm Hand;S4.Calculate the tangent value of the center point rotation angle of each frame image excavator body Body and machine arm Hand area boundary box;S5.Calculate the absolute value of the difference of adjacent frame rotation angle tangent value, and compare with threshold value to identify the working condition of excavator.The working condition of the method can monitor site excavator, improve site safety level and management efficiency, strengthen the intelligent management of site.Excavator rotates when working Angle tangent value changes obviously, easy to judge the working condition of excavator, improve the accuracy of working condition identification, reduce false alarm rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of construction site supervision, and particularly relates to a method for recognizing working state of excavator based on deep learning. BACKGROUND

[0002] With the rapid development of China's economy, the scale and level of construction have developed to a new height, and there are many construction sites, construction equipment and construction personnel. However, the production operation environment of the construction site is complex, the site is relatively closed, the personnel are complex, and the problems of scattered construction sites and difficult construction site management are presented. Many safety accidents are caused by the loopholes in management, and there are many construction safety accidents causing personnel casualties every year in recent years.

[0003] The relevant departments of the state pay great attention to the safety of the construction site, and issue a series of policy documents to guarantee the safety of the construction site, and improve the intelligent degree of the construction site management in combination with the advanced information technology. In the current construction site management system, the engineering progress quality, personnel safety and environment are monitored and managed, and the monitoring and management of the mechanical equipment are relatively insufficient, especially in underground engineering, the excavator is one of the most important and most frequently used mechanical equipment in engineering construction, and it is very important to monitor the working state of the excavator. Therefore, it is urgent to develop a method for recognizing the working state of the excavator. SUMMARY

[0004] In view of the problem that the working state of the excavator is not effectively recognized in the prior art, which may cause safety accidents, the present application provides a method for recognizing the working state of the excavator based on deep learning.

[0005] To solve the above technical problems, the present application comprises the following technical solutions:

[0006] A method for recognizing the working state of the excavator based on deep learning, comprising the following steps:

[0007] S1. Collecting a top-view perspective construction site excavator image;

[0008] S2. Establishing a target detection network, and training the target detection network according to the excavator image with labeled target region boundary frame of the excavator body Body and the excavator arm Hand until the training parameters meet the requirements;

[0009] S3. Inputting the photographed construction site excavator video to be tested into the target detection network trained in step S2, recognizing the excavator body Body and the excavator arm Hand in each frame of image, and returning the boundary frame parameters of the excavator body Body and the excavator arm Hand;

[0010] S4. Calculate the tangent value of the rotation angle of the center point of the boundary box of the excavator body Body and the arm Hand in each frame of image according to the boundary box parameters of the target region obtained in step S3.

[0011] S5. Calculate the absolute value of the difference of the tangent value of the rotation angle between adjacent frames, and compare it with the threshold value to identify the working state of the excavator according to the tangent value of the rotation angle of each frame of image obtained in step S4.

[0012] Further, in step S2, a target detection network is established, and the target detection network is trained according to the excavator images with labeled boundary box parameters of the target region of the excavator body Body and the arm Hand until the training parameters meet the requirements, which specifically includes the following steps:

[0013] S21. Establish a sample database containing a plurality of overhead view construction site excavator images;

[0014] S22. Randomly divide the excavator images in the sample database into a training set and a test set according to a ratio of 9:1, and then label the boundary box parameters and the class of the target region of the excavator images in the training set, wherein the class includes the excavator body Body and the arm Hand;

[0015] S23. Establish a target detection network YOLOv3, and iteratively train the excavator images in the training set, and continuously adjust the structure and parameters of the network until the target detection network YOLOv3 after training is tested to meet the requirements through the test set of excavator images, and the trained target detection network is obtained.

[0016] Further, the boundary box parameters in step S3 include four-dimensional position parameters {x left ,x right ,y top ,y bottom} and a one-dimensional confidence parameter ω;

[0017] The position parameters are the x-coordinate x left of the left boundary, the x-coordinate x right of the right boundary, the y-coordinate y top of the upper boundary, and the y-coordinate y bottom of the lower boundary of the target region boundary box in the excavator image, wherein the coordinate origin is located at the upper left corner of the excavator image; the confidence parameter ω is the probability that the target region is of this class.

[0018] Further, step S4 calculates the tangent value of the rotation angle of the center point of the boundary box of the excavator body Body and the arm Hand in each frame of image, which specifically includes the following steps:

[0019] S41. Calculate the coordinates (X B ,Y B) and the center point coordinate (X H ,Y H ) of the Hand bounding box;

[0020] The calculation formula is as follows:

[0021] X B =(x Bleft +x Bright ) / 2;

[0022] Y B =(y Btop +y Bbottom ) / 2;

[0023] X H =(x Hleft +x Hright ) / 2;

[0024] Y H =(y Htop +y Hbottom ) / 2;

[0025] Wherein, {x Bleft ,x Bright ,y Btop ,y Bbottom} is the parameter of the Body bounding box, and {x Hleft ,x Hright ,y Htop ,y Hbottom} is the parameter of the Hand bounding box.

[0026] S42. Calculate the tangent value tanα of the rotation angle; the calculation formula is as follows:

[0027]

[0028] Further, step S5 specifically comprises the following steps:

[0029] S51. According to the tangent value results of the center point rotation angle of the Body and Hand region bounding box of each frame image, calculate the difference value between the current frame rotation angle tangent value and the last frame rotation angle tangent value, and the calculation formula is as follows:

[0030] Diftan=tanα-tanβ;

[0031] Wherein, Diftan is the difference value of the tangent value of the rotation angle between adjacent frames; tanα and tanβ are the tangent value of the current frame rotation angle and the tangent value of the last frame rotation angle, respectively.

[0032] S52. According to the difference Diftan of the adjacent frame inter-rotation tangent value obtained in the last step, compare its absolute value with the threshold TH to identify the working state of the excavator; when |Diftan| is greater than or equal to the threshold TH, the identification result is that the excavator is in the working state, otherwise the identification result is that the excavator is in the static state;

[0033] The working state identification formula is as follows:

[0034]

[0035] Wherein State represents the working state of the excavator in the current frame; move represents that the excavator is in the working state; static represents that the excavator is in the static state.

[0036] Compared with the prior art, the excavator working state recognition method based on deep learning provided in the present application has the following advantages and positive effects: the excavator working state recognition method based on deep learning provided in the present application can identify the excavator body and arm target regions in the excavator image according to the target detection network trained, and calculate the rotation tangent value according to the boundary box parameters of the target regions to obtain the rotation tangent value of each frame, and judge the working state of the excavator according to the absolute value of the difference between the rotation tangent values of adjacent frames and the threshold. Through the technical scheme provided in the present application, the working state of the construction site excavator can be monitored, the safety level and management efficiency of the construction site can be improved, and the intelligent management of the construction site can be strengthened. The present application calculates the tangent value of the angle formed by the line connecting the center points of the excavator body boundary box and the arm boundary box and the vertical line, which has obvious changes when the excavator is working, and is easy to judge the working state of the excavator, improve the accuracy of working state recognition, and reduce the false alarm rate. By calculating the absolute value of the difference between the rotation tangent values of adjacent frames and comparing it with the set threshold, the situation that the working state is misjudged due to the slight changes (offset, scaling, etc.) of the boundary box of the excavator body and arm region caused by the accuracy of the target detection network can be reduced, and the working state of the excavator can be more accurately identified. The present application provides a method for training and identification using the target detection network YOLOv3 to improve the efficiency of training and identification. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The flow chart of the excavator working state recognition method based on deep learning in an embodiment of the present application;

[0038] Figure 2 The construction site excavator image collected;

[0039] Figure 3 The excavator image with the boundary box of the excavator body and arm target region labeled; Figure 2 The excavator image with the boundary box of the excavator body and arm target region labeled;

[0040] Figure 4 is an embodiment of the present application using a target detection network to identify the excavator body Body and arm Hand in the image;

[0041] Figure 5 is an example image of the defined rotation angle in an embodiment of the present application;

[0042] Figure 6 is a result visualization image of the working state recognition according to the tangent value change of the rotation angle in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application provides a kind of based on deep learning's excavator working state recognition method, which is further detailed below in conjunction with the drawings and specific embodiments. The advantages and features of the present application will be clearer in conjunction with the following description. It should be noted that the drawings are very simplified and all use non-precise scale, only to facilitate, clearly assist the purpose of explaining the embodiments of the present application.

[0044] The present application provides a kind of based on deep learning's excavator working state recognition method, Figure 1 is a flow chart of the excavator working state recognition method based on deep learning in an embodiment of the present application, as Figure 1 The method comprises the following steps:

[0045] S1. Collect overhead view construction site excavator image;

[0046] S2. Establish target detection network, and train the target detection network according to the labeled excavator body Body and arm Hand target area bounding box of excavator image until the training parameters meet the requirements;

[0047] S3. The construction site excavator video to be measured photographed by unmanned aerial vehicle is input into the target detection network trained in step S2, the excavator body Body and arm Hand in each frame image are identified, and the boundary box parameters of the excavator body Body and arm Hand are returned;

[0048] S4. According to the boundary box parameters of the target area obtained in step S3, the tangent value of the rotation angle of the center point of the boundary box of the excavator body Body and arm Hand region of each frame image is calculated;

[0049] S5. According to the tangent value of the rotation angle of each frame image obtained in step S4, the absolute value of the difference of the tangent value of the rotation angle between adjacent frames is calculated, and compared with the threshold value to identify the working state of the excavator.

[0050] The excavator working state recognition method based on deep learning provided in the application can identify the excavator body and arm target regions in the excavator image according to the target detection network trained, and calculate the rotation angle tangent value according to the boundary box parameters of the target regions to obtain each frame of rotation angle tangent value, and judge the working state of the excavator according to the absolute value of the difference between the rotation angle tangent values of adjacent frames and the threshold value. Through the technical scheme provided in the application, the working state of the construction site excavator can be monitored, the safety level and management efficiency of the construction site can be improved, and the intelligent management of the construction site can be strengthened.

[0051] The target detection network is established in step S2, and the existing convolutional neural network training method can be used for training to obtain different weight values, which will not be described again. In order to improve the training and recognition efficiency, a preferred scheme is given in the embodiment, and step S2 specifically includes the following steps:

[0052] S21. Establish a sample database containing a plurality of overhead view construction site excavator images, as shown in FIG. 2; in order to improve the accuracy of recognition, the number of overhead view construction site excavator images can be provided as needed, and a large number of provided images will be beneficial to improve the accuracy of recognition. Figure 2

[0053] S22. Randomly divide the excavator images in the sample database into a training set and a test set according to a ratio of 9:1, and then label the boundary box parameters and categories of the target regions in the excavator images in the training set, wherein the categories include excavator body Body and arm Hand, as shown in FIG. 3; in order to improve the accuracy of recognition, the number of overhead view construction site excavator images can be provided as needed, and a large number of provided images will be beneficial to improve the accuracy of recognition. Figure 3

[0054] S23. Establish a target detection network YOLOv3, and iteratively train the excavator images in the training set, and continuously adjust the structure and parameters of the network until the training parameters meet the requirements, and obtain the trained target detection network.

[0055] The target detection network YOLOv3 is prior art, and step S23 of the embodiment only applies the existing target detection network YOLOv3 to iteratively train the excavator images, and then uses the trained target detection network YOLOv3 to recognize the to-be-tested video and output the boundary box parameters of the target region through steps S24 and S25. In order to facilitate understanding, the application will make a simple introduction to YOLOv3. The target detection network YOLOv3 includes 52 convolutional layers and 1 fully connected layer, also known as Darknet53.

[0056] In a specific embodiment, the specific steps of the above steps S21 to S23 are as follows:

[0057] ​​A sample database of excavator images is established, including 506 mass excavator working images collected from a construction site by a UAV; the original image size of the excavator is m×n=640×368;

[0058] The excavator images in the sample database are randomly divided into a training set and a test set according to a ratio of 9:1, and the boundary box parameters and the class of the target region of the excavator images in the training set are labeled, wherein the class includes an excavator body Body and an excavator arm Hand, wherein the excavator arm Hand region is the region where the operating arm and the bucket are located, and the excavator body Body region is the remaining excavator region excluding the excavator arm region, and the excavator arm region usually rotates with the excavator body;

[0059] A target detection network YOLOv3 is established, and the excavator images in the training set are iteratively trained, and the structure and parameters of the network are continuously adjusted until the average loss value loss of the training is less than 10 and no longer obviously decreases, and a trained target detection network is obtained.

[0060] In an embodiment, step S3 specifically comprises: inputting the to-be-detected construction site excavator video into the trained target detection network for identification, as shown in Figure 4 , the excavator body Body and the excavator arm Hand target region boundary box are identified, and the boundary box parameters of each frame target region are output, including four-dimensional position parameters {x left ,x right ,y top ,y bottom} and a one-dimensional confidence parameter ω. As an example, the parameters of the excavator body Body region boundary box identified in the previous frame excavator image are {241, 356, 126, 188}, and the confidence is 0.80; the parameters of the excavator arm Hand region boundary box are {318, 486, 139, 174}, and the confidence is 0.94; the current frame detection result, the parameters of the excavator body Body region boundary box are {256, 360, 101, 194}, and the confidence is 0.97; the parameters of the excavator arm Hand region boundary box are {316, 403, 155, 333}, and the confidence is 0.37.

[0061] The optional embodiments of step (4) of the present application will be described in detail below. Figure 6

[0062] In an embodiment, step S4 calculates the tangent value of the rotation angle of the center point of the excavator body Body and the excavator arm Hand region boundary box of each frame image, as shown in Figure 5 , and specifically comprises the following steps:

[0063] S41. Calculate the coordinates (X B ,Y​B ) and the center point coordinate (X H ,Y H ) of the arm Hand bounding box;

[0064] The calculation formula is as follows:

[0065] X B =(x Bleft +x Bright ) / 2;

[0066] Y B =(y Btop +y Bbottom ) / 2;

[0067] X H =(x Hleft +x Hright ) / 2;

[0068] Y H =(y Htop +y Hbottom ) / 2;

[0069] Wherein, {x Bleft ,x Bright ,y Btop ,y Bbottom} is the parameter of the excavator body Body bounding box, {x Hleft ,x Hright ,y Htop ,y Hbottom} is the parameter of the excavator arm Hand bounding box.

[0070] S42. Calculate the tangent value tanα of the rotation angle; Figure 5 is an example image of the rotation angle defined in the embodiment of the application, that is, the included angle formed by the center point connecting line of the two bounding boxes in each frame image and the vertical line. According to the calculated center point parameters of the bounding box, the tangent value of the rotation angle can be calculated, and the calculation formula is as follows:

[0071]

[0072] In an embodiment, the parameter of the excavator body Body region bounding box in the previous frame image is {241, 356, 126, 188}, and the confidence is 0.80; the parameter of the excavator arm Hand region bounding box is {318, 486, 139, 174}, and the confidence is 0.94; the center point coordinate (X B ,Y B ) of the excavator body bounding box is (298.5, 157), and the center point coordinate (X H ,Y HThe coordinates of the center point of the excavator body region are (402, 156.5), and the tangent of the rotation angle tanα is -207.00. In the current frame image, the parameters of the bounding box of the excavator body region are {256, 360, 101, 194}, with a confidence level of 0.97; the parameters of the bounding box of the excavator arm region are {316, 403, 155, 333}, and the coordinates of the center point of the excavator body bounding box (X...) are... B ,Y B The coordinates of the center point of the excavator boom bounding box are (308, 147.5). H ,Y H The value of the rotation angle is (359.5, 244); the tangent of the rotation angle is tanβ, which is 0.53.

[0073] In one embodiment, step S5 specifically includes the following steps:

[0074] S51. Based on the tangent value of the rotation angle of the center point of the excavator body and arm / hand region bounding box in each frame, calculate the difference between the tangent value of the rotation angle in the current frame and the tangent value of the rotation angle in the previous frame. The calculation formula is as follows:

[0075] Diftan = tanα - tanβ;

[0076] Where Diftan is the difference in the tangent of the rotation angle between adjacent frames; tanα and tanβ are the tangent of the rotation angle of the current frame and the tangent of the rotation angle of the previous frame, respectively, so Diftan = -207.53;

[0077] S52. Based on the Diftan result of the difference between the rotation angle tangent values ​​between adjacent frames obtained in the previous step, compare its absolute value with the threshold TH to identify the working state of the excavator; when |Diftan| is greater than or equal to the threshold TH, the identification result is that the excavator is in a working state, otherwise the identification result is that the excavator is in a stationary state.

[0078] The formula for identifying the working status is as follows:

[0079]

[0080] Where State represents the excavator's working state in the current frame; move indicates that the excavator is in a working state; static indicates that the excavator is in a stationary state. For example, in this embodiment, the threshold TH = 0.16. The current frame's excavator working state is determined according to the state recognition formula: State = move. Figure 6 This is a visualized image of the result of calculating the change in the rotation angle tangent and identifying the working state of the video under test according to an embodiment of the present invention. Figure 6The current frame excavator body and arm region detection frame, detection frame confidence, current frame rotation angle tangent value and current frame excavator working state recognition result are displayed.

[0081] After the processing of the steps, the results of the detection frame, detection frame confidence and current frame rotation angle tangent value of the excavator body and arm region of each frame image of the video to be tested are obtained, and the working state of the excavator is grasped in real time, so that the construction site supervision department can have real-time understanding of the construction site construction status and improve the intelligent management degree of the construction site.

[0082] A working cycle of the excavator includes four basic actions of excavation, lifting and rotating, unloading and rotating back, and the overall work state is a rotating state in place, so the detection frame size of the excavator identified by the construction site monitoring video taken by the unmanned aerial vehicle changes little, and the center point of the detection frame hardly changes, and there is a large error in identifying the working state of the excavator according to the detection frame size change or the detection frame center point change. Therefore, in this paper, the body and arm of the excavator are detected respectively, and the angle change between the detection frames of the two is used to extract the rotating feature of the excavator, so as to realize real-time identification of the working state of the excavator and improve the safety of the construction site and strengthen the intelligent management of the construction site.

[0083] The technical features of the above-described embodiments can be combined arbitrarily, and to make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0084] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for recognizing a working state of a shovel based on deep learning, characterized by, The method comprises the following steps: S1. Collecting a top-view construction site excavator image; S2. Establishing a target detection network, and training the target detection network according to excavator images with labeled target region bounding boxes of the excavator body Body and the excavator arm Hand until the training parameters meet the requirements; S3. Inputting a captured construction site excavator video to be tested into the target detection network trained in step S2, identifying the excavator body Body and the excavator arm Hand in each frame of image, and returning the bounding box parameters of the excavator body Body and the excavator arm Hand; S4. Calculating the tangent value of the rotation angle of the center point of the bounding box of the excavator body Body and the excavator arm Hand region in each frame of image according to the bounding box parameters of the target region obtained in step S3; S5. Calculating the absolute value of the difference of the tangent values of the rotation angles between adjacent frames according to the tangent values of the rotation angles of each frame of image obtained in step S4, and comparing the absolute value with a threshold value to identify the working state of the excavator.

2. The excavator working state identification method based on deep learning according to claim 1, wherein the target detection network is established in step S2, and the target detection network is trained according to excavator images with labeled target region bounding boxes of the excavator body Body and the excavator arm Hand until the training parameters meet the requirements, and the step specifically comprises the following steps: S21. Establishing a sample database containing a plurality of top-view construction site excavator images; S22. Randomly dividing the excavator images in the sample database into a training set and a test set according to a ratio of 9:1, labeling the bounding box parameters and the categories of the target regions in the excavator images in the training set, and the categories include the excavator body Body and the excavator arm Hand; S23. Establishing a target detection network YOLOv3, iteratively training the excavator images in the training set, constantly adjusting the structure and parameters of the network, until the target detection network YOLOv3 is tested by the excavator images in the test set and meets the requirements, and the trained target detection network is obtained.

3. The excavator working state identification method based on deep learning according to claim 1, wherein 4. The excavator working state identification method based on deep learning according to claim 1, wherein The bounding box parameters in step S3 comprise four-dimensional position parameters {x left ,y right ,y top ,y bottom} and a one-dimensional confidence parameter ω; The position parameter is the x coordinate x of the left boundary of the target region bounding box in the excavator image left , the x coordinate x of the right boundary right , the y coordinate y of the upper boundary top , and the y coordinate y of the lower boundary bottom , wherein the coordinate origin is located at the upper left corner of the excavator image; and the confidence parameter ω is the probability that the target region is of this category. The step S4 of calculating the tangent value of the rotation angle of the center point of the bounding box of the excavator body Body and the excavator arm Hand region in each frame of image specifically comprises the following steps: The calculation formula is as follows: S41. Calculate the center point coordinates (X B ,Y B ) of the excavator body boundary box Body and the center point coordinates (X H ,Y H ) of the arm boundary box Hand; S42. Calculating the tangent value tanα of the rotation angle; the calculation formula is as follows: X B = (x Bleft + x Bright ) / 2; Y B = (y Btop + y Bbottom ) / 2; X H = (x Hleft + x Hright ) / 2; Y H = (y Htop + y Hbottom ) / 2; wherein {x Bleft ,x Bright ,y Btop ,y Bbottom} are parameters of the boundary box of the excavator body Body, and {x Hleft ,x Hright ,y Htop ,y Hbottom} are parameters of the boundary box of the excavator arm Hand.

5. The excavator working state identification method based on deep learning according to claim 4, wherein The step S5 specifically comprises the following steps: S51. Calculating the difference between the tangent value of the rotation angle of the current frame and the tangent value of the rotation angle of the previous frame according to the tangent value of the rotation angle of the center point of the bounding box of the excavator body Body and the excavator arm Hand region in each frame of image, and the calculation formula is as follows: Diftan=tanα-tanβ; wherein Diftan is the difference between the tangent values of the rotation angles between adjacent frames; tanα and tanβ are the tangent values of the rotation angles of the current frame and the previous frame, respectively. ​ S52. According to the difference Diftan of the adjacent frame inter-rotation angle tangent value obtained in the last step, compare its absolute value with the threshold TH to identify the working state of the excavator; when |Diftan| is greater than or equal to the threshold TH, the identification result is that the excavator is in the working state, otherwise the identification result is that the excavator is in the static state; The working state identification formula is as follows: Wherein State represents the working state of the excavator in the current frame; move represents that the excavator is in the working state; static represents that the excavator is in the static state.

Images