Excavator operation pose monitoring method, system, equipment and medium

Abstract

The invention discloses an excavator operation pose monitoring method, system and equipment and a medium, and relates to the field of excavators. The method comprises the following steps: processing a video by using a rotating target detection model to obtain four-dimensional coordinates and a rotating angle of a forearm; the first distance between the tail end of the forearm and the cab is obtained according to the four-dimensional coordinates and the rotation angle, the first height of the tail end of the forearm is calculated according to the width and the vertical coordinates of the tail end of the forearm in the image frame of the video in combination with the height from the camera to the ground and the first distance, and the second distance and the second height are obtained through correction of the gradienter; working point coordinates of the tail end of the forearm are obtained according to cab positioning information, azimuth angle information, the second height and the second distance; when the working state of the excavator bucket is changed, working information is recorded, and whether the state of the excavator is normal or not is detected according to the working information. By implementing the technical scheme provided by the invention, the precision and reliability of monitoring the operation pose of the excavator are improved.

Description

Technical Field

[0001] The present application relates to the technical field of excavators, and in particular to a method, system, equipment and medium for monitoring the working posture of an excavator. Background Art

[0002] With the rapid development of the construction industry and technological progress, excavators, as an important type of engineering machinery, have been widely used in earthwork excavation, road construction and other fields. In order to improve the dynamic monitoring capability of excavator operations and at the same time improve the operating efficiency and safety of excavators, accurate monitoring of their operating posture is particularly important.

[0003] Existing excavator working posture monitoring technology mainly relies on image processing technology and sensor data fusion algorithm. The above methods still have some shortcomings in practical applications. For example, due to the complex and changeable working environment of the excavator, the traditional image processing technology and sensor data fusion algorithm are prone to false detection and missed detection when monitoring and identifying the telescopic rotation of the large and small arms of the excavator, resulting in low accuracy of working posture monitoring.

[0004] Therefore, a more accurate method for monitoring the working posture of the excavator is needed. Summary of the invention

[0005] The present application provides a method, system, equipment and medium for monitoring the working posture of an excavator. The excavator arm is subjected to target detection through rotating target detection, thereby achieving accurate framing of the forearm, thereby improving the accuracy of calculating the forearm width in the image, and then accurately estimating the distance between the forearm and the visual sensor, thereby improving the accuracy of monitoring the working posture of the excavator.

[0006] In a first aspect of the present application, a method for monitoring an excavator's working posture is provided, which is applied to an excavator's working posture monitoring platform, wherein the excavator's working posture monitoring platform comprises a camera, a level, a global navigation satellite system device and an electronic compass installed in a cab of the excavator, and the method comprises: The camera is used to collect a video of the bucket, the forearm and the excavation surface, and the video is processed using a rotating target detection model to obtain the four coordinates of the forearm, the rotation angle and the state of the bucket; Obtaining a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle, calculating a first height of the end of the forearm according to the width and vertical coordinates of the end of the forearm in the image frame of the video, in combination with the height of the camera from the ground and the first distance, correcting the first distance by the level meter to obtain a second distance, and correcting the first height by the level meter to obtain a second height; The cab positioning information is obtained through the global navigation satellite system device, the azimuth information is obtained through the electronic compass, and the coordinates of the working point of the end of the forearm are obtained according to the cab positioning information, the azimuth information, the second height and the second distance; When the working state of the bucket changes, the operation information is recorded, and the operation information includes the coordinates of the operation point, the time point and the real-time working state. The excavator state is detected according to the operation information, that is, the key state of the loading and unloading points is identified.

[0007] Optionally, obtaining a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle includes: Determine the right edge of the forearm, and determine the positional relationship between the right edge and the vertical line; When the right edge is located in the clockwise direction of the vertical line, a first distance between the end of the forearm and the cab is calculated based on a first preset formula according to the first actual width of the top of the forearm, the rotation angle between the right edge of the forearm and the vertical line, and the coordinates of the two vertices of the top of the forearm in the rotation frame; When the right edge is located in the counterclockwise direction of the vertical line, the first distance between the end of the forearm and the cab is calculated based on the second preset formula according to the second actual width of the end of the forearm and the coordinates of the two vertices of the end of the forearm in the rotation frame.

[0008] Optionally, the first preset formula is as follows: ; Among them, a, b and c are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d0 is the first actual width of the top of the arm, L is the length of the arm, is the rotation angle between the right edge of the forearm and the vertical line, D is the first distance between the end of the forearm and the cab, (X a , Ymin1) and (X b , Ymin2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X a , Ymin1) is the coordinate with the smallest ordinate among the two vertex coordinates, The second preset formula is as follows: ; Among them, k and t are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d1 is the second actual width of the end of the arm, (X c , Ymax1) and (X d , Ymax2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X c, Ymax1) is the coordinate with the largest ordinate among the two vertex coordinates.

[0009] Optionally, calculating the first height of the forearm end according to the width and the vertical coordinate of the forearm end in the image frame of the video, in combination with the height of the camera from the ground and the first distance, comprises: Calculate the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body according to the first distance; The first height of the forearm end is calculated based on a third preset formula according to the first vertical coordinate, the second vertical coordinate of the forearm end in the image frame, the width of the forearm end in the image frame and the second actual width of the forearm end.

[0010] Optionally, the third preset formula is as follows: ; Wherein, H represents the first height of the end of the forearm, h represents the height of the camera from the ground, and y t is the second vertical coordinate of the forearm end in the image frame, y m It represents the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body, w m represents the width of the end of the forearm in the image frame, and d1 represents the second actual width of the end of the forearm.

[0011] Optionally, the correcting the first distance by the level meter to obtain a second distance, and correcting the first height by the level meter to obtain a second height includes: The first distance is corrected by the following formula: ; Among them, D z represents the second distance, D represents the first distance, Indicates the inclination angle between the level and the horizontal plane. The first height is corrected by the following formula: ; Among them, H z represents the second height, H represents the first height, Indicates the inclination angle of the level instrument to the horizontal plane.

[0012] Optionally, obtaining the coordinates of the working point of the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance includes: The coordinates of the working point are determined by the following formula: ; Among them, (X0, Y0, Z0) represents the cab positioning information, Indicates azimuth information, D z Indicates the second distance, H z Indicates the second height.

[0013] In a second aspect of the present application, a monitoring system for an excavator working posture is provided, comprising a video acquisition module, a first calculation module, a second calculation module and a status monitoring module, wherein: A video acquisition module is configured to acquire a video of the bucket, the forearm and the excavation surface through a camera, and process the video using a rotating target detection model to obtain the four coordinates of the forearm, the rotation angle and the state of the bucket; a first calculation module, configured to obtain a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle, calculate a first height of the end of the forearm according to the width and vertical coordinates of the end of the forearm in the image frame of the video, in combination with the height of the camera from the ground and the first distance, correct the first distance by a level meter to obtain a second distance, and correct the first height by the level meter to obtain a second height; a second calculation module, configured to obtain cab positioning information through a global navigation satellite system device, obtain azimuth information through an electronic compass, and obtain the coordinates of the operating point of the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance; The status monitoring module is configured to record operation information when the working state of the bucket changes, the operation information including the coordinates of the operation point, the time point and the latest working state, and detect whether the excavator state is normal based on the operation information.

[0014] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0015] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0016] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The video of the bucket, arm and excavation surface is collected by the camera and processed by the rotating target detection model to accurately obtain the position and status of the arm and bucket, which helps the driver or remote monitoring personnel to accurately understand the operation of the excavator, thereby improving the operation accuracy and safety; 2. Combining the data of the level, global navigation satellite system equipment and electronic compass, it can achieve accurate three-dimensional spatial positioning of the operating point at the end of the forearm. This helps to accurately judge the operating position and posture of the excavator in complex working environments such as construction sites or mines; 3. When the working state of the bucket changes, the system will record the operation information, including the coordinates of the operation point, time point and the latest working state. This information can be used to monitor the working state of the excavator in real time and issue an early warning when an abnormality or potential failure occurs, thereby avoiding accidents. 4. By recording operation information, it can also provide a basis for subsequent data analysis and mining. For example, it can analyze the operating efficiency, energy consumption, failure frequency, etc. of the excavator, providing a scientific basis for the maintenance and management of the excavator; 5. Combining data from multiple sensors and devices, it can adapt to different operating environments and operating requirements. At the same time, by processing video data through the rotating target detection model, it can realize the operating posture monitoring of excavators of different models and specifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a method for monitoring the working posture of an excavator disclosed in an embodiment of the present application; Figure 2 is a schematic diagram of the training effect evaluation of the bucket and arm rotating target detection model disclosed in the embodiment of the present application; Figure 3 It is a diagram of the detection effect of the forearm and a schematic diagram of the calculation of the width of the forearm in the image when the end of the forearm is extended to the far end during operation of the excavator disclosed in the embodiment of the present application; Figure 4 It is a diagram of the detection effect of the forearm and a schematic diagram of the calculation of the width of the forearm in the image when the end of the forearm of the excavator disclosed in the embodiment of the present application is retracted to the proximal end during operation; Figure 5 It is a schematic diagram of an excavator bucket in an empty state disclosed in an embodiment of the present application; Figure 6 It is a schematic diagram of the loading state of the excavator bucket disclosed in the embodiment of the present application; Figure 7 It is a module schematic diagram of the monitoring system for the working posture of an excavator disclosed in an embodiment of the present application; Figure 8 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0018] Explanation of the accompanying drawings: 701, video acquisition module; 702, first calculation module; 703, second calculation module; 704, status monitoring module; 801, processor; 802, communication bus; 803, user interface; 804, network interface; 805, memory. DETAILED DESCRIPTION

[0019] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0020] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0021] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0022] This embodiment discloses a method for monitoring the working posture of an excavator, which is applied to an excavator working posture monitoring platform. The excavator working posture monitoring platform includes a camera, a level, a global navigation satellite system device and an electronic compass installed in a cab of the excavator. Figure 1 is a flow chart of the method for monitoring the working posture of an excavator disclosed in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S110, collecting a video of the bucket, the forearm, and the excavation surface through the camera, and processing the video using a rotating target detection model to obtain the four coordinates of the forearm, the rotation angle, and the state of the bucket; A large number of images are collected in the cab of the excavator through the camera for labeling. When selecting excavator videos or images, the images or pictures of the excavator in various states such as loading, rotation, unloading, and no-load are taken into consideration to establish a multi-model and multi-state original image set. The collected images or image data sets are labeled, and the labeling software is selected to label the bucket and arm of the original image, that is, the range of the arm, the no-load state and the loaded state of the bucket are labeled separately by rotating the rectangular frame. The collected bucket and arm image data are divided into independent and non-repetitive verification sets and test sets according to a certain proportion by random sampling. Before the model training begins, the images are uniformly processed, the image size is unified to 640×640, the Mosaic data enhancement processing is uniformly performed, and the classification of the forearm prior frame is obtained by clustering analysis. The classification of the forearm prior frame is clustered by the length and width information of the forearm annotation frame formed in the annotation process, and is divided into three categories of frames. The average length and width data of each category of frames are calculated respectively, and the adaptive anchor frame is obtained as [8, 11; 15, 32; 27, 41; 30, 61; 62, 53; 62, 124; 104, 108; 144, 206; 363, 322]. In order to obtain faster convergence speed and reduce the training time cost, this embodiment performs transfer learning based on the YOLOv5s pre-trained weights under the YOLOv5 framework. Among them, the hardware platform parameters are the central processor of Intel i7-7700k, the graphics computing card of Nvidia Ge Force GTX 1060Ti, the batch number is 16, and the training is 120 rounds, with a total of 92160 iterations. The labeled data and images are loaded into the rotation target detection model for training. The rotation target detection model mainly trains the position and size of the labeled forearm. The loss function of the training output layer is R-CIoU Loss, which is the complete intersection-over-union ratio of rotation. An angle penalty term is added (R-CIoU Loss, which takes into account the center point distance and shape information, can more accurately measure the gap between the predicted box and the real box, including the overlap rate, center point distance, border shape and rotation angle), as well as the mAP curve. The weight file pt is trained and obtained. Figure 2 It is a schematic diagram of the training effect evaluation of the bucket and arm rotating target detection model disclosed in the embodiment of the present application.

[0023] The camera is installed in the excavator cab to capture real-time video of the bucket, arm, and excavation surface. The video will serve as the basis for subsequent processing and analysis. The camera needs to have sufficient resolution and frame rate to ensure that the various movements of the bucket and arm and the state of the excavation surface can be clearly captured. The collected video data will be input into the rotating object detection model. The rotating object detection model is trained by the above steps. The rotating object detection model can accurately identify the bucket and arm in the video and calculate their position and posture. Specifically, the rotating object detection model processes each frame in the video. For each frame of the image, the rotating object detection model detects the position and size of the arm: the range of the arm is marked by a rotating rectangular box. The four vertex coordinates of this rectangular box ([X1, Y1], [X2, Y2], [X3, Y3], [X4, Y4]) will represent the position of the arm in the image. At the same time, the model will also calculate the rotation angle of the arm to represent its posture. The rotating object detection model also identifies the state of the bucket: the model determines whether the bucket is in an empty or loaded state. This is achieved by annotating the bucket image data and training the model. After processing the rotating object detection model, the following information can be obtained: The four coordinates of the forearm: These four coordinate points will accurately represent the position and shape of the forearm in the image.

[0024] Rotation Angle: This angle indicates the degree of rotation of the arm relative to the image plane, which helps to more accurately understand the working posture of the excavator.

[0025] Bucket status: This information determines whether the bucket is empty or loaded with material, which is very important for evaluating the operating efficiency and status of the excavator.

[0026] S120, obtaining a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle, calculating a first height of the end of the forearm according to the width and vertical coordinate of the end of the forearm in the image frame of the video, in combination with the height of the camera from the ground and the first distance, correcting the first distance by the level meter to obtain a second distance, and correcting the first height by the level meter to obtain a second height; The focal length of the camera, the size of the image sensor, and the distance between the camera and the object together determine the size of the object in the image. Therefore, this information can be used in combination with known camera parameters (such as focal length, sensor size, etc.) to calculate the actual distance between the end of the arm and the cab through a proportional relationship. After obtaining the vertical coordinate (i.e., Y coordinate) of the end of the arm in the image, the first height of the end of the arm can be estimated by combining the width of the end of the arm in the image (which can be obtained by calculating the number of pixels at the end of the arm and multiplying it by the actual size represented by each pixel) and the height of the camera from the ground. The key here is to understand the proportional relationship between the size in the image and the actual size. Since the camera is fixed in the cab, the height of the camera from the ground is known. By comparing the width of the end of the arm in the image with its actual width, the actual size represented by each pixel in the image can be calculated. Then, using this proportional relationship and the vertical coordinate, the first height of the end of the arm can be estimated. Due to possible installation errors or tilts of the camera, as well as possible errors in the image processing process, the first distance and the first height calculated above may have certain deviations. In order to correct these deviations, a level can be used to assist in the measurement. A level is a tool that can measure the inclination angle of an object. By placing it at a fixed position in the cab and reading the inclination angle it displays, the degree of inclination of the cab relative to the horizontal plane can be obtained. Then, this inclination angle can be used to correct the first distance and the first height, thereby obtaining a more accurate second distance and second height.

[0027] Optionally, obtaining a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle includes: Determine the right edge of the forearm, and determine the positional relationship between the right edge and the vertical line; When the right edge is located in the clockwise direction of the vertical line, a first distance between the end of the forearm and the cab is calculated based on a first preset formula according to the first actual width of the top of the forearm, the rotation angle between the right edge of the forearm and the vertical line, and the coordinates of the two vertices of the top of the forearm in the rotation frame; When the right edge is located in the counterclockwise direction of the vertical line, the first distance between the end of the forearm and the cab is calculated based on the second preset formula according to the second actual width of the end of the forearm and the coordinates of the two vertices of the end of the forearm in the rotation frame.

[0028] In the rotating target detection image, the right edge of the forearm is first identified and its position relative to the gravity line is determined. The gravity line is a straight line perpendicular to the ground, which appears as a straight line perpendicular to the bottom edge of the image in the image. By comparing the position of the right edge of the forearm with the gravity line, the rotation direction of the forearm can be determined. When the right edge of the forearm is located in the clockwise direction of the gravity line, the rotation angle is set to be positive. This means that the end of the forearm (connected to the bucket end) is away from the cab direction. When the right edge of the forearm is located in the counterclockwise direction of the gravity line, the rotation angle is set to be negative. This means that the end of the forearm (connected to the bucket end) is close to the cab direction. When the rotation angle is positive, due to the rotation of the forearm, the width of the forearm top (connected to the boom end) detected by the rotating target detection frame is usually consistent with the width in the actual image, while the width detected at the end of the forearm (connected to the bucket end) may be greater than the actual width. Therefore, when calculating the first distance, it mainly depends on the width of the forearm top, the rotation angle between the right edge of the forearm and the gravity line, and the two vertex coordinates of the forearm top in the rotation frame. When the rotation angle is negative, the width of the forearm end (connected to the bucket end) detected by the rotating target detection frame usually matches the width in the actual image, while the width detected at the top of the forearm (connected to the boom end) may be greater than the actual width. Therefore, when calculating the first distance, it mainly depends on the width of the forearm end, the rotation angle between the right edge of the forearm and the vertical line, and the coordinates of the two vertices of the forearm end in the rotation frame.

[0029] By determining the positional relationship between the right edge of the arm and the vertical line, and using the rotation angle and width information, the actual distance between the end of the arm and the cab can be calculated more accurately. It can automatically adapt to different models and specifications of excavators, without the need for individual calibration or adjustment for each excavator. At the same time, it can also adapt to different working environments and light conditions to ensure accurate measurement results in all situations. Operators can obtain distance information between the end of the arm and the cab more quickly, so as to plan the working path more efficiently and avoid potential dangers. Accurate measurement helps operators better understand the position and posture of the excavator, thereby avoiding collisions with obstacles or people during operation.

[0030] Optionally, the first preset formula is as follows: ; Among them, a, b and c are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d0 is the first actual width of the top of the arm, L is the length of the arm, is the rotation angle between the right edge of the forearm and the vertical line, D is the first distance between the end of the forearm and the cab, (X a , Ymin1) and (X b , Ymin2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (Xa , Ymin1) is the coordinate with the smallest ordinate among the two vertex coordinates, The second preset formula is as follows: ; Among them, k and t are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d1 is the second actual width of the end of the arm, (X c , Ymax1) and (X d , Ymax2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X c , Ymax1) is the coordinate with the largest ordinate among the two vertex coordinates.

[0031] The test equipment of the embodiment of the present application is a Sany 75c model, the height of the installed camera from the ground is 2560mm, the length of the excavator arm is 3.5m, and the first actual width of the top of the arm is 0.24m. When the right edge of the arm is located in the clockwise direction of the vertical line (the rotation angle is positive), a series of data are recorded: ; Substituting the data in the above table into the first preset formula, the least square method is used to solve the correlation coefficients a, b and c, and the results are a=323.29, b=0.412, c=1.58, that is, when the right edge of the arm is located in the clockwise direction of the vertical line, the end of the arm (bucket end) is away from the cab direction, and the distance calculation formula corresponding to the Sany 75c model is: ; Figure 3 This is a diagram of the detection effect of the forearm and the calculation diagram of the width of the forearm in the image when the end of the forearm is extended to the far end during the operation of the excavator disclosed in the embodiment of the present application, such as Figure 3 As shown in the figure, (X1, Y1) and (X4, Y4) are the coordinates of the two vertices at the top of the forearm in the rotation frame, (X a , Ymin1) can be (X4, Y4), (X b , Ymin2) can be (X1, Y1).

[0032] Similarly, the second actual width at the end of the jib is 0.20m. When the right edge of the jib is located counterclockwise from the vertical line, a series of data are recorded: ; Substitute the data in the above table into the second preset formula, and solve the correlation coefficients k and t by the least square method, and get k=639.72, b=-0.0367, that is, when the right edge of the arm is located in the counterclockwise direction of the vertical line, the end of the arm (bucket end) is close to the cab direction, and the distance calculation formula corresponding to the Sany 75c model is: ; Figure 4 This is a diagram of the detection effect of the forearm and a schematic diagram of the calculation of the width of the forearm in the image when the end of the forearm of the excavator is retracted to the proximal end during operation disclosed in the embodiment of the present application, such as Figure 4 As shown in the figure, (X2, Y2) and (X3, Y3) are the coordinates of the two vertices at the end of the forearm in the rotation frame, (X c , Ymax1) can be (X3, Y3), (X d , Ymax2) can be (X2, Y2) Optionally, calculating the first height of the forearm end according to the width and the vertical coordinate of the forearm end in the image frame of the video, in combination with the height of the camera from the ground and the first distance, comprises: Calculate the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body according to the first distance; The first height of the forearm end is calculated based on a third preset formula according to the first vertical coordinate, the second vertical coordinate of the forearm end in the image frame, the width of the forearm end in the image frame and the second actual width of the forearm end.

[0033] Calculated based on similar triangles, that is ; where w m is the width of the forearm end in the image. When the right edge of the forearm is located in the clockwise direction of the vertical line, the width of the forearm end in the image is expressed as |X a -X b |, when the right edge of the forearm is located in the counterclockwise direction of the vertical line, the width of the forearm end in the image is represented by |X c -X d |; d1 is the second actual width at the end of the forearm, y t is the vertical coordinate (second vertical coordinate) of the end of the forearm (bucket end) in the image, y m It indicates the vertical coordinate (first vertical coordinate) of the end of the forearm on the image when the bucket is placed on the horizontal plane of the chassis. Indicates the height of the forearm end relative to the camera. The first height can be determined by the difference between the height of the camera from the ground and the height of the forearm end relative to the camera.

[0034] Optionally, the third preset formula is as follows: ; Wherein, H represents the first height of the end of the forearm, h represents the height of the camera from the ground, and y t is the second vertical coordinate of the forearm end in the image frame, y m It represents the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body, w m represents the width of the end of the forearm in the image frame, and d1 represents the second actual width of the end of the forearm.

[0035] For the same excavator and fixed camera, when the arm end (bucket end) is at different distances from the cab, the corresponding coordinates when the arm end is placed on the horizontal plane of the chassis will change. Continuous measurement and calibration are required to obtain the corresponding camera to arm end (bucket end) D and y m The relationship is as follows: ; The first distance D is obtained by the first preset formula or the second preset formula, and y is obtained by the calibration data. m Substituting the value into the third preset formula can obtain the first height H.

[0036] Optionally, the correcting the first distance by the level meter to obtain a second distance, and correcting the first height by the level meter to obtain a second height includes: The first distance is corrected by the following formula: ; Among them, D z represents the second distance, D represents the first distance, Indicates the inclination angle between the level and the horizontal plane. The first height is corrected by the following formula: ; Among them, H z represents the second height, H represents the first height, Indicates the inclination angle of the level instrument to the horizontal plane.

[0037] The first distance and the first height are mainly corrected by the built-in level gauge of the excavator. The inclination angle between the level gauge and the horizontal plane is , the azimuth of the level is The first distance and the first height can be corrected by the above two formulas.

[0038] S130, obtaining cab positioning information through the global navigation satellite system device, obtaining azimuth information through the electronic compass, and obtaining the coordinates of the working point of the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance; The precise latitude and longitude coordinates and elevation information of the cab (i.e. the location of the excavator) can be obtained through the Global Navigation Satellite System (GNSS) device. An electronic compass is a device that can measure and display the current direction (relative to the earth's magnetic field). Through the electronic compass, the current azimuth of the excavator, that is, the angle of the excavator relative to the true north direction, can be obtained. A reference coordinate system needs to be determined so that the cab positioning information, azimuth information, and the position information of the end of the arm can be unified into this coordinate system. Using the cab positioning information and azimuth information, the direction of the excavator on the horizontal plane can be calculated. Combined with the second distance, the position of the end of the arm on the horizontal plane can be determined. After determining the position of the end of the arm on the horizontal plane, its elevation needs to be calculated. This can be achieved by adding the second height to the elevation of the cab (or making other appropriate adjustments). Combining the position of the end of the arm on the horizontal plane and the elevation information, the coordinates of the working point of the end of the arm can be obtained. This coordinate is usually expressed in the form of longitude, latitude, and elevation.

[0039] Optionally, obtaining the coordinates of the working point of the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance includes: The coordinates of the working point are determined by the following formula: ; Among them, (X0, Y0, Z0) represents the cab positioning information, Indicates azimuth information, D z Indicates the second distance, H z Indicates the second height.

[0040] S140. When the working state of the bucket changes, record the operation information, the operation information including the coordinates of the operation point, the time point and the latest working state, and detect whether the excavator state is normal based on the operation information.

[0041] The working state of the bucket is mainly divided into no-load state and loaded state. Figure 5 is a schematic diagram of an excavator bucket in an empty state according to an embodiment of the present application, Figure 6 Schematic diagram of the excavator bucket loading state disclosed in the embodiment of the present application. Figure 5 As shown, the no-load state means that there is no material or soil in the bucket. Figure 6As shown in the figure, the loading state means that the bucket has been loaded with materials or soil. Through sensors or other monitoring equipment, the weight, posture and other parameters of the bucket can be monitored in real time to determine whether the bucket is currently in an empty state or a loaded state. When the bucket changes from an empty state to a loaded state, it means that the excavator has completed the material loading at a certain excavation point, and the coordinates of the operation point can be determined as a loading point. On the contrary, when the bucket changes from a loaded state to an empty state, it means that the excavator has completed the material unloading, and the coordinates of the operation point can be determined as an unloading point. When the working state of the bucket changes, the system will record a series of operation information, including the coordinates of the operation point, the time point and the latest working state. The time point is the specific moment when the bucket state changes. High-precision time synchronization equipment is usually used to ensure the accuracy of the time. The latest working state refers to whether the bucket is currently in a loaded state or an empty state. The recorded operation information is not only used to record the working process of the bucket, but also to detect whether the state of the excavator is normal. For example, by analyzing the frequency, time interval and distribution of the operation points of the bucket loading and unloading, it can be determined whether the excavator is in an efficient working state, and whether there are problems such as abnormal shutdown or excessive wear. If the excavator is found to be in an abnormal state, the system can issue an alarm in time to remind the operator or maintenance personnel to check and repair. The recorded operation information will be transmitted back to the cloud platform in real time through the 4G IoT signal for analysis. The cloud platform can process and analyze a large amount of data and extract useful information to evaluate the performance and working status of the excavator. Through data analysis, the cloud platform can also provide optimization suggestions to help improve the working efficiency of the excavator and reduce the failure rate.

[0042] This embodiment also discloses a monitoring system for the working posture of an excavator. Figure 7 Schematic diagram of a module of a monitoring system for an excavator operation posture disclosed in an embodiment of the present application, such as Figure 7 As shown, the system includes a video acquisition module 701, a first calculation module 702, a second calculation module 703 and a state monitoring module 704, wherein: The video acquisition module 701 is configured to acquire the video of the bucket, the forearm and the excavation surface through a camera, and process the video using a rotating target detection model to obtain the four coordinates of the forearm, the rotation angle and the state of the bucket; A first calculation module 702 is configured to obtain a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle, calculate a first height of the end of the forearm according to the width and vertical coordinates of the end of the forearm in the image frame of the video, combined with the height of the camera from the ground and the first distance, correct the first distance by a level meter to obtain a second distance, and correct the first height by the level meter to obtain a second height; The second calculation module 703 is configured to obtain cab positioning information through a global navigation satellite system device, obtain azimuth information through an electronic compass, and obtain the coordinates of the operating point of the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance; The state monitoring module 704 is configured to record operation information when the working state of the bucket changes, the operation information including the coordinates of the operation point, the time point and the latest working state, and detect whether the excavator state is normal according to the operation information.

[0043] Optionally, the first calculation module 702 is configured to: Determine the right edge of the forearm, and determine the positional relationship between the right edge and the vertical line; When the right edge is located in the clockwise direction of the vertical line, a first distance between the end of the forearm and the cab is calculated based on a first preset formula according to the first actual width of the top of the forearm, the rotation angle between the right edge of the forearm and the vertical line, and the coordinates of the two vertices of the top of the forearm in the rotation frame; When the right edge is located in the counterclockwise direction of the vertical line, the first distance between the end of the forearm and the cab is calculated based on the second preset formula according to the second actual width of the end of the forearm and the coordinates of the two vertices of the end of the forearm in the rotation frame.

[0044] Optionally, the first preset formula is as follows: ; Among them, a, b and c are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d0 is the first actual width of the top of the arm, L is the length of the arm, is the rotation angle between the right edge of the forearm and the vertical line, D is the first distance between the end of the forearm and the cab, (X a , Ymin1) and (X b , Ymin2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X a , Ymin1) is the coordinate with the smallest ordinate among the two vertex coordinates, The second preset formula is as follows: ; Among them, k and t are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d1 is the second actual width of the end of the arm, (X c , Ymax1) and (X d , Ymax2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X c, Ymax1) is the coordinate with the largest ordinate among the two vertex coordinates.

[0045] Optionally, the first calculation module 702 is configured to: Calculate the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body according to the first distance; The first height of the forearm end is calculated based on a third preset formula according to the first vertical coordinate, the second vertical coordinate of the forearm end in the image frame, the width of the forearm end in the image frame and the second actual width of the forearm end.

[0046] Optionally, the third preset formula is as follows: ; Wherein, H represents the first height of the end of the forearm, h represents the height of the camera from the ground, and y t is the second vertical coordinate of the forearm end in the image frame, y m It represents the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body, w m represents the width of the end of the forearm in the image frame, and d1 represents the second actual width of the end of the forearm.

[0047] Optionally, the first calculation module 702 is further configured to: The first distance is corrected by the following formula: ; Among them, D z represents the second distance, D represents the first distance, Indicates the inclination angle between the level and the horizontal plane. The first height is corrected by the following formula: ; Among them, H z represents the second height, H represents the first height, Indicates the inclination angle of the level instrument to the horizontal plane.

[0048] Optionally, the second calculation module 703 is configured to: The coordinates of the working point are determined by the following formula: ; Among them, (X0, Y0, Z0) represents the cab positioning information, Indicates azimuth information, D z Indicates the second distance, H z Indicates the second height.

[0049] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0050] This embodiment also discloses an electronic device, referring to Figure 8 The electronic device may include: at least one processor 801 , at least one communication bus 802 , a user interface 803 , a network interface 804 , and at least one memory 805 .

[0051] The communication bus 802 is used to realize the connection and communication between these components.

[0052] The user interface 803 may include a display screen (Display) and a camera (Camera). The optional user interface 803 may also include a standard wired interface and a wireless interface.

[0053] The network interface 804 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0054] Among them, the processor 801 may include one or more processing cores. The processor 801 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 805, and calling data stored in the memory 805. Optionally, the processor 801 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 801 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 801, and it can be implemented separately through a chip.

[0055] Among them, the memory 805 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 805 may also be optionally at least one storage device located away from the aforementioned processor 801. As Figure 8 As shown, the memory 805 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a method for monitoring the working posture of an excavator.

[0056] exist Figure 8 In the electronic device shown, the user interface 803 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 801 can be used to call the application program of the monitoring method of the excavator working posture stored in the memory 805. When executed by one or more processors 801, the electronic device executes one or more methods such as those in the above-mentioned embodiments.

[0057] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0058] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0059] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0060] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0062] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 805, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 805 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0063] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for monitoring the working posture of an excavator, characterized in that: Applied to an excavator operation posture monitoring platform, the excavator operation posture monitoring platform includes a camera, a level, a global navigation satellite system device and an electronic compass installed in the cab of the excavator, and the method includes: The camera is used to collect a video of the bucket, the forearm and the excavation surface, and the video is processed using a rotating target detection model to obtain the four coordinates of the forearm, the rotation angle and the state of the bucket; Obtaining a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle, calculating a first height of the end of the forearm according to the width and vertical coordinates of the end of the forearm in the image frame of the video, in combination with the height of the camera from the ground and the first distance, correcting the first distance by the level meter to obtain a second distance, and correcting the first height by the level meter to obtain a second height; The cab positioning information is obtained through the global navigation satellite system device, the azimuth information is obtained through the electronic compass, and the coordinates of the working point of the end of the forearm are obtained according to the cab positioning information, the azimuth information, the second height and the second distance; When the working state of the bucket changes, operation information is recorded, the operation information including the coordinates of the operation point, the time point and the latest working state, and whether the excavator state is normal is detected based on the operation information.

2. The method for monitoring the working posture of an excavator based on rotating target detection according to claim 1 is characterized in that: The obtaining of the first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle comprises: Determine the right edge of the forearm, and determine the positional relationship between the right edge and the vertical line; When the right edge is located in the clockwise direction of the vertical line, a first distance between the end of the forearm and the cab is calculated based on a first preset formula according to the first actual width of the top of the forearm, the rotation angle between the right edge of the forearm and the vertical line, and the coordinates of the two vertices of the top of the forearm in the rotation frame; When the right edge is located in the counterclockwise direction of the vertical line, the first distance between the end of the forearm and the cab is calculated based on the second preset formula according to the second actual width of the end of the forearm and the coordinates of the two vertices of the end of the forearm in the rotation frame.

3. The method for monitoring the working posture of an excavator based on rotating target detection according to claim 2 is characterized in that: The first preset formula is as follows: ； Among them, a, b and c are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d0 is the first actual width of the top of the arm, L is the length of the arm, is the rotation angle between the right edge of the forearm and the vertical line, D is the first distance between the end of the forearm and the cab, (X a , Ymin1) and (X b , Ymin2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X a , Ymin1) is the coordinate with the smallest ordinate among the two vertex coordinates, The second preset formula is as follows: ； Among them, k and t are corresponding calibration coefficients, which are related to the focal length of the image acquisition and the specifications of the excavator arm, and are fixed values ​​for the same model of equipment; d1 is the second actual width of the end of the arm, (X c , Ymax1) and (X d , Ymax2) are the coordinates of the two vertices at the top of the forearm in the rotation frame, where (X c , Ymax1) is the coordinate with the largest ordinate among the two vertex coordinates.

4. The method for monitoring the working posture of an excavator based on rotating target detection according to claim 1, characterized in that: Calculating the first height of the forearm end according to the width and the vertical coordinate of the forearm end in the image frame of the video, combined with the height of the camera from the ground and the first distance, comprises: Calculate the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body according to the first distance; The first height of the forearm end is calculated based on a third preset formula according to the first vertical coordinate, the second vertical coordinate of the forearm end in the image frame, the width of the forearm end in the image frame and the second actual width of the forearm end.

5. The method for monitoring the working posture of an excavator based on rotating target detection according to claim 4 is characterized in that: The third preset formula is as follows: ； Wherein, H represents the first height of the end of the forearm, h represents the height of the camera from the ground, and y t is the second vertical coordinate of the forearm end in the image frame, y m It represents the first vertical coordinate of the end of the forearm when the bucket is placed on the horizontal plane of the vehicle body, w m represents the width of the end of the forearm in the image frame, and d1 represents the second actual width of the end of the forearm.

6. The method for monitoring the working posture of an excavator based on rotating target detection according to claim 1, characterized in that: The correcting the first distance by the level meter to obtain the second distance, and correcting the first height by the level meter to obtain the second height comprises: The first distance is corrected by the following formula: ； Among them, D z represents the second distance, D represents the first distance, Indicates the inclination angle between the level and the horizontal plane. The first height is corrected by the following formula: ； Among them, H z represents the second height, H represents the first height, Indicates the inclination angle of the level instrument to the horizontal plane.

7. The method for monitoring the working posture of an excavator based on rotating target detection according to claim 6, characterized in that: The step of obtaining the coordinates of the working point at the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance comprises: The coordinates of the working point are determined by the following formula: ； Among them, (X0, Y0, Z0) represents the cab positioning information, Indicates azimuth information, D z Indicates the second distance, H z Indicates the second height.

8. A monitoring system for an excavator's operating posture, characterized in that: It includes a video acquisition module, a first calculation module, a second calculation module and a status monitoring module, wherein: A video acquisition module is configured to acquire a video of the bucket, the forearm and the excavation surface through a camera, and process the video using a rotating target detection model to obtain the four coordinates of the forearm, the rotation angle and the state of the bucket; a first calculation module, configured to obtain a first distance between the end of the forearm and the cab according to the four-axis coordinates and the rotation angle, calculate a first height of the end of the forearm according to the width and vertical coordinates of the end of the forearm in the image frame of the video, in combination with the height of the camera from the ground and the first distance, correct the first distance by a level meter to obtain a second distance, and correct the first height by the level meter to obtain a second height; a second calculation module, configured to obtain cab positioning information through a global navigation satellite system device, obtain azimuth information through an electronic compass, and obtain the coordinates of the operating point of the end of the forearm according to the cab positioning information, the azimuth information, the second height and the second distance; The status monitoring module is configured to record operation information when the working state of the bucket changes, the operation information including the coordinates of the operation point, the time point and the latest working state, and detect whether the excavator state is normal based on the operation information.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images