Method for analyzing the behavior of a loader and device therefor

Abstract

The application provides a loader operation behavior analysis method and device, and relates to the technical field of coal. Different angles of a target area are collected in real time based on multiple cameras to obtain multiple candidate video streams, wherein the target area includes at least one loader; target detection is performed on candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the multiple candidate video streams; for any loader, target tracking is performed on target image frames contained in each target video stream corresponding to the loader to obtain a motion trajectory of the loader, and key point recognition is performed on the target image frames to obtain coordinate information of a target part of the loader; and the current operation behavior state of the loader is classified according to the motion trajectory and the coordinate information, so that real-time analysis of the operation behavior of the loader is realized, illegal operation behavior of the loader is supervised, and the loader with illegal operation can be rectified subsequently.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal, in particular to a method and device for analyzing operation behavior of a loader. BACKGROUND

[0002] In the process of coal mining, the loader is an important engineering vehicle of the open-pit coal mine and is widely used for loading and transporting ores, workpieces and the like and for road maintenance. In order to ensure the safe operation of production activities, the operation behavior of such large working vehicles needs to be supervised. In a related scene, according to the safety operation rules of the loader, for safety reasons, the loader should retract the bucket when driving and is strictly prohibited from driving with the bucket raised. However, in the related art, there is little research on the supervision of the operation behavior of the loader driving with the bucket raised in violation of the rules, and there is no mature solution. SUMMARY

[0003] The present application aims to at least partly solve one of the technical problems in the related art.

[0004] To this end, one object of the present application is to provide a method for analyzing operation behavior of a loader, comprising: collecting, based on a plurality of cameras, a plurality of candidate video streams corresponding to different angles of a target area, wherein the target area includes at least one loader; performing target detection on candidate image frames included in each candidate video stream to determine at least one target video stream corresponding to each loader from the plurality of candidate video streams; for any loader, performing target tracking on target image frames included in each target video stream corresponding to the loader to obtain a motion trajectory of the loader, and performing key point recognition on the target image frames to obtain coordinate information of a target part of the loader; and classifying a current operation behavior state of the loader according to the motion trajectory and the coordinate information, the classification including a non-moving state category, a moving and bucket-lowered state category, and a moving and bucket-raised state category.

[0005] A second object of the present application is to provide a device for analyzing operation behavior of a loader.

[0006] A third object of the present application is to provide an electronic device.

[0007] A fourth object of the present application is to provide a non-transitory computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above object, the first aspect of the present application proposes a loader operation behavior analysis method, comprising: collecting multiple candidate video streams corresponding to different angles of a target area in real time based on multiple cameras, wherein the target area includes at least one loader; performing target detection on candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the multiple candidate video streams; for any loader, performing target tracking on target image frames contained in each target video stream corresponding to the loader, obtaining a motion trajectory of the loader, and performing key point identification on the target image frames to obtain coordinate information of a target part of the loader; and classifying a current operation behavior state of the loader according to the motion trajectory and the coordinate information, wherein the classification includes a non-moving state category, a moving and bucket lowering state category, and a moving and bucket lifting state category.

[0010] According to one embodiment of the present application, the loader operation behavior analysis method further comprises: in response to the loader being in the moving and bucket lifting state category, issuing an alarm instruction to the loader.

[0011] According to one embodiment of the present application, for any loader, the target tracking on the target image frames contained in each target video stream corresponding to the loader to obtain the motion trajectory of the loader comprises: for any loader, obtaining multiple target image frames contained in each target video stream corresponding to the loader; placing the target image frames in a to-be-processed image frame queue in chronological order as to-be-processed target image frames in the to-be-processed image frame queue; and performing target tracking processing on the to-be-processed target image frames based on a target tracking algorithm to obtain the motion trajectory of the loader.

[0012] According to one embodiment of the present application, the loader operation behavior analysis method further comprises: in response to the number of to-be-processed target image frames in the to-be-processed image frame queue reaching a full capacity of the to-be-processed image frame queue, determining a to-be-deleted image frame to be deleted in the to-be-processed image frame queue; deleting the to-be-deleted image frame from the to-be-processed image frame queue, and placing the target image frames in the to-be-processed image frame queue in chronological order to update the to-be-processed image frame queue.

[0013] According to one embodiment of the present application, target detection is performed on candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the plurality of candidate video streams, including: performing target detection on candidate image frames contained in each candidate video stream to determine a target detection rectangular frame corresponding to each loader; obtaining an aspect ratio of each target detection rectangular frame, and determining a number percentage of pixel points of each target detection rectangular frame relative to a total number of pixel points of the entire candidate image frame; for any loader, a candidate video stream in which a target detection rectangular frame with an aspect ratio greater than an aspect ratio threshold and a number percentage greater than a number percentage threshold is located is taken as a target video stream corresponding to the loader.

[0014] According to one embodiment of the present application, the current working behavior state of the loader is classified according to the motion trajectory and the coordinate information, including: analyzing the motion trajectory to determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determining that the loader is in a non-moving state category; in response to the loader being in a moving state, determining whether the loader is in a bucket lifting state according to the coordinate information; in response to the loader being in a bucket lifting state, determining that the loader is in a moving and bucket lifting state category; in response to the loader being in a bucket lowering state, determining that the loader is in a moving and bucket lowering state category.

[0015] According to one embodiment of the present application, the motion trajectory is analyzed to determine whether the loader is in a moving state, including: for any target video stream of the loader, obtaining a first horizontal coordinate value of a trajectory point corresponding to the motion trajectory of the loader in the i-th target image frame corresponding to the current time of the target video stream; obtaining a second horizontal coordinate value of a trajectory point corresponding to the motion trajectory of the loader in the i-m-th target image frame of the target video stream; obtaining a difference value of the first horizontal coordinate value and the second horizontal coordinate value; in response to the difference value being greater than a preset threshold, determining that the loader is in a moving state.

[0016] According to one embodiment of the present application, whether the loader is in a bucket lifting state is determined according to the coordinate information, including: obtaining a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to the loader bucket and a second direction vector corresponding to the loader body; inputting the direction vector into a trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM, wherein the current bucket state category includes a bucket lifting state and a bucket retracting state.

[0017] According to one embodiment of the present application, the analysis method of the working behavior of the loader further includes: updating the target video stream corresponding to the loader in real time.

[0018] To achieve the above object, an embodiment of the second aspect of the present application provides an analysis device for a loader operation behavior, comprising: a collection module configured to collect, based on a plurality of cameras, a plurality of candidate video streams corresponding to different angles of a target area in real time, wherein the target area comprises at least one loader; a detection module configured to perform target detection on candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the plurality of candidate video streams; a tracking module configured to, for any loader, perform target tracking on target image frames contained in each target video stream corresponding to the loader, obtain a motion trajectory of the loader, and perform key point identification on the target image frames to obtain coordinate information of a target part of the loader; and a classification module configured to classify a current operation behavior state of the loader according to the motion trajectory and the coordinate information, wherein the classification comprises a non-moving state category, a moving and bucket lowering state category, and a moving and bucket lifting state category.

[0019] According to an embodiment of the present application, the analysis device for the loader operation behavior further comprises an alarm module configured to, in response to the loader being in the moving and bucket lifting state category, issue an alarm instruction to the loader.

[0020] According to an embodiment of the present application, the tracking module is further configured to, for any loader, obtain a plurality of target image frames contained in each target video stream corresponding to the loader; place the target image frames in a to-be-processed image frame queue in chronological order as to-be-processed target image frames in the to-be-processed image frame queue; and perform target tracking processing on the to-be-processed target image frames based on a target tracking algorithm to obtain the motion trajectory of the loader.

[0021] According to an embodiment of the present application, the tracking module is further configured to, in response to a quantity of to-be-processed target image frames in the to-be-processed image frame queue reaching a full capacity of the to-be-processed image frame queue, determine a to-be-deleted image frame to be deleted in the to-be-processed image frame queue; delete the to-be-deleted image frame from the to-be-processed image frame queue, and place the target image frames in the to-be-processed image frame queue in chronological order to update the to-be-processed image frame queue.

[0022] According to an embodiment of the present application, the detection module is further configured to perform target detection on candidate image frames contained in each candidate video stream to determine a target detection rectangular frame corresponding to each loader; obtain an aspect ratio of each target detection rectangular frame, and determine a number percentage of a quantity of pixel points of each target detection rectangular frame in a total quantity of pixel points of an entire candidate image frame; and for any loader, take a candidate video stream in which a target detection rectangular frame with an aspect ratio greater than an aspect ratio threshold value and a number percentage greater than a number percentage threshold value is located as a target video stream corresponding to the loader.

[0023] According to one embodiment of the present application, the classification module is further configured to: analyze the motion trajectory of the loader, determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determine that the loader is in a non-moving state category; in response to the loader being in a moving state, determine whether the loader is in a bucket lifting state according to the coordinate information; in response to the loader being in the bucket lifting state, determine that the loader is in a moving and bucket lifting state category; and in response to the loader being in a bucket lowering state, determine that the loader is in a moving and bucket lowering state category.

[0024] According to one embodiment of the present application, the classification module is further configured to: for any target video stream of the loader, obtain a first horizontal coordinate value of a track point corresponding to a motion trajectory of the loader in an i-th target image frame corresponding to a current time of the target video stream; obtain a second horizontal coordinate value of the track point corresponding to the motion trajectory of the loader in an i-m-th target image frame of the target video stream; obtain a difference value between the first horizontal coordinate value and the second horizontal coordinate value; and in response to the difference value being greater than a preset threshold, determine that the loader is in a moving state.

[0025] According to one embodiment of the present application, the classification module is further configured to: obtain a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to a bucket of the loader and a second direction vector corresponding to a body of the loader; and input the direction vector into a trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM, wherein the current bucket state category includes a bucket lifting state and a bucket retracting state.

[0026] According to one embodiment of the present application, the detection module is further configured to: update the target video stream corresponding to the loader in real time.

[0027] To achieve the above object, a third aspect of the present application provides an electronic device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to implement the analysis method of the working behavior of the loader as described in the first aspect of the present application.

[0028] To achieve the above object, a fourth aspect of the present application provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to implement the analysis method of the working behavior of the loader as described in the first aspect of the present application.

[0029] To achieve the above object, a fifth aspect of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the analysis method of the working behavior of the loader as described in the first aspect of the present application.

[0030] The present application at least realizes the following beneficial effects: the present application realizes real-time analysis of the working behavior of the loader, supervises illegal working behavior of the loader, facilitates subsequent rectification of the illegal working loader, and improves the safety of the workers during work. BRIEF DESCRIPTION OF DRAWINGS

[0031] The above-mentioned and / or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 is a schematic diagram of an exemplary implementation of a loader working behavior analysis method according to an embodiment of the present application.

[0033] Figure 2 is a schematic diagram of a key point of a target site of a loader according to an embodiment of the present application.

[0034] Figure 3 is a schematic diagram of an exemplary implementation of a loader working behavior analysis method according to an embodiment of the present application.

[0035] Figure 4 is a schematic diagram of an overall flow of a loader working behavior analysis method according to an embodiment of the present application.

[0036] Figure 5 is a schematic diagram of a loader working behavior analysis device according to an embodiment of the present application.

[0037] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0039] Figure 1 is a schematic diagram of an exemplary implementation of a loader working behavior analysis method according to an embodiment of the present application, as Figure 1 shown, the loader working behavior analysis method comprises the following steps:

[0040] S101, based on a plurality of cameras, real-time collection of a plurality of candidate video streams corresponding to different angles of a target area, wherein the target area includes at least one loader.

[0041] The open-pit coal mine loader operation area is taken as a target area, a plurality of movable cameras are arranged at multiple angles around the target area, a plurality of video streams corresponding to different angles of the target area are collected in real time based on the plurality of cameras, and the video stream collected by each camera in real time is taken as a candidate video stream, wherein the target area includes at least one loader.

[0042] Optionally, the camera can be a wireless camera, and the wireless camera transmits the collected video stream data to the server in a wireless transmission mode such as 4G / 5G / Wi-Fi, to analyze the operation behavior of the loader.

[0043] Since the coal mining working face is constantly moving forward with the progress of the mining work, the target area is also constantly updated. In this application, a tracked chassis can be provided for the camera to facilitate the movement of the camera in the direction of mining movement, so that the target area can be covered by the camera at all times, and the image of the loader target collected in the target area occupies a sufficient number of pixels.

[0044] S102, target detection is performed on the candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the plurality of candidate video streams.

[0045] Since the plurality of angle cameras simultaneously collect the video stream of the loader in the target area, each loader can be oriented towards the camera at multiple angles such as the side, front, and rear. In addition, since the target detection rectangular frame corresponding to each loader includes the shovel part, when the shovel is in different lifting states, it will also cause the aspect ratio of the target detection rectangular frame to change. In order to accurately detect the movement and posture of the loader, in this application, in order to select the preferred video stream suitable for analysis corresponding to each loader from the plurality of candidate video streams, the aspect ratio of each target detection rectangular frame in the candidate image frames contained in each candidate video stream is obtained, and the number percentage of the pixel points of each target detection rectangular frame in the total number of pixel points of the entire candidate image frame is determined. For any loader, the candidate video stream in which the target detection rectangular frame with an aspect ratio greater than an aspect ratio threshold and a number percentage greater than a number percentage threshold is taken as the target video stream corresponding to the loader.

[0046] For example, if there are 3 loaders in the target area, each loader is numbered, and each loader corresponds to a number, numbered as loader 1, loader 2 and loader 3. If there are 8 cameras, numbered as camera 1, camera 2, …, camera 8, the video data collected by the cameras correspond to candidate video stream 1, candidate video stream 2, …, candidate video stream 8. After comparing the length-width ratio and the number percentage of the target detection rectangular frame corresponding to each loader, if only the length-width ratio of loader 1 in candidate video stream 1 and candidate video stream 3 is greater than the length-width ratio threshold and the number percentage is greater than the number percentage threshold, then candidate video stream 1 and candidate video stream 3 are taken as the target video stream corresponding to loader 1. The determination method of the target video stream of other loaders is similar to the above, which will not be described here.

[0047] Wherein, as the loader moves continuously, the target video stream corresponding to the loader needs to be updated in real time.

[0048] S103, for any loader, target tracking is performed on the target image frames contained in each target video stream corresponding to the loader, the motion trajectory of the loader is obtained, and key point recognition is performed on the target image frames to obtain the coordinate information of the target part of the loader.

[0049] The image frames contained in the target video stream are taken as target image frames, and target tracking is performed on the target image frames contained in each target video stream corresponding to any loader, the motion trajectory of the loader is obtained, and key point recognition is performed on the target image frames to obtain the coordinate information of the target part of the loader.

[0050] In this application, for any loader, target tracking can be performed on one of the target video streams corresponding to the loader to obtain a motion trajectory corresponding to the loader, or target tracking can be performed on multiple target video streams corresponding to the loader to obtain multiple motion trajectories of the loader.

[0051] In this application, in addition to obtaining the motion trajectory of the loader, the coordinate information of the target part of the loader is also needed. Wherein, the target part needs to be selected as a position on the loader that can represent the posture of the bucket, and has symmetry and universality to be suitable for identification of different sides of various models of loaders. The target part can be set according to actual conditions. In this application, the target part is set as the front end bottom of the bucket, the front end top of the bucket, the front wheel shaft center of the loader, the rear wheel shaft center of the loader and the front end top of the driver's cabin of the loader. Figure 2 is a schematic diagram of a target part key point of a loader shown in this application, as Figure 2As shown, the bottom of the front end of the bucket is taken as the key point 1, the top of the front end of the bucket is taken as the key point 2, the center of the front wheel of the loader is taken as the key point 3, the center of the rear wheel of the loader is taken as the key point 4, and the top of the front end of the cab of the loader is taken as the key point 5.

[0052] In the present application, for any loader, if only one motion trajectory corresponding to the loader is obtained, only the coordinate information of the target part of the loader in the target image frame contained in the target video stream corresponding to the motion trajectory is obtained; if multiple motion trajectories of the loader are obtained, multiple sets of coordinate information of the target part of the loader in the target image frames contained in multiple target video streams corresponding to the multiple motion trajectories are obtained.

[0053] In S104, the current working behavior state of the loader is classified according to the motion trajectory and the coordinate information. The classification includes a non-moving state category, a moving and bucket lowering state category, and a moving and bucket lifting state category.

[0054] According to the motion trajectory of the loader, it is determined whether the loader moves. For example, if only one motion trajectory corresponding to the loader is obtained, according to the motion trajectory of the loader, if it is found that the trajectory point of the current time of the loader changes from the trajectory point corresponding to one minute ago, it is determined that the loader moves. If multiple motion trajectories of the loader are obtained, the moving judgment results of the loader corresponding to the multiple motion trajectories are obtained, and the moving judgment results of the multiple motion trajectories are voted, and the moving judgment result with a larger quantity is taken as the final result.

[0055] If it is finally determined that the loader does not move, it is determined that the loader is in the non-moving state category.

[0056] If it is finally determined that the loader moves, the posture of the loader is identified according to the coordinate information of the target part. If only one motion trajectory corresponding to the loader is obtained, only the coordinate information of the target part of the loader in the target image frame contained in the target video stream corresponding to the motion trajectory is obtained. When the included angle between the straight line where the key point 1 and the key point 2 are connected and the straight line where the key point 3 and the key point 4 are connected is less than γ, it is considered that the bucket of the loader is in the lifting state, and it is determined that the loader is in the moving and bucket lifting state category. When the included angle between the straight line where the key point 1 and the key point 2 are connected and the straight line where the key point 3 and the key point 4 are connected is greater than or equal to γ, it is considered that the bucket of the loader is in the lowering state, and it is determined that the loader is in the moving and bucket lowering state category. If multiple motion trajectories of the loader are obtained, multiple sets of coordinate information of the target part of the loader in the target image frames contained in multiple target video streams corresponding to the multiple motion trajectories are obtained, and multiple bucket lifting and lowering judgment results corresponding to the multiple sets of coordinate information are obtained. The multiple bucket lifting and lowering judgment results are voted, and the bucket lifting and lowering judgment result with a larger quantity is taken as the final result.

[0057] The embodiment of the present application provides a loader operation behavior analysis method, which comprises the following steps: collecting multiple candidate video streams corresponding to different angles of a target area in real time based on multiple cameras, wherein the target area comprises at least one loader; performing target detection on candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the multiple candidate video streams; for any loader, performing target tracking on target image frames contained in each target video stream corresponding to the loader to obtain a motion trajectory of the loader, and performing key point identification on the target image frames to obtain coordinate information of a target part of the loader; and classifying a current operation behavior state of the loader according to the motion trajectory and the coordinate information, wherein the classification comprises a non-moving state category, a moving and bucket lowering state category and a moving and bucket lifting state category. The loader operation behavior analysis method can realize real-time analysis of the operation behavior of the loader, monitor illegal operation behavior of the loader, facilitate subsequent rectification of the loader with illegal operation behavior, and improve the safety of workers during operation.

[0058] Further, if the loader is in the moving and bucket lifting state category, current illegal behavior video data is saved, and an alarm instruction is sent to the loader.

[0059] Figure 3 is a schematic diagram of an exemplary embodiment of a loader operation behavior analysis method shown in the present application, as shown in the figure, the loader operation behavior analysis method comprises the following steps: Figure 3

[0060] S301, collecting multiple candidate video streams corresponding to different angles of a target area in real time based on multiple cameras, wherein the target area comprises at least one loader.

[0061] For the specific implementation of step S301, refer to the specific description of the related part in the above embodiment, which will not be repeated here.

[0062] S302, performing target detection on candidate image frames contained in each candidate video stream to determine a target detection rectangular frame corresponding to each loader.

[0063] The target detection algorithm is used to perform target detection on the candidate image frames contained in each candidate video stream to determine a target detection rectangular frame corresponding to each loader.

[0064] S303, obtaining an aspect ratio of each target detection rectangular frame, and determining a number percentage of pixel points of each target detection rectangular frame in total pixel points of the entire candidate image frame.

[0065] ​Obtaining the aspect ratio of each target detection rectangular frame, and determining the number percentage of the pixel points of each target detection rectangular frame in the total number of pixel points of the entire candidate image frame.

[0066] S304, for any loader, the candidate video stream in which the target detection rectangular frame with the aspect ratio greater than the aspect ratio threshold and the number percentage greater than the number percentage threshold is taken as the target video stream corresponding to the loader.

[0067] For any loader, the candidate video stream in which the target detection rectangular frame with the aspect ratio greater than the aspect ratio threshold and the number percentage greater than the number percentage threshold is taken as the target video stream corresponding to the loader.

[0068] For example, if there are 3 loaders in the target area, they are numbered respectively, each loader corresponds to a number, and are numbered as loader 1, loader 2 and loader 3 respectively. If there are 8 cameras in total, they are camera 1, camera 2, …, camera 8 respectively, and the video data collected by them correspond to candidate video stream 1, candidate video stream 2, …, candidate video stream 8 respectively. After comparing the aspect ratio and number percentage of the target detection rectangular frame corresponding to each loader, if only the aspect ratio of loader 1 in candidate video stream 1 and candidate video stream 3 is greater than the aspect ratio threshold and the number percentage is greater than the number percentage threshold, then candidate video stream 1 and candidate video stream 3 are taken as the target video stream corresponding to loader 1. The determination method of the target video stream of other loaders is similar to the above, which will not be described here.

[0069] S305, for any loader, obtaining the multiple target image frames contained in each target video stream corresponding to the loader.

[0070] For any loader, the image frames contained in each target video stream corresponding to the loader are taken as target image frames, and multiple target image frames contained in each target video stream corresponding to the loader are obtained.

[0071] S306, the target image frames are put into the to-be-processed image frame queue in time sequence as to-be-processed target image frames in the to-be-processed image frame queue.

[0072] For any target video stream, the multiple target image frames contained in the target video stream are put into the to-be-processed image frame queue in time sequence as to-be-processed target image frames in the to-be-processed image frame queue.

[0073] S307, based on the target tracking algorithm, the to-be-processed target image frames are processed for target tracking to obtain the motion trajectory of the loader.

[0074] The target tracking algorithm is used to track the target in the target image frame to be processed, and the movement trajectory of the loader is obtained. The right edge 1 / 2 of the loader target detection box in the target image frame to be processed is taken as the trajectory point of the current image of the loader, and the movement trajectory of the loader is drawn. The movement trajectory of the loader M can be represented by a sequence K M

[0075] K M = {(x1,y1,frame1),(x2,y2,frame2),…(x n ,y n ,frame n )}

[0076] wherein x n and y n are the horizontal and vertical coordinates of the right edge 1 / 2 point of the target detection rectangular box, and frame n represents the image in which the loader target is detected in the nth frame.

[0077] After the current target image frame to be processed in the target image frame queue is processed, a new target image frame to be processed is obtained from the target image frame queue to obtain the movement trajectory of the loader.

[0078] The capacity of the target image frame queue is fixed. If the number of target image frames to be processed in the target image frame queue reaches the full capacity of the target image frame queue, a target image frame to be deleted is determined to be deleted from the target image frame queue, wherein the target image frame to be deleted is an image frame collected earlier in time. The target image frame to be deleted is deleted from the target image frame queue, and the target image frame is placed in the target image frame queue in chronological order to update the target image frame queue, so that the latest image frame can be obtained for target tracking processing at all times, and accumulation of target image frames to be processed is avoided.

[0079] Optionally, the target tracking algorithm can use an improved target detection network model (YOLOv5s) to obtain the position information of the loader in the image frame.

[0080] Optionally, a tracking-by-detection (TBD) method is used to identify the same loader target in the video. For example, a DeepSort algorithm is used to associate the loaders in different frames of images, and the same ID is assigned to the same loader appearing in different frames of images, and then the movement trajectory of the same loader is drawn.

[0081] In addition, in this application, a multi-threading technology can be used, and independent threads are used to read the video stream and obtain real-time images in the video stream, and the latest images are placed in the processing queue. ​

[0082] S308, and the target image frame is subjected to key point recognition to obtain coordinate information of a target part of the loader.

[0083] In the present application, in addition to obtaining the motion trajectory of the loader, coordinate information of a target part of the loader is also needed. The target part needs to be selected as a position on the loader that can represent the posture of the bucket and has symmetry and universality to be suitable for different side recognition of various models of loaders, which can be set according to actual conditions. In the present application, the target part is set as the front end bottom of the bucket, the front end top of the bucket, the front wheel shaft center of the loader, the rear wheel shaft center of the loader and the front end top of the cab of the loader.

[0084] Optionally, a top-down key point detection network can be used to detect the key parts of the loader, such as Mask Region-based Convolutional Neural Networks (Mask R-CNN), High Resolution Net (HRNet), Cascaded Pyramid Network (CPN) and the like.

[0085] S309, according to the motion trajectory and the coordinate information, classifying the current working behavior state of the loader, the classification including a non-moving state category, a moving and bucket lowering state category and a moving and bucket lifting state category.

[0086] The position of the moving loader in the image is changing, and by detecting the position change of the current loader target detection frame in the front and rear images, whether the loader is moving can be recognized.

[0087] For any target video stream of the loader, a first horizontal coordinate value of a trajectory point corresponding to the motion trajectory of the loader in the i-th target image frame corresponding to the current time of the target video stream is obtained, a second horizontal coordinate value of a trajectory point corresponding to the motion trajectory of the loader in the i-m-th target image frame of the target video stream is obtained, a difference value between the first horizontal coordinate value and the second horizontal coordinate value is obtained, and if the difference value is greater than a preset threshold, it is determined that the loader is in a moving state.

[0088] If it is finally determined that the loader does not move, it is determined that the loader is in a non-moving state category.

[0089] If it is finally determined that the loader is in a moving state, a direction vector corresponding to the loader is obtained according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to the bucket of the loader and a second direction vector corresponding to the body of the loader. In the present application, as shown in the following formula, by extracting a first direction vector corresponding to the loader bucket, by extracting a second direction vector corresponding to the loader body, wherein:

[0090]

[0091]

[0092]

[0093] In the above formula, denotes a first direction vector corresponding to the loader bucket, denotes a second direction vector corresponding to the loader body, x n denotes the horizontal coordinate of the key point n, y n denotes the vertical coordinate of the key point n, and Figure 2 Since 5 target points are set, n can take values 1, 2, 3, 4, and 5.

[0094] The first direction vector and the second direction vector are input into a trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM, wherein the current bucket state category includes a bucket lifting state and a bucket retracting state. If the current bucket state of the loader output by the SVM is in the lifting state, it is determined that the loader is in the moving and bucket lifting state category; if the current bucket state of the loader output by the SVM is in the landing state, it is determined that the loader is in the moving and bucket landing state category.

[0095] The training process of the SVM is as follows: a large number of sample loader pictures are pre-collected, key points are labeled, and a sample first direction vector corresponding to the bucket of the sample loader and a sample second direction vector corresponding to the body of the sample loader in each sample loader picture are obtained. Based on the bucket lifting label or the bucket landing label labeled in advance for each sample loader picture, the initial SVM is trained to obtain the trained SVM.

[0096] The embodiment of the present application realizes real-time analysis of the working behavior of the loader, supervises the illegal working behavior of the loader, facilitates subsequent rectification of the loader with illegal working behavior, and improves the safety of the workers during work.

[0097] Figure 4 is a general flowchart of a loader working behavior analysis method shown in the present application, as Figure 4 shown, the loader working behavior analysis method comprises the following steps:

[0098] S401, acquire multiple candidate video streams corresponding to different angles of the target region based on multiple cameras in real time, wherein the target region includes at least one loader.

[0099] For the specific implementation of step S401, refer to the specific description of the related part in the above embodiment, which will not be repeated here.

[0100] S402, target detection is performed on the candidate image frames contained in each candidate video stream to determine the target detection rectangular frame corresponding to each loader.

[0101] S403, the aspect ratio of each target detection rectangular frame is obtained, and the number percentage of the pixel points of each target detection rectangular frame in the total number of pixel points of the entire candidate image frame is determined.

[0102] S404, for any loader, the candidate video stream in which the target detection rectangular frame with an aspect ratio greater than the aspect ratio threshold and a number percentage greater than the number percentage threshold is taken as the target video stream corresponding to the loader.

[0103] For the specific implementation of steps S402-S404, refer to the specific description of the related part in the above embodiment, which will not be repeated here.

[0104] S405, for any loader, multiple target image frames contained in each target video stream corresponding to the loader are obtained.

[0105] S406, the target image frames are put into the to-be-processed image frame queue in chronological order as the to-be-processed target image frames in the to-be-processed image frame queue.

[0106] S407, target tracking processing is performed on the to-be-processed target image frames based on a target tracking algorithm to obtain the motion trajectory of the loader.

[0107] S408, key point recognition is performed on the target image frames to obtain the coordinate information of the target part of the loader.

[0108] For the specific implementation of steps S405-S408, refer to the specific description of the related part in the above embodiment, which will not be repeated here.

[0109] S409, analyze the motion trajectory to determine whether the loader is in a moving state.

[0110] S410, in response to the loader being in a non-moving state, determine that the loader is in a non-moving state category.

[0111] S411, in response to the loader being in a moving state, determine whether the loader is in a bucket lifting state according to the coordinate information.

[0112] S412, in response to the loader being in the bucket lowering state, determining that the loader is in a moving and bucket lowering state category.

[0113] S413, in response to the loader being in the bucket raising state, determining that the loader is in a moving and bucket raising state category.

[0114] S414, in response to the loader being in the moving and bucket raising state category, issuing an alarm indication to the loader.

[0115] For specific implementation of steps S409-S414, refer to the specific description of the related part in the above embodiment, which will not be repeated here.

[0116] The embodiment of the application realizes real-time analysis of the working behavior of the loader, supervises the illegal working behavior of the loader, and alarms the loader that is working illegally, so as to facilitate subsequent rectification of the loader that is working illegally, and improve the safety of the workers when working.

[0117] Figure 5 is a schematic diagram of a loader working behavior analysis device shown in the application, as Figure 5 shown, the loader working behavior analysis device 500 includes a collection module 501, a detection module 502, a tracking module 503, and a classification module 504, wherein:

[0118] The collection module 501 is configured to collect multiple candidate video streams corresponding to different angles of a target area in real time based on multiple cameras, wherein the target area includes at least one loader.

[0119] The detection module 502 is configured to perform target detection on candidate image frames contained in each candidate video stream to determine at least one target video stream corresponding to each loader from the multiple candidate video streams.

[0120] The tracking module 503 is configured to, for any loader, perform target tracking on target image frames contained in each target video stream corresponding to the loader, obtain a motion trajectory of the loader, and perform key point recognition on the target image frames to obtain coordinate information of a target part of the loader.

[0121] The classification module 504 is configured to classify a current working behavior state of the loader according to the motion trajectory and the coordinate information, and the classification includes a non-moving state category, a moving and bucket lowering state category, and a moving and bucket raising state category.

[0122] The device realizes real-time analysis on the working behavior of the loader, supervises illegal working behavior of the loader, and alarms the loader working illegally, so as to facilitate subsequent rectification of the loader working illegally and improve the safety of workers during working.

[0123] According to one embodiment of the present application, the analysis device 500 of the working behavior of the loader further comprises an alarm module 505 configured to issue an alarm instruction to the loader in response to the loader being in the moving and bucket lifting state category.

[0124] According to one embodiment of the present application, the tracking module 503 is further configured to: for any loader, acquire a plurality of target image frames contained in each target video stream corresponding to the loader; put the target image frames into a to-be-processed image frame queue in chronological order as to-be-processed target image frames in the to-be-processed image frame queue; and perform target tracking processing on the to-be-processed target image frames based on a target tracking algorithm to acquire a motion trajectory of the loader.

[0125] According to one embodiment of the present application, the tracking module 503 is further configured to: in response to the number of to-be-processed target image frames in the to-be-processed image frame queue reaching the full capacity of the to-be-processed image frame queue, determine a to-be-deleted image frame to be deleted in the to-be-processed image frame queue; delete the to-be-deleted image frame from the to-be-processed image frame queue, and put the target image frames into the to-be-processed image frame queue in chronological order to update the to-be-processed image frame queue.

[0126] According to one embodiment of the present application, the detection module 502 is further configured to: perform target detection on the candidate image frames contained in each candidate video stream to determine a target detection rectangular frame corresponding to each loader; acquire an aspect ratio of each target detection rectangular frame, and determine a number percentage of the number of pixel points of each target detection rectangular frame in the total number of pixel points of the entire candidate image frame; and for any loader, take the candidate video stream in which the target detection rectangular frame with an aspect ratio greater than an aspect ratio threshold and a number percentage greater than a number percentage threshold as the target video stream corresponding to the loader.

[0127] According to one embodiment of the present application, the classification module 504 is further configured to: analyze the motion trajectory to determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determine that the loader is in a non-moving state category; in response to the loader being in a moving state, determine whether the loader is in a bucket lifting state according to the coordinate information; in response to the loader being in a bucket lifting state, determine that the loader is in a moving and bucket lifting state category; and in response to the loader being in a bucket lowering state, determine that the loader is in a moving and bucket lowering state category.

[0128] According to an embodiment of the present application, the classification module 504 is further configured to: obtain a first horizontal coordinate value of a track point corresponding to a movement track of the loader in an i-th target image frame corresponding to a current time of the target video stream of the loader; obtain a second horizontal coordinate value of the track point corresponding to the movement track of the loader in an i-m-th target image frame of the target video stream; obtain a difference value between the first horizontal coordinate value and the second horizontal coordinate value; and determine that the loader is in a moving state in response to the difference value being greater than a preset threshold.

[0129] According to an embodiment of the present application, the classification module 504 is further configured to: obtain a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to a bucket of the loader and a second direction vector corresponding to a body of the loader; and input the direction vector into the trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM, wherein the current bucket state includes a bucket lifting state and a bucket retracting state.

[0130] According to an embodiment of the present application, the detection module 502 is further configured to: update the target video stream corresponding to the loader in real time.

[0131] To achieve the above-mentioned embodiments, the embodiments of the present application further propose an electronic device 600, as shown in the accompanying drawings, which comprises a processor 601 and a memory 602 in communication connection with the processor 601, and the memory 602 stores instructions executable by the at least one processor 601, and the instructions are executed by the at least one processor 601 to implement the analysis method of the working behavior of the loader as shown in the above-mentioned embodiments. Figure 6

[0132] To achieve the above-mentioned embodiments, the embodiments of the present application further propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to implement the analysis method of the working behavior of the loader as shown in the above-mentioned embodiments.

[0133] To achieve the above-mentioned embodiments, the embodiments of the present application further propose a computer program product comprising a computer program, and the computer program is executed by a processor to implement the analysis method of the working behavior of the loader as shown in the above-mentioned embodiments.

[0134] ​In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0135] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0136] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0137] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An analysis method of a loader work behavior, characterized by, The method comprises: acquiring a plurality of candidate video streams corresponding to different angles of a target region in real time based on a plurality of cameras, wherein the target region comprises at least one loader; performing target detection on candidate image frames contained in each of the candidate video streams to determine at least one target video stream corresponding to each of the loaders from the plurality of candidate video streams; for any of the loaders, performing target tracking on target image frames contained in each of the target video streams corresponding to the loader, acquiring a motion trajectory of the loader, and performing key point identification on the target image frames to acquire coordinate information of a target part of the loader; analyzing the motion trajectory to determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determining that the loader is in a non-moving state category; in response to the loader being in a moving state, acquiring a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector comprises a first direction vector corresponding to a bucket of the loader and a second direction vector corresponding to a body of the loader; inputting the direction vector into a trained support vector machine (SVM) to acquire a current bucket state of the loader output by the SVM; in response to the loader being in a bucket lifting state, determining that the loader is in a moving and bucket lifting state category; in response to the loader being in a bucket lowering state, determining that the loader is in a moving and bucket lowering state category; wherein the analyzing the motion trajectory to determine whether the loader is in a moving state comprises: For any one target video stream of the loader, a first horizontal coordinate value of a track point corresponding to a motion track of the loader in a target image frame of a current time point of the target video stream is acquired i the target image frame acquiring a first target video stream of the target video stream i a second horizontal coordinate value of the trajectory point corresponding to the motion trajectory of the loader in the target image frame of the m frame acquiring a difference between the first horizontal coordinate value and the second horizontal coordinate value; in response to the difference being greater than a preset threshold, determining that the loader is in a moving state.

2. The method of analyzing loader work behavior according to claim 1, wherein, The method further comprises: in response to the loader being in a moving and bucket lifting state category, issuing an alarm instruction to the loader.

3. The method of analyzing loader work behavior according to claim 1 or 2, characterized in that, The method further comprises: for any of the loaders, acquiring a plurality of target image frames contained in each of the target video streams corresponding to the loader; placing the target image frames in a time sequence into a to-be-processed image frame queue as to-be-processed target image frames in the to-be-processed image frame queue; performing target tracking processing on the to-be-processed target image frames based on a target tracking algorithm to acquire the motion trajectory of the loader.

4. The method of analyzing loader work behavior according to claim 3, wherein, The method further comprises: in response to a number of to-be-processed target image frames in the to-be-processed image frame queue reaching a full capacity of the to-be-processed image frame queue, determining a to-be-deleted image frame to be deleted in the to-be-processed image frame queue; deleting the to-be-deleted image frame from the to-be-processed image frame queue, and placing the target image frames in a time sequence into the to-be-processed image frame queue to update the to-be-processed image frame queue.

5. The method of analyzing loader work behavior according to claim 1, wherein, The method further comprises: performing target detection on candidate image frames contained in each of the candidate video streams to determine at least one target video stream corresponding to each of the loaders from the plurality of candidate video streams comprises: target detection rectangular frame corresponding to each of the loaders is determined; a length-width ratio of each of the target detection rectangular frames is obtained, and a number percentage of pixel points of each of the target detection rectangular frames in total pixel points of a candidate image frame is determined; for any of the loaders, a candidate video stream in which a target detection rectangular frame with a length-width ratio greater than a length-width ratio threshold and a number percentage greater than a number percentage threshold is located is determined as the target video stream corresponding to the loader.

6. The method of analyzing loader work behavior according to claim 1, wherein, The method further comprises: real-time updating of the target video stream corresponding to the loader.

7. An apparatus for analyzing a loading machine work behavior, characterized by, It comprises: a collection module configured to collect multiple candidate video streams corresponding to different angles of a target region in real time based on multiple cameras, wherein the target region includes at least one loader; a detection module configured to perform target detection on candidate image frames included in each of the candidate video streams to determine at least one target video stream corresponding to each of the loaders from the multiple candidate video streams; a tracking module configured to, for any of the loaders, perform target tracking on target image frames included in each of the target video streams corresponding to the loader, obtain a motion trajectory of the loader, and perform key point identification on the target image frames to obtain coordinate information of a target part of the loader; The classification module is configured to analyze the motion trajectory to determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determine that the loader is in a non-moving state category; in response to the loader being in a moving state, obtain a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to a loader bucket and a second direction vector corresponding to a loader body; input the direction vector into a trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM; in response to the loader being in a bucket lifting state, determine that the loader is in a moving and bucket lifting state category; in response to the loader being in a bucket lowering state, determine that the loader is in a moving and bucket lowering state category; wherein the analysis of the motion trajectory to determine whether the loader is in a moving state includes: for any target video stream of the loader, obtaining a first horizontal coordinate value of a trajectory point corresponding to the motion trajectory of the loader in a target image frame of the target video stream at a current time; obtaining a second horizontal coordinate value of the trajectory point corresponding to the motion trajectory of the loader in an mth target image frame of the target video stream; obtaining a difference value of the first horizontal coordinate value and the second horizontal coordinate value; and in response to the difference value being greater than a preset threshold, determining that the loader is in a moving state. i The classification module is configured to analyze the motion trajectory to determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determine that the loader is in a non-moving state category; in response to the loader being in a moving state, obtain a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to a loader bucket and a second direction vector corresponding to a loader body; input the direction vector into a trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM; in response to the loader being in a bucket lifting state, determine that the loader is in a moving and bucket lifting state category; in response to the loader being in a bucket lowering state, determine that the loader is in a moving and bucket lowering state category; wherein the analysis of the motion trajectory to determine whether the loader is in a moving state includes: for any target video stream of the loader, obtaining a first horizontal coordinate value of a trajectory point corresponding to the motion trajectory of the loader in a target image frame of the target video stream at a current time; obtaining a second horizontal coordinate value of the trajectory point corresponding to the motion trajectory of the loader in an mth target image frame of the target video stream; obtaining a difference value of the first horizontal coordinate value and the second horizontal coordinate value; and in response to the difference value being greater than a preset threshold, determining that the loader is in a moving state. i The classification module is configured to analyze the motion trajectory to determine whether the loader is in a moving state; in response to the loader being in a non-moving state, determine that the loader is in a non-moving state category; in response to the loader being in a moving state, obtain a direction vector corresponding to the loader according to the coordinate information, wherein the direction vector includes a first direction vector corresponding to a loader bucket and a second direction vector corresponding to a loader body; input the direction vector into a trained support vector machine (SVM) to obtain a current bucket state of the loader output by the SVM; in response to the loader being in a bucket lifting state, determine that the loader is in a moving and bucket lifting state category; in response to the loader being in a bucket lowering state, determine that the loader is in a moving and bucket lowering state category; wherein the analysis of the motion trajectory to determine whether the loader is in a moving state includes: for any target video stream of the loader, obtaining a first horizontal coordinate value of

Images