Methods, devices, and products for monitoring dump truck operations based on multi-label action classification

By using a multi-label action classification method, action category sequences are generated from image data of dump truck operations, solving the problem of low accuracy in dump truck operation analysis and achieving more efficient operation status recognition and analysis.

CN119360259BActive Publication Date: 2026-11-10CHINA THREE GORGES CORPORATION
2 Cites 0 Cited by

Patent Information

Application Number
CN202411010001.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2026-11-10
Estimated Expiration
2044-07-25

Smart Images

  • Figure CN119360259B_ABST
    Figure CN119360259B_ABST
Patent Text Reader

Abstract

This application discloses a method, device, and product for monitoring dump truck operations based on multi-label action classification, comprising: inputting continuous video clips containing a target dump truck into a first action classification model to obtain an action category sequence output by the first action classification model; wherein, the first action classification model is trained based on sample images containing dump truck operation actions and multiple action category labels corresponding to the sample images; and determining the operation status of the target dump truck based on the action category sequence. This method fully utilizes the various operation posture data contained in the image data of the dump truck during operation, thereby improving the accuracy of dump truck operation analysis to a certain extent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of deep learning technology, specifically relating to a method, device, and product for monitoring dump truck operations based on multi-label action classification. Background Technology

[0002] Dump trucks are the most common and frequently used construction machinery in hydropower and civil engineering projects. Monitoring and analyzing dump truck operations is crucial for optimizing construction progress, reducing the environmental impact of construction, improving construction quality, and ensuring construction safety. For example, analyzing dump truck materials and unloading times allows for optimization of dump truck scheduling, improving the effectiveness of construction progress control; real-time monitoring of dump trucks transporting materials without proper covering ensures compliance with environmental protection management requirements and reduces the environmental impact of construction; precise control of water addition during material transport to dams is implemented based on the different properties of the transported soil and rock, improving dam filling quality; and monitoring the safe distance between dump trucks and construction workers during loading and unloading operations ensures that safety management requirements are met, providing early warnings and preventing construction safety risks.

[0003] In related technologies, multiple deep learning models can be used for step-by-step calculations. Based on the analysis of the dump truck's position using an object detection model, a key point detection model is then used to analyze the positions of the dump truck's front and rear, and a classification model is used to analyze the lifting of the dump truck's hopper.

[0004] However, this method consumes a lot of time in both the early training and later use of the model. At the same time, the propagation of prediction error increases the cumulative error and reduces the prediction accuracy. Furthermore, it fails to fully consider the operating characteristics of dump trucks, resulting in poor accuracy of the above method in analyzing the operation of dump trucks. Summary of the Invention

[0005] This application aims to provide a method, device, electronic device, readable storage medium, and computer program product for monitoring dump truck operations based on multi-label action classification, in order to solve the problem of low accuracy in the prior art for analyzing dump truck operations.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a method for monitoring dump truck operations based on multi-tag action classification, the method comprising:

[0008] A continuous video clip containing a target dump truck is input into a first action classification model to obtain a sequence of action categories output by the first action classification model; wherein, the first action classification model is trained based on sample images containing dump truck operation actions and multiple action category labels corresponding to the sample images;

[0009] Based on the sequence of action categories, the operating status of the target dump truck is determined.

[0010] Optionally, the method further includes:

[0011] Obtain the initial image frames of the video file containing the consecutive video segments;

[0012] The initial image frame and the continuous video segment are respectively input into the first action classification model to obtain the first target detection box and the second target detection box of the target dump truck output by the first action classification model.

[0013] When the first target detection box and the second target detection box are in an overlapping state, it is determined that the target dump truck is in a stopped operation state;

[0014] When the first target detection box and the second target detection box are not overlapping, it is determined that the target dump truck is in a mobile operation state.

[0015] Optionally, the method further includes:

[0016] Determine the first abscissa of the first preset pixel point of the first target detection box, and the second abscissa of the second preset pixel point;

[0017] Determine the third abscissa of the third preset pixel point and the fourth abscissa of the fourth preset pixel point of the second target detection box;

[0018] Calculate the first difference between the first horizontal coordinate and the third horizontal coordinate, and the second difference between the second horizontal coordinate and the fourth horizontal coordinate;

[0019] If the first difference is greater than or equal to the first threshold, and / or the second difference is greater than or equal to the first threshold, it is determined that the first target detection box and the second target detection box are in a non-overlapping state.

[0020] If both the first difference and the second difference are less than the first threshold, it is determined that the first target detection box and the second target detection box are in an overlapping state.

[0021] Optionally, the stop operation state includes a short-term stop operation state and a long-term stop operation state, and determining that the target dump truck is in a stop operation state includes:

[0022] Determine the time interval between the initial image frame and the last image frame of the consecutive video segment;

[0023] If the time interval is greater than or equal to the second threshold, it is determined that the target dump truck is in a long-term stopped operation state;

[0024] If the time interval is less than the second threshold, the target dump truck is determined to be in a short-term stop operation state.

[0025] Optionally, the mobile operation state includes an empty mobile operation state and a material transport operation state, and determining that the target dump truck is in a mobile operation state includes:

[0026] If the action category sequence includes a level hopper and material in the hopper, it is determined that the target dump truck is in a material handling operation state.

[0027] If the sequence of actions includes a level hopper and an empty hopper, the target dump truck is determined to be in an unloaded mobile operation state.

[0028] Optionally, determining the operating status of the target dump truck based on the action category sequence includes:

[0029] Acquire video image frames of the continuous video segment and the preceding image frame of the video image frame;

[0030] The video image frame and the preceding image frame are respectively input into the first action classification model to obtain the third target detection box and the fourth target detection box corresponding to the target dump truck output by the first action classification model.

[0031] Determine the similarity between the image within the first box of the third target detection box and the image within the second box of the fourth target detection box;

[0032] If the action category sequence includes hopper level and hopper containing material, and the similarity is less than a third threshold, the operating state of the target dump truck is determined to be a loading operation state.

[0033] If the action category sequence includes hopper lifting and material in the hopper, and the similarity is less than a third threshold, the operation state is determined to be an unloading operation state.

[0034] Optionally, determining the operating status of the target dump truck based on the action category sequence includes:

[0035] If the action category sequence includes hopper level, hopper top obstruction, and hopper material, the target dump truck's operating state is determined to be a top-obstructed material transport operating state.

[0036] If the action category sequence includes hopper level, hopper top unobstructed, and hopper containing material, the operation state is determined to be the top unobstructed material conveying operation state.

[0037] Optionally, the method further includes:

[0038] When the operation status is unloading operation status, determine the first distance between the personnel around the target dump truck and the target dump truck;

[0039] If the first distance is less than or equal to the fourth threshold, an alarm command is sent to the target dump truck to cause the target dump truck to sound an alarm.

[0040] Secondly, embodiments of this application provide a dump truck operation monitoring device based on multi-tag action classification, the device comprising:

[0041] The first input module is used to input a continuous video clip containing a target dump truck into a first action classification model to obtain an action category sequence output by the first action classification model; wherein, the first action classification model is trained based on sample images containing dump truck operation actions and multiple action category labels corresponding to the sample images;

[0042] The first determining module is used to determine the operating status of the target dump truck based on the action category sequence.

[0043] Optionally, the device further includes:

[0044] The acquisition module is used to acquire the initial image frames of the video file containing the continuous video segments;

[0045] The second input module is used to input the initial image frame and the continuous video segment into the first action classification model respectively, so as to obtain the first target detection box and the second target detection box of the target dump truck respectively output by the first action classification model.

[0046] The second determining module is used to determine that the target dump truck is in a stopped operation state when the first target detection box and the second target detection box are in an overlapping state.

[0047] The third determining module is used to determine that the target dump truck is in a mobile operation state when the first target detection box and the second target detection box are not overlapping.

[0048] Optionally, the device further includes:

[0049] The fourth determining module is used to determine the first abscissa of the first preset pixel point of the first target detection box and the second abscissa of the second preset pixel point;

[0050] The fifth determining module is used to determine the third abscissa of the third preset pixel point and the fourth abscissa of the fourth preset pixel point of the second target detection box;

[0051] The calculation module is used to calculate the first difference between the first horizontal coordinate and the third horizontal coordinate, and the second difference between the second horizontal coordinate and the fourth horizontal coordinate;

[0052] The sixth determining module is used to determine that the first target detection box and the second target detection box are in a non-overlapping state when the first difference is greater than or equal to the first threshold and / or the second difference is greater than or equal to the first threshold.

[0053] The seventh determining module is used to determine that the first target detection box and the second target detection box are in an overlapping state when both the first difference and the second difference are less than the first threshold.

[0054] Optionally, the stop operation state includes a short-term stop operation state and a long-term stop operation state, and the second determining module includes:

[0055] The first determining submodule is used to determine the time interval between the initial image frame and the last image frame of the continuous video segment;

[0056] The second determining submodule is used to determine that the target dump truck is in a long-term stop operation state when the time interval is greater than or equal to the second threshold.

[0057] The third determining submodule is used to determine that the target dump truck is in a short-term stop operation state when the time interval is less than the second threshold.

[0058] Optionally, the mobile operation status includes an empty mobile operation status and a material handling operation status, and the third determining module includes:

[0059] The fourth determination submodule is used to determine that the target dump truck is in a material transportation operation state when the action category sequence includes a hopper level and material in the hopper;

[0060] The fifth determination submodule is used to determine that the target dump truck is in an unloaded mobile operation state when the action category sequence includes a hopper level and no material in the hopper.

[0061] Optionally, the first determining module includes:

[0062] The acquisition submodule is used to acquire the video image frames of the continuous video segment and the preceding image frames of the video image frames;

[0063] The input submodule is used to input the initial image frame and the preceding image frame into the first action classification model respectively, so as to obtain the third target detection box and the fourth target detection box of the target dump truck respectively output by the first action classification model;

[0064] The sixth determining submodule is used to determine the similarity between the image within the first box of the third target detection box and the image within the second box of the fourth target detection box;

[0065] The seventh determination submodule is used to determine the operating state of the target dump truck as a loading operation state when the action category sequence includes hopper level and material in hopper, and the similarity is less than the third threshold.

[0066] The eighth determination submodule is used to determine the operation state as unloading operation state when the action category sequence includes hopper lifting and material in the hopper, and the similarity is less than the third threshold.

[0067] Optionally, the first determining module includes:

[0068] The ninth determining submodule is used to determine the operating state of the target dump truck as a top-obstructed material transport operation state when the action category sequence includes hopper level, hopper top obstruction and hopper material.

[0069] The tenth determination submodule is used to determine the operation state as an unobstructed material conveying operation state when the action category sequence includes hopper level, hopper top unobstructed, and hopper containing material.

[0070] Optionally, the device further includes:

[0071] The eighth determining module is used to determine a first distance between the workers around the target dump truck and the target dump truck when the operation status is unloading operation status;

[0072] An alarm module is used to send an alarm command to the target dump truck when the first distance is less than or equal to a fourth threshold, so that the target dump truck will sound an alarm.

[0073] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the dump truck operation monitoring method based on multi-tag action classification as described above.

[0074] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the dump truck operation monitoring method based on multi-tag action classification as described above.

[0075] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the method described in any of the above claims.

[0076] Compared to existing technologies, this application obtains a sequence of action categories output by inputting continuous video clips containing the target dump truck into a first action classification model. The first action classification model is trained based on sample images containing the dump truck's operational actions and multiple action category labels corresponding to those images. Based on the action category sequence, the operational status of the target dump truck is determined. By using continuous video clips containing the target dump truck and the trained first action classification model to identify multiple action categories constituting the action category sequence, the operational status of the target dump truck is determined. This fully utilizes the various operational posture data contained in the image data of the dump truck during operation, thereby improving the accuracy of operational analysis of dump trucks to a certain extent.

[0077] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0078] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0079] Figure 1 A flowchart illustrating a dump truck operation monitoring method based on multi-tag action classification provided in this application embodiment;

[0080] Figure 2A schematic diagram of the operation of a dump truck provided in an embodiment of this application;

[0081] Figure 3 A schematic diagram of a first action classification model provided in an embodiment of this application;

[0082] Figure 4 A schematic diagram of a decoupling fusion head provided in an embodiment of this application;

[0083] Figure 5 A schematic diagram of a channel space encoder provided in an embodiment of this application;

[0084] Figure 6 A flowchart illustrating a dump truck operation monitoring method based on multi-tag action classification provided in this application embodiment;

[0085] Figure 7 A flowchart illustrating a method for determining the overlap state of a target detection box, provided in an embodiment of this application;

[0086] Figure 8 A flowchart illustrating a specific implementation of a dump truck operation monitoring method based on multi-tag action classification provided in this application embodiment;

[0087] Figure 9 A logic block diagram of a dump truck operation monitoring device based on multi-tag action classification provided in this application embodiment;

[0088] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0089] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0090] Reference Figure 1 , Figure 1 This is a flowchart illustrating a method and system for monitoring dump truck operations based on multi-tag action classification, as provided in an embodiment of this application. The method may include:

[0091] Step 101: Input the continuous video clips containing the target dump truck into the first action classification model to obtain the action category sequence output by the first action classification model; wherein, the first action classification model is trained based on sample images containing dump truck operation actions and multiple action category labels corresponding to the sample images.

[0092] In embodiments of this application, a dump truck is a transport vehicle that includes a liftable hopper.

[0093] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the operation of a dump truck provided in an embodiment of this application. In the figure, "1" represents the liftable hopper of the dump truck, and "2" represents the chassis of the dump truck.

[0094] In the embodiments of this application, the initial action classification network model corresponding to the first action classification model can be a commonly used action classification model. The first action classification model can be obtained by training the initial action classification network model based on sample images containing dump truck operation actions and multiple action category labels corresponding to the sample images.

[0095] For example, such as Figure 3 As shown, Figure 3This diagram illustrates a first action classification model provided in an embodiment of this application. In the diagram, the 2D backbone network can be constructed based on a YOLOv7 model without prior anchor boxes. The backbone network in the YOLOv7 model can be constructed based on a Shufflenetv2 model or an ELANNet model, used to extract three feature maps of different sizes from continuous video clips containing a target dump truck. The 3D backbone network can be constructed based on a Shufflenetv2 model or a ResNet101 model, used to extract spatiotemporal features from K consecutive video clips of continuous dump truck operations. "×2" indicates two consecutive convolutions. First, the continuous video clips can be input into the 2D backbone network, outputting three feature maps of different sizes. Then, these three feature maps of different sizes are each input into a convolutional layer with a 1×1 kernel, and then passed through two parallel convolutional layers with 3×3 kernels ×2, respectively, to obtain three classification features (classification feature 1, classification feature 2, and classification feature 3) and three regression features (regression feature 1, regression feature 2, and regression feature 3). The classification features are used to generate action classification results for dump trucks, such as whether the hopper is raised, whether the hopper is obstructed, and whether there is material present. The regression features are used to generate target detection boxes for dump trucks, used to identify them. After obtaining the classification and regression features, groups of classification and regression features (classification feature 1 + regression feature 1, classification feature 2 + regression feature 2, classification feature 3 + regression feature 3) are input into a decoupled fusion head to obtain the classification result, regression result, and confidence score of the dump truck. Since three feature maps of different sizes are input, there can be differences in categories between the classification features; therefore, multiple decoupled fusion heads can output multiple types of classification results. Furthermore, after acquiring the spatiotemporal features of K consecutive video segments, the 3D backbone network directly inputs them into the third decoupled fusion head, while simultaneously inputting the spatiotemporal features obtained after one upsampling into the second decoupled fusion head and the spatiotemporal features obtained after two upsamplings into the first decoupled fusion head. Finally, by combining the outputs of the three decoupled fusion heads, we can obtain the classification result (action classification of the dump truck), regression result (object detection box of the dump truck), and confidence score of the first action classification model. Among these, such as... Figure 4 As shown, Figure 4This diagram illustrates a decoupled fusion head provided in an embodiment of this application. In the diagram, classification features and spatiotemporal features are input into a defined channel spatial encoder module for feature fusion. After calculation by two 3×3 convolutional layers and one 1×1 convolutional layer, the classification result of the dump truck action in the current frame image is output. Similarly, regression features and spatiotemporal features are input into the defined channel spatial encoder module for feature fusion. After calculation by two 3×3 convolutional layers, the target detection box position (regression result) of the dump truck in the current frame image is calculated in parallel by one 1×1 convolutional layer. Finally, the confidence score of the target detection box of the dump truck in the current frame image is calculated by one 1×1 convolutional layer. The channel spatial encoder is as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of a channel spatial encoder provided in an embodiment of this application. Taking classification features as an example, the input classification features and spatiotemporal features are first concatenated along the channel dimension, and preliminary channel integration is achieved using a convolutional layer with a kernel size of 1×1 and a kernel size of 3×3. By calculating the channel attention matrix and the spatial attention matrix, the features after spatial self-attention calculation are obtained. The attention matrix can be manually preset to specify the emphasis of the output results. Figure 5 Other elements, such as channel merging, dimension transformation, query, key, value, and convolution kernel, are all existing spatial encoder concepts and will not be elaborated upon here.

[0096] In the embodiments of this application, when using the first action classification model to monitor the operation of target dump trucks, the angle at which the target dump truck image is captured is not always located at the direct side of the dump truck. Typically, the elevation of the capture point is higher than the elevation of the dump truck's work surface, and the work surface can include multiple target dump trucks, allowing for simultaneous monitoring of multiple target dump trucks. Multiple target dump trucks can be distinguished based on their detection frames when entering the work surface. Continuous video clips can be extracted from video files. After inputting continuous video clips containing target dump trucks into the first action classification model, multiple action categories output by the first action classification model can be obtained, forming an action category sequence. According to the relevant description of the first action classification model, the classification result can include at least three categories; that is, at least three different types of action categories can exist in the action category sequence.

[0097] Step 102: Determine the operating status of the target dump truck based on the action category sequence.

[0098] In the embodiments of this application, after obtaining an action category sequence containing multiple action categories, the operating state of the target dump truck can be determined based on each action category in the action category sequence and the changes in the images within the dump truck detection box in the preceding and following video frames. For example, if the action categories include hopper lifting and material in the hopper, and the similarity between the video image frames of the consecutive video segments and the images within the detection boxes of the preceding image frames is less than a third threshold, then it can be said that the operating state of the target dump truck is unloading operation.

[0099] In the embodiments of this application, by inputting continuous video clips containing the target dump truck into a first action classification model, an action category sequence output by the first action classification model is obtained. The first action classification model is trained based on sample images containing the dump truck's operational actions and multiple action category labels corresponding to those sample images. Based on the action category sequence, the operational state of the target dump truck is determined. The training first action classification model can identify multiple action categories of the target dump truck constituting an action category sequence from the continuous video clips containing the target dump truck, and the operational state of the target dump truck is determined through the action category sequence. This fully utilizes the various operational posture data contained in the image data of the dump truck during operation, thereby improving the accuracy of operational analysis of the dump truck to a certain extent.

[0100] Reference Figure 6 , Figure 6 This is a flowchart illustrating another method and system for monitoring dump truck operations based on multi-tag action classification provided in this application. The method may include:

[0101] Step 201: Input the continuous video clips containing the target dump truck into the first action classification model to obtain the action category sequence output by the first action classification model; wherein, the first action classification model is trained based on sample images containing dump truck operation actions and multiple action category labels corresponding to the sample images.

[0102] In the embodiments of this application, the implementation of this step can be referred to the embodiment of step 101, and will not be repeated here.

[0103] Step 202: Determine the operating status of the target dump truck based on the action category sequence.

[0104] In the embodiments of this application, the implementation of this step can be referred to the embodiment of step 102, and will not be repeated here.

[0105] Optionally, step 202 may include the following sub-steps:

[0106] Sub-step 2021: Obtain the video image frames of the continuous video segment and the preceding image frames of the video image frames.

[0107] In the embodiments of this application, any image frame can be obtained as a video image frame from the video file where the consecutive video segments are located. After obtaining the video image frame, the preceding image frame of the video image frame can be obtained.

[0108] Sub-step 2022: Input the initial image frame and the preceding image frame into the first action classification model respectively to obtain the third target detection box and the fourth target detection box of the target dump truck output by the first action classification model respectively.

[0109] In embodiments of this application, video image frames from a continuous video segment and preceding image frames can be input into a first action classification model. The first action classification model can process the video image frames and preceding image frames to output a third target detection box and a fourth target detection box for the target dump truck. The third target detection box can correspond to the initial image frame, and the fourth target detection box can correspond to the preceding image frame. Both the third and fourth target detection boxes can contain the same target dump truck.

[0110] Sub-step 2023: Determine the similarity between the first frame image of the third target detection box and the second frame image of the fourth target detection box.

[0111] In embodiments of this application, the similarity between the first frame image of the third target detection box and the second frame image of the fourth target detection box can be calculated using a preset method. For example, the proportion of pixels with the same pixel value in the first and second frame images within the third target detection box can be obtained, and this proportion can be used as the similarity between the first and second frame images. Alternatively, it can be determined whether the pixel values ​​of pixels at the same position in the first and second frame images are the same, and the number of pixels with the same pixel value can be counted. The ratio between this number and the total number of pixels can then be used as the similarity between the first and second frame images.

[0112] Sub-step 2024: If the action category sequence includes hopper level and material in hopper, and the similarity is less than the third threshold, determine the operating state of the target dump truck as loading operation state.

[0113] In embodiments of this application, the action category sequence may include three action categories: one of "hopper level" and "hopper raised", one of "material present in hopper" and "material absent in hopper", and one of "top of hopper obstructed" and "top of hopper unobstructed". If the action category sequence includes both "hopper level" and "material present in hopper", and the similarity is less than a third threshold, then the operating state of the target dump truck can be determined to be a loading operation state.

[0114] Sub-step 2025: If the action category sequence includes hopper lifting and material in the hopper, and the similarity is less than the third threshold, determine the operation state as unloading operation state.

[0115] In the embodiments of this application, if the action category sequence includes hopper lifting and hopper containing material, and if the similarity is less than the third threshold, it indicates that the state of the dump truck has changed significantly between two frames of images. At this time, it can be determined that the operating state of the target dump truck is the unloading operation state.

[0116] In the embodiments of this application, by acquiring video image frames of the continuous video segment and preceding image frames of the video image frames, and inputting the video image frames and preceding image frames into the first action classification model respectively, the third target detection box and the fourth target detection box corresponding to the target dump truck are obtained respectively output by the first action classification model. The similarity between the first frame image of the third target detection box and the second frame image of the fourth target detection box is determined. If the action category sequence includes hopper level and material in hopper, and the similarity is less than a third threshold, the operation state of the target dump truck is determined to be loading operation state. If the action category sequence includes hopper lifting and material in hopper, and the similarity is less than a third threshold, the operation state is determined to be unloading operation state. The accuracy and reliability of the operation state determination can be improved by combining the action categories included in the action category sequence with the similarity between the images in the detection boxes of the video image frames of the continuous video segment and the preceding image frames.

[0117] Sub-step 2026: If the action category sequence includes hopper level, hopper top obstruction, and hopper material, determine the target dump truck's operating state as top obstruction material transport operation state.

[0118] In the embodiments of this application, if the action category sequence includes hopper level, hopper top obstruction, and hopper material, then the operating state of the target dump truck can be determined as top obstruction material transportation operation state.

[0119] Sub-step 2027: If the action category sequence includes hopper level, hopper top unobstructed, and hopper containing material, determine the operation state as hopper top unobstructed material conveying operation state.

[0120] In the embodiments of this application, if the action category sequence includes hopper level, hopper top unobstructed, and hopper containing material, then the operating state of the target dump truck can be determined as the top unobstructed material transportation operating state.

[0121] In the embodiments of this application, the operating state of the target dump truck is determined to be a top-obstructed material handling operating state when the action category sequence includes the following conditions: the hopper is horizontal, the top of the hopper is not obstructed, and there is material in the hopper. When the action category sequence includes the following conditions: the hopper is horizontal, the top of the hopper is not obstructed, and there is material in the hopper, the operating state is determined to be a top-unobstructed material handling operating state. The operating state of the target dump truck can be determined based on the action category sequence, which improves the accuracy and reliability of the operating state.

[0122] Step 203: Obtain the initial image frame of the video file containing the continuous video segments.

[0123] In embodiments of this application, the video file containing the continuous video segments can be determined first, and then an initial image frame can be extracted from that video file. The initial image frame can refer to the first frame of the video file. If the continuous video segments are considered as the video file itself, then the initial image frame can be the first frame of the continuous video segments.

[0124] Step 204: Input the initial image frame and the continuous video segment into the first action classification model respectively to obtain the first target detection box and the second target detection box of the target dump truck output by the first action classification model respectively.

[0125] In embodiments of this application, an initial image frame and consecutive video clips can be input into a first action classification model, thereby obtaining a first target detection box and a second target detection box of the target dump truck, respectively output by the first action classification model. The target dump truck can be contained within both the first and second target detection boxes.

[0126] Step 205: When the first target detection box and the second target detection box are in an overlapping state, it is determined that the target dump truck is in a stopped operation state.

[0127] In the embodiments of this application, if the first target detection box and the second target detection box are in an overlapping state, it can be considered that the target dump truck has not moved between the initial image frame and the continuous video segment, and at this time, it can be determined that the target dump truck is in a stopped operation state. When determining the overlapping state, image errors caused by camera point and environmental jitter in the video file can be considered simultaneously. That is, when the deviation between the first target detection box and the second target detection box is within a preset range, it can be determined that the first target detection box and the second target detection box are in an overlapping state.

[0128] Optionally, step 205 may include the following sub-steps:

[0129] Sub-step 2051: Determine the time interval between the initial image frame and the last image frame of the continuous video segment.

[0130] In embodiments of this application, the stop operation state can include a short-term stop operation state and a long-term stop operation state. The time interval between the initial image frame and the last image frame of the continuous video segment can be determined based on the time track of the video file. The timing speed of the video file can be consistent with the normal timing speed in reality; the last image frame can refer to the last frame of the continuous video segment.

[0131] Sub-step 2052: If the time interval is greater than or equal to the second threshold, determine that the target dump truck is in a long-term stop operation state.

[0132] In the embodiments of this application, a distinction point between long and short time periods, namely a second threshold, can be set. If the time interval is greater than or equal to the second threshold, it can be determined that the target dump truck is in a long-term stopped operation state.

[0133] Sub-step 2053: If the time interval is less than the second threshold, determine that the target dump truck is in a short-term stop operation state.

[0134] In the embodiments of this application, if the time interval is less than the second threshold, it can be determined that the target dump truck is in a short-term stop operation state.

[0135] In the embodiments of this application, by determining the time interval between the initial image frame and the last image frame of the continuous video segment, if the time interval is greater than or equal to a second threshold, it is determined that the target dump truck is in a long-term stop operation state; if the time interval is less than the second threshold, it is determined that the target dump truck is in a short-term stop operation state. The accuracy and availability of the target dump truck's operation status can be improved by determining whether the target dump truck is in a short-term stop operation state or a long-term stop operation state based on the time interval between the initial image frame and the last image frame of the continuous video segment.

[0136] Step 206: If the first target detection box and the second target detection box are not overlapping, determine that the target dump truck is in a mobile operation state.

[0137] In the embodiments of this application, if the first target detection box and the second target detection box are in a non-overlapping state, it can be determined that the target dump truck is in a mobile operation state. When determining the non-overlapping state, image errors caused by camera movement and environmental jitter in the video file can be considered simultaneously. That is, when the deviation between the first target detection box and the second target detection box is outside a preset range, it can be determined that the first target detection box and the second target detection box are in a non-overlapping state.

[0138] Optionally, step 206 may include the following sub-steps:

[0139] Sub-step 2061: If the action category sequence includes hopper level and material in the hopper, determine that the target dump truck is in a material transportation operation state.

[0140] In embodiments of this application, the mobile operation state may include a material handling operation state and an empty mobile operation state, wherein a loaded state refers to the presence of material in the hopper of the dump truck, and an empty state refers to the absence of material in the hopper of the dump truck. If the action category sequence includes "hopper level" and "material present in hopper", it can be determined that the target dump truck is in a material handling operation state.

[0141] Sub-step 2062: If the action category sequence includes a hopper level or no material in the hopper, determine that the target dump truck is in an unloaded mobile operation state.

[0142] In the embodiments of this application, if the action category sequence includes a level hopper and no material in the hopper, it can be determined that the target dump truck is in an unloaded mobile operation state.

[0143] In the embodiments of this application, when the action category sequence includes a level hopper and material in the hopper, it is determined that the target dump truck is in a material transportation operation state. When the action category sequence includes a level hopper and material in the hopper, it is determined that the target dump truck is in an empty moving operation state. The accuracy and reliability of the target dump truck's operation state can be improved by combining the action category sequence with the first target detection box and the second target detection box.

[0144] In the embodiments of this application, by obtaining the initial image frames of the video file containing the continuous video segments, and inputting the initial image frames and the continuous video segments into the first action classification model respectively, the first target detection box and the second target detection box of the target dump truck output by the first action classification model are obtained. When the first target detection box and the second target detection box are in an overlapping state, it is determined that the target dump truck is in a stopped operation state. When the first target detection box and the second target detection box are in a non-overlapping state, it is determined that the target dump truck is in a moving operation state. The accuracy of the target dump truck's operation state is improved by determining whether the target dump truck is in a stopped operation state or a moving operation state based on whether the first target detection box and the second target detection box corresponding to the initial image frame and the continuous video segments are in an overlapping or non-overlapping state.

[0145] Step 207: When the operation status is unloading operation status, determine the first distance between the personnel around the target dump truck and the target dump truck.

[0146] In the embodiments of this application, if the dump truck is in an unloading operation state, a first distance between the personnel around the target dump truck and the target dump truck can be determined. The first distance can be determined by target recognition combined with proportional magnification, or by using a ranging device to perform distance measurement and difference calculation.

[0147] Step 208: If the first distance is less than or equal to the fourth threshold, send an alarm command to the target dump truck to cause the target dump truck to sound an alarm.

[0148] In the embodiments of this application, since the materials loaded by dump trucks are typically stones or soil, there may be certain safety hazards during unloading. The target dump truck can be equipped with an alarm device, and the alarm device can be connected to the dump truck's operation monitoring and control center. A fourth threshold can be manually set as a safe distance. If the first distance is less than or equal to the fourth threshold, an alarm command can be sent to the target dump truck to trigger an alarm via the alarm device.

[0149] In one possible embodiment, different operations can be performed on the target dump truck based on the type of material it carries. For example, if the target dump truck carries stone, the time for adding water to the hopper is the same as the time for adding water for stone; if the target dump truck carries soil, the time for adding water to the hopper is the same as the time for adding water for soil. The time for adding water for stone and the time for adding water for soil are different, and both can be predetermined experimentally. Furthermore, the type of material in the target dump truck's hopper can be determined by the texture of the material loading in a continuous video clip; if the texture is clear, the material type is stone; if the texture is blurry, the material type is soil.

[0150] In the embodiments of this application, when the operation state is unloading operation state, a first distance between the personnel around the target dump truck and the target dump truck is determined. If the first distance is less than or equal to a fourth threshold, an alarm command is sent to the target dump truck to make the target dump truck alarm. This can improve the safety of the target dump truck during operation to a certain extent when the first distance between the target dump truck and the surrounding personnel is less than or equal to the fourth threshold.

[0151] Based on the above implementation method, refer to Figure 7 , Figure 7 This is a flowchart illustrating a method for determining the overlap state of a target detection box according to an embodiment of this application. Before step 205, the method may further include:

[0152] Step 209: Determine the first horizontal coordinate of the first preset pixel point of the first target detection box and the second horizontal coordinate of the second preset pixel point.

[0153] In embodiments of this application, whether the target detection boxes overlap can be determined by whether the abscissas of the preset pixels of the first target detection box and the second target detection box are equal. The first abscissa of the first preset pixel of the first target detection box and the second abscissa of the second preset pixel can be determined. The first preset pixel can be the pixel located at the top-left corner of the first target detection box, and the second preset pixel can be the pixel located at the bottom-right corner of the first target detection box.

[0154] Step 210: Determine the third horizontal coordinate of the third preset pixel point and the fourth horizontal coordinate of the fourth preset pixel point of the second target detection box.

[0155] In embodiments of this application, the third abscissa of the third preset pixel point and the fourth abscissa of the fourth preset pixel point of the second target detection box can be further determined. The third preset pixel point may be the pixel point where the upper left corner vertex of the second target detection box is located, and the fourth preset pixel point may be the pixel point where the lower right corner vertex of the second target detection box is located.

[0156] Step 211: Calculate the first difference between the first horizontal coordinate and the third horizontal coordinate, and the second difference between the second horizontal coordinate and the fourth horizontal coordinate.

[0157] In the embodiments of this application, since the third preset pixel and the first preset pixel are pixels at the same position in different target detection boxes, the first horizontal coordinate and the third horizontal coordinate are corresponding coordinates. Similarly, the second horizontal coordinate and the fourth horizontal coordinate are corresponding coordinates. Based on this, the first difference between the first horizontal coordinate and the third horizontal coordinate, and the second difference between the second horizontal coordinate and the fourth horizontal coordinate can be calculated.

[0158] Step 212: If the first difference is greater than or equal to the first threshold, and / or the second difference is greater than or equal to the first threshold, determine that the first target detection box and the second target detection box are in a non-overlapping state.

[0159] In the embodiments of this application, when determining whether the first target detection box and the second target detection box are in an overlapping state, the error caused by the shaking of the operation monitoring point and the environment can be considered, and an error threshold can be set as the first threshold based on this error. When both the first difference and the second difference are greater than or equal to the first threshold, it can be determined that the first target detection box and the second target detection box are in a non-overlapping state.

[0160] Step 213: If both the first difference and the second difference are less than the first threshold, determine that the first target detection box and the second target detection box are in an overlapping state.

[0161] In the embodiments of this application, when both the first difference and the second difference are less than the first threshold, it can be determined that the first target detection box and the second target detection box are in an overlapping state.

[0162] In one possible embodiment, if both the first difference and the second difference are less than the first threshold, the first ordinate of the first preset pixel, the second ordinate of the second preset pixel, the third ordinate of the third preset pixel, and the fourth ordinate of the fourth preset pixel can be further obtained. If the third difference between the first and third ordinates and the fourth difference between the second and fourth ordinates are both less than the first threshold, it can be determined that the first target detection box and the second target detection box are in an overlapping state.

[0163] In the embodiments of this application, by determining the first abscissa of the first preset pixel point of the first target detection box and the second abscissa of the second preset pixel point, determining the third abscissa of the third preset pixel point of the second target detection box and the fourth abscissa of the fourth preset pixel point, calculating the first difference between the first abscissa and the third abscissa, and the second difference between the second abscissa and the fourth abscissa, if both the first difference and the second difference are greater than or equal to the first threshold, it is determined that the first target detection box and the second target detection box are in a non-overlapping state; if both the first difference and the second difference are less than the first threshold, it is determined that the first target detection box and the second target detection box are in an overlapping state. The accuracy and reliability of the overlapping state and non-overlapping state judgment are improved by determining whether the first target detection box and the second target detection box are in an overlapping state or a non-overlapping state based on the pixel coordinate difference of the preset pixel points.

[0164] Reference Figure 8 , Figure 8 This flowchart illustrates a specific implementation of a dump truck operation monitoring method based on multi-label action classification, as provided in this application. First, a multi-label classification image dataset for dump trucks is constructed using sample operation videos. Each sample image in this dataset can contain multiple labels (e.g., whether the hopper is level, whether the hopper is obstructed, whether the hopper is loaded, and the type of material being loaded). These labels can be manually annotated. Next, a multi-label action classification model for dump trucks can be constructed. This model framework can use existing models and be trained using the multi-label classification image dataset for dump trucks constructed in the above process. Finally, the trained multi-label action classification model can be used to monitor dump truck operations, and the monitoring results can be used to optimize the dump truck operation process, such as optimizing dump truck scheduling and improving dump truck work efficiency.

[0165] Reference Figure 9 , Figure 9 This is a logic block diagram of a dump truck operation monitoring device based on multi-tag action classification provided in an embodiment of this application. The device 900 may include:

[0166] The first input module 901 is used to input a continuous video clip containing a target dump truck into the first action classification model to obtain an action category sequence output by the first action classification model; wherein, the first action classification model is trained based on a sample image containing the operation action of the dump truck and multiple action category labels corresponding to the sample image.

[0167] The first determining module 902 is used to determine the operating status of the target dump truck based on the action category sequence.

[0168] Optionally, the device 900 further includes:

[0169] The acquisition module is used to acquire the initial image frames of the video file containing the continuous video segments;

[0170] The second input module is used to input the initial image frame and the continuous video segment into the first action classification model respectively, so as to obtain the first target detection box and the second target detection box of the target dump truck respectively output by the first action classification model.

[0171] The second determining module is used to determine that the target dump truck is in a stopped operation state when the first target detection box and the second target detection box are in an overlapping state.

[0172] The third determining module is used to determine that the target dump truck is in a mobile operation state when the first target detection box and the second target detection box are not overlapping.

[0173] Optionally, the device 900 further includes:

[0174] The fourth determining module is used to determine the first abscissa of the first preset pixel point of the first target detection box and the second abscissa of the second preset pixel point;

[0175] The fifth determining module is used to determine the third abscissa of the third preset pixel point and the fourth abscissa of the fourth preset pixel point of the second target detection box;

[0176] The calculation module is used to calculate the first difference between the first horizontal coordinate and the third horizontal coordinate, and the second difference between the second horizontal coordinate and the fourth horizontal coordinate;

[0177] The sixth determining module is used to determine that the first target detection box and the second target detection box are in a non-overlapping state when the first difference is greater than or equal to the first threshold and / or the second difference is greater than or equal to the first threshold.

[0178] The seventh determining module is used to determine that the first target detection box and the second target detection box are in an overlapping state when both the first difference and the second difference are less than the first threshold.

[0179] Optionally, the stop operation state includes a short-term stop operation state and a long-term stop operation state, and the second determining module includes:

[0180] The first determining submodule is used to determine the time interval between the initial image frame and the last image frame of the continuous video segment;

[0181] The second determining submodule is used to determine that the target dump truck is in a long-term stop operation state when the time interval is greater than or equal to the second threshold.

[0182] The third determining submodule is used to determine that the target dump truck is in a short-term stop operation state when the time interval is less than the second threshold.

[0183] Optionally, the mobile operation status includes an empty mobile operation status and a material handling operation status, and the third determining module includes:

[0184] The fourth determination submodule is used to determine that the target dump truck is in a material transportation operation state when the action category sequence includes a hopper level and material in the hopper;

[0185] The fifth determination submodule is used to determine that the target dump truck is in an unloaded mobile operation state when the action category sequence includes a hopper level and no material in the hopper.

[0186] Optionally, the first determining module 902 includes:

[0187] The acquisition submodule is used to acquire the video image frames of the continuous video segment and the preceding image frames of the video image frames;

[0188] The input submodule is used to input the initial image frame and the preceding image frame into the first action classification model respectively, so as to obtain the third target detection box and the fourth target detection box of the target dump truck respectively output by the first action classification model;

[0189] The sixth determining submodule is used to determine the similarity between the image within the first box of the third target detection box and the image within the second box of the fourth target detection box;

[0190] The seventh determination submodule is used to determine the operating state of the target dump truck as a loading operation state when the action category sequence includes hopper level and material in hopper, and the similarity is less than the third threshold.

[0191] The eighth determination submodule is used to determine the operation state as unloading operation state when the action category sequence includes hopper lifting and material in the hopper, and the similarity is less than the third threshold.

[0192] Optionally, the first determining module 902 includes:

[0193] The ninth determining submodule is used to determine the operating state of the target dump truck as a top-obstructed material transport operation state when the action category sequence includes hopper level, hopper top obstruction and hopper material.

[0194] The tenth determination submodule is used to determine the operation state as an unobstructed material conveying operation state when the action category sequence includes hopper level, hopper top unobstructed, and hopper containing material.

[0195] Optionally, the device 900 further includes:

[0196] The eighth determining module is used to determine a first distance between the workers around the target dump truck and the target dump truck when the operation status is unloading operation status;

[0197] An alarm module is used to send an alarm command to the target dump truck when the first distance is less than or equal to a fourth threshold, so that the target dump truck will sound an alarm.

[0198] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0199] The dump truck operation monitoring device based on multi-label action classification has the same advantages over the prior art as the dump truck operation monitoring method based on multi-label action classification described in the foregoing embodiments, and will not be repeated here.

[0200] This application provides an electronic device, see [link to relevant documentation] Figure 10 The electronic device 70 includes a processor 701, a memory 702, and a computer program 7021 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 7021, it implements the dump truck operation monitoring method based on multi-label action classification of the foregoing embodiment.

[0201] This application provides a readable storage medium storing a program or instructions, which, when executed by a processor, implements the dump truck operation monitoring method based on multi-tag action classification as described above.

[0202] This application also provides a computer program product that, when run on an electronic device, enables a processor to execute the steps in the dump truck operation monitoring method based on multi-tag action classification disclosed in this application.

[0203] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0204] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0207] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0208] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0209] The above provides a detailed description of a dump truck operation monitoring method based on multi-label action classification provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for monitoring dump truck operations based on multi-label action classification, characterized in that, The method includes: A continuous video clip containing the target dump truck is input into a first action classification model to obtain an action category sequence output by the first action classification model; wherein, the first action classification model is trained based on sample images containing the dump truck's operation actions and multiple action category labels corresponding to the sample images; the action category sequence includes three action categories, namely one of hopper level and hopper lifting, one of material in hopper and material not in hopper, and one of hopper top obstruction and hopper top unobstructed; the dump truck is a transport vehicle including a liftable hopper; Based on the sequence of action categories, the operating status of the target dump truck is determined; Determining the operating status of the target dump truck based on the action category sequence includes: Acquire video image frames of the continuous video segment and the preceding image frame of the video image frame; The video image frame and the preceding image frame are respectively input into the first action classification model to obtain the third target detection box and the fourth target detection box corresponding to the target dump truck output by the first action classification model; the third target detection box corresponds to the video image frame, the fourth target detection box corresponds to the preceding image frame, and both the third target detection box and the fourth target detection box contain the same target dump truck. Determine the similarity between the image within the first box of the third target detection box and the image within the second box of the fourth target detection box; If the action category sequence includes hopper level and hopper containing material, and the similarity is less than a third threshold, the operating state of the target dump truck is determined to be a loading operation state. If the action category sequence includes hopper lifting and material in the hopper, and the similarity is less than a third threshold, the operation state is determined to be an unloading operation state.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the initial image frames of the video file containing the consecutive video segments; The initial image frame and the continuous video segment are respectively input into the first action classification model to obtain the first target detection box and the second target detection box of the target dump truck output by the first action classification model. When the first target detection box and the second target detection box are in an overlapping state, it is determined that the target dump truck is in a stopped operation state; When the first target detection box and the second target detection box are not overlapping, it is determined that the target dump truck is in a mobile operation state.

3. The method according to claim 2, characterized in that, The method further includes: Determine the first abscissa of the first preset pixel point of the first target detection box, and the second abscissa of the second preset pixel point; Determine the third abscissa of the third preset pixel point and the fourth abscissa of the fourth preset pixel point of the second target detection box; Calculate the first difference between the first horizontal coordinate and the third horizontal coordinate, and the second difference between the second horizontal coordinate and the fourth horizontal coordinate; If the first difference is greater than or equal to the first threshold, and / or the second difference is greater than or equal to the first threshold, it is determined that the first target detection box and the second target detection box are in a non-overlapping state. If both the first difference and the second difference are less than the first threshold, it is determined that the first target detection box and the second target detection box are in an overlapping state.

4. The method according to claim 2, characterized in that, The shutdown state includes short-term shutdown state and long-term shutdown state. Determining that the target dump truck is in a shutdown state includes: Determine the time interval between the initial image frame and the last image frame of the consecutive video segment; If the time interval is greater than or equal to the second threshold, it is determined that the target dump truck is in a long-term stopped operation state; If the time interval is less than the second threshold, the target dump truck is determined to be in a short-term stop operation state.

5. The method according to claim 2, characterized in that, The mobile operation status includes an empty mobile operation status and a material transport operation status. Determining that the target dump truck is in a mobile operation status includes: If the action category sequence includes a level hopper and material in the hopper, it is determined that the target dump truck is in a material handling operation state. If the sequence of actions includes a level hopper and an empty hopper, the target dump truck is determined to be in an unloaded mobile operation state.

6. The method according to claim 1, characterized in that, Determining the operating status of the target dump truck based on the action category sequence includes: If the action category sequence includes hopper level, hopper top obstruction, and hopper material, the target dump truck's operating state is determined to be a top-obstructed material transport operating state. If the action category sequence includes hopper level, hopper top unobstructed, and hopper containing material, the operation state is determined to be the top unobstructed material conveying operation state.

7. The method according to claim 1, characterized in that, The method further includes: When the operation status is unloading operation status, determine the first distance between the personnel around the target dump truck and the target dump truck; If the first distance is less than or equal to the fourth threshold, an alarm command is sent to the target dump truck to cause the target dump truck to sound an alarm.

8. A dump truck operation monitoring device based on multi-label action classification, characterized in that, The device includes: The first input module is used to input continuous video clips containing the target dump truck into the first action classification model to obtain the action category sequence output by the first action classification model; wherein, the first action classification model is trained based on sample images containing the dump truck's operation actions and multiple action category labels corresponding to the sample images; the action category sequence includes three action categories, namely one of hopper level and hopper lifting, one of material in hopper and material not in hopper, and one of hopper top obstruction and hopper top unobstructed; the dump truck is a transport vehicle including a liftable hopper; The first determining module is used to determine the operating status of the target dump truck based on the action category sequence; The first determining module includes: The acquisition submodule is used to acquire the video image frames of the continuous video segment and the preceding image frame of the video image frame; The input submodule is used to input the video image frame and the preceding image frame into the first action classification model respectively, and obtain the third target detection box and the fourth target detection box corresponding to the target dump truck output by the first action classification model respectively; the third target detection box corresponds to the video image frame, the fourth target detection box corresponds to the preceding image frame, and both the third target detection box and the fourth target detection box contain the same target dump truck. The sixth determining submodule is used to determine the similarity between the image within the first box of the third target detection box and the image within the second box of the fourth target detection box; The seventh determination submodule is used to determine the operating state of the target dump truck as a loading operation state when the action category sequence includes hopper level and material in hopper, and the similarity is less than the third threshold. The eighth determination submodule is used to determine the operation state as unloading operation state when the action category sequence includes hopper lifting and material in the hopper, and the similarity is less than the third threshold.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Loading state recognition method, device and equipment and computer readable storage medium

    CN115775361A

  • Monitoring method, device and equipment for hoisting operation and storage medium

    CN117775985A