A real-time monitoring and analysis method of excavation volume during construction based on motion recognition

Abstract

The embodiment of the present application provides a method for real-time monitoring and analysis of excavation volume during construction period based on action recognition, which specifically includes: performing target detection on multiple frame images arranged in time; tracking construction machinery in multiple frame images arranged in time; determining equipment matching pairs in multiple frame images arranged in time according to equipment categories, equipment identifications and detection frames corresponding to construction machinery in frame images; performing action recognition on construction machinery in multiple frame images arranged in time; when the action category of the excavation equipment in the equipment matching pair changes by a preset, it is considered that the excavation equipment in the equipment matching pair has completed one excavation, and according to the rated excavation capacity corresponding to the excavation equipment, the cumulative excavation volume corresponding to the construction area to be counted is determined. The embodiment of the present application can realize real-time monitoring of excavation volume during construction period at a relatively low cost and relatively fast speed, and can effectively improve the accuracy of real-time monitoring of excavation volume during construction period.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of open excavation of earth and stone, and in particular to a method for real-time monitoring and analysis of excavation volume during construction based on motion recognition. Background Art

[0002] Open excavation of earthwork is one of the common types of construction operations. Real-time monitoring of excavation volume is of great significance for closely following the progress of on-site construction, and can provide strong support for the planning of excavation construction plans and the timely deployment of the number of construction machinery. For example, the construction of pumped-storage power station projects usually involves simultaneous excavation at multiple construction sites and variable on-site construction conditions, which determines that the real-time monitoring and analysis of excavation volume during the construction period of pumped-storage power station projects needs to continuously cover multiple construction sites.

[0003] When calculating the excavation volume, related technologies usually use drone oblique photography to obtain digital elevation model data before excavation, or use three-dimensional laser scanning technology to collect spatial point cloud data to record the original terrain conditions; after excavation, the same technology is used again to obtain the corresponding data; by comparing the data before and after excavation, the volume corresponding to the difference between the two is calculated, and this volume is the excavation volume, thereby realizing the calculation of the excavation volume.

[0004] In practical applications, whether it is drone oblique photography or 3D laser scanning technology, professionals are required to operate the equipment, collect data, and sort out the subsequent data during the operation process, which undoubtedly consumes a certain amount of manpower costs. In addition, the amount of data collected is huge and complex, and the subsequent data processing process is cumbersome, requiring a lot of time to clean, analyze and calculate the data, resulting in a longer data processing cycle, which in turn makes it impossible to timely feedback the calculation results of the excavation volume, and the real-time performance is poor, making it difficult to meet the needs of real-time monitoring and analysis of the excavation volume during the construction period. Summary of the invention

[0005] The embodiment of the present application provides a real-time monitoring and analysis method for excavation volume during construction period based on motion recognition, which can realize real-time monitoring of excavation volume during construction period at a relatively low cost and at a relatively fast speed, and can effectively improve the accuracy of real-time monitoring of excavation volume during construction period.

[0006] Correspondingly, the embodiments of the present application also provide a real-time monitoring and analysis device for excavation volume during construction based on motion recognition, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the embodiment of the present application discloses a method for real-time monitoring and analysis of excavation volume during construction period based on action recognition, comprising: Acquire video stream data of the construction area to be counted; the video stream data includes: a plurality of frame images arranged in time; Performing target detection on the frame image to obtain equipment categories and detection frames corresponding to the construction machinery in the frame image; the equipment categories include: self-unloading equipment and excavating equipment; Tracking the construction machinery in the multiple frame images arranged in time according to the equipment category and the detection frame corresponding to the construction machinery in the frame image to obtain the equipment identification corresponding to the construction machinery; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating equipment in the equipment matching pair, it is considered that the excavating equipment in the equipment matching pair completes one excavation, and the cumulative excavation volume corresponding to the construction area to be counted is determined according to the rated excavation capacity corresponding to the excavating equipment.

[0008] The embodiment of the present application also discloses a device for real-time monitoring and analysis of excavation volume during construction based on action recognition, the device comprising: The video stream acquisition module is used to acquire the video stream data of the construction area to be counted; the video stream data includes: a plurality of frame images arranged in time; A target detection module is used to perform target detection on the frame image to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image; the equipment category includes: self-unloading equipment and excavation equipment; A target tracking module, used for tracking the construction machinery in the plurality of frame images arranged in time according to the equipment category and detection frame corresponding to the construction machinery in the frame image, so as to obtain the equipment identification corresponding to the construction machinery; The equipment matching module is used to determine the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: an excavating device and a matching target dump device; an action recognition module, configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, so as to obtain the action category corresponding to the construction machinery; The excavation volume determination module is used to determine the cumulative excavation volume corresponding to the construction area to be counted based on the rated excavation capacity corresponding to the excavation equipment, when the action category of the excavation equipment in the equipment matching pair undergoes a preset change.

[0009] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0010] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0011] The embodiment of the present application also discloses a computer program product, including a computer program / instruction, which implements the aforementioned method when executed by a processor.

[0012] The embodiments of the present application include the following advantages: In the technical solution of the embodiment of the present application, a real-time monitoring and analysis process of excavation volume during the construction period executed by a computer is provided, which process first obtains video stream data of the construction area to be counted, and the video stream data specifically includes: multiple frame images arranged in time; then, target detection is performed on the above frame images to obtain the equipment category (dump equipment and excavation equipment) and detection frame corresponding to the construction machinery; then, according to the equipment category and the detection frame, the construction machinery in the multiple frame images is tracked to obtain the equipment identification corresponding to the construction machinery; thereafter, an equipment matching pair is determined according to the equipment category, the equipment identification and the detection frame, and the above equipment matching pair specifically represents an excavation equipment and a matching target dump equipment; then, according to the equipment category, the equipment identification and the detection frame, the action of the construction machinery is identified to obtain the corresponding action category; further, when the action category of the excavation equipment in the equipment matching pair undergoes a preset change, the cumulative excavation volume corresponding to the construction area to be counted is determined according to the rated excavation capacity of the excavation equipment.

[0013] The above process of the embodiment of the present application performs real-time monitoring of the excavation volume during the construction period according to the objective law of the cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can combine the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair to accurately determine the actual occurrence of each effective excavation, thereby determining the cumulative excavation volume corresponding to the construction area to be counted by accumulating the rated excavation capacity.

[0014] Since the above-mentioned objective laws of the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period based on the intrinsic logical connection between the actual process of the excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters. Therefore, the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period at a lower cost and faster speed based on the analysis of the video stream data. In other words, the embodiments of the present application can save the manpower cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0015] Furthermore, since the above-mentioned objective laws of the embodiments of the present application do not involve subjective assumptions and complex manual measurement and estimation, the embodiments of the present application can effectively improve the accuracy of real-time monitoring of excavation volume during the construction period. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of an application environment of a method for real-time monitoring and analysis of excavation volume during construction based on action recognition according to an embodiment of the present application; Figure 2 This is a schematic diagram of the steps of a method for real-time monitoring and analysis of excavation volume during construction based on motion recognition according to an embodiment of the present application; Figure 3 is a structural diagram of an action recognition model according to an embodiment of the present application; Figure 4 It is a structural schematic diagram of a device for real-time monitoring and analyzing excavation volume during construction based on motion recognition according to an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] The embodiments of the present application can be applied to engineering industries such as hydropower and civil engineering, and can be used for open excavation operations in the engineering industry to monitor the corresponding excavation volume in real time.

[0019] Taking the pumped-storage power station project as an example, the construction project presents a situation where excavation operations are carried out simultaneously at multiple construction sites. At the same time, due to the continuous changes in the on-site construction situation, there is a need to carry out real-time monitoring and analysis of the excavation volume during the construction period for multiple construction sites.

[0020] The traditional method for calculating the excavation volume during the construction phase usually involves first obtaining a digital elevation model of the excavation area using drone oblique photography before and after excavation, or using 3D laser scanning technology to obtain spatial point cloud data of the excavation area; then, the volume of the excavation area is calculated based on the difference between the digital elevation model of the excavation area or the spatial point cloud data; then, the excavation volume is calculated based on the volume of the excavation area. The above-mentioned on-site measurement method not only requires a certain amount of manpower and data collection costs, but also has the disadvantages of long data processing time and poor real-time performance.

[0021] In view of the technical problems of traditional technology, such as labor consumption, data collection cost, long processing time and poor real-time performance, the embodiment of the present application provides a real-time monitoring and analysis method of excavation volume during construction period based on action recognition, which specifically includes the following steps: Acquire video stream data of the construction area to be counted; the video stream data includes: a plurality of frame images arranged in time; Performing target detection on the frame image to obtain equipment categories and detection frames corresponding to the construction machinery in the frame image; the equipment categories include: self-unloading equipment and excavating equipment; Tracking the construction machinery in the multiple frame images arranged in time according to the equipment category and the detection frame corresponding to the construction machinery in the frame image to obtain the equipment identification corresponding to the construction machinery; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating equipment in the equipment matching pair, it is considered that the excavating equipment in the equipment matching pair completes one excavation, and the cumulative excavation volume corresponding to the construction area to be counted is determined according to the rated excavation capacity corresponding to the excavating equipment.

[0022] An embodiment of the present application provides a real-time monitoring and analysis process of excavation volume during construction period executed by a computer, the process first acquires video stream data of the construction area to be counted, the video stream data specifically includes: a plurality of frame images arranged in time; then, target detection is performed on the frame images to obtain the equipment category (dump equipment and excavation equipment) and detection frame corresponding to the construction machinery; then, based on the equipment category and the detection frame, the construction machinery in the plurality of frame images is tracked to obtain the equipment identification corresponding to the construction machinery; thereafter, an equipment matching pair is determined based on the equipment category, the equipment identification and the detection frame, the equipment matching pair specifically represents an excavation equipment and a matching target dump equipment; then, based on the equipment category, the equipment identification and the detection frame, the action of the construction machinery is identified to obtain the corresponding action category; further, in the event that the action category of the excavation equipment in the equipment matching pair undergoes a preset change, the cumulative excavation volume corresponding to the construction area to be counted is determined based on the rated excavation capacity of the excavation equipment.

[0023] The above process of the embodiment of the present application performs real-time monitoring of the excavation volume during the construction period according to the objective law of the cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can combine the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair to accurately determine the actual occurrence of each effective excavation, thereby determining the cumulative excavation volume corresponding to the construction area to be counted by accumulating the rated excavation capacity.

[0024] Since the above-mentioned objective laws of the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period based on the intrinsic logical connection between the actual process of the excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters. Therefore, the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period at a lower cost and faster speed based on the analysis of the video stream data. In other words, the embodiments of the present application can save the manpower cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0025] Furthermore, since the above-mentioned objective laws of the embodiments of the present application do not involve subjective assumptions and complex manual measurement and estimation, the embodiments of the present application can effectively improve the accuracy of real-time monitoring of excavation volume during the construction period.

[0026] Reference Figure 1, showing a schematic diagram of an application environment of a method for real-time monitoring and analysis of excavation volume during construction period based on motion recognition according to an embodiment of the present invention, wherein the image acquisition end 101 and the service end 102 can perform data exchange based on a wireless network or a wired network.

[0027] In practical applications, the image acquisition terminal 101 may be equipped with an image acquisition device having an image acquisition function, such as an image sensor, etc. The image acquisition device may acquire construction videos and send the construction videos to the server 102 according to a preset time period.

[0028] After receiving the construction video sent by the image acquisition terminal 101, the server 102 can extract video stream data from the construction video and process the video stream data using the method of the embodiment of the present application to determine the cumulative excavation volume corresponding to the video stream data of the construction area to be counted.

[0029] It should be noted that, as the video stream data is updated, the embodiment of the present application can continuously determine the cumulative excavation volume corresponding to the video stream data. Therefore, the embodiment of the present application can realize real-time monitoring of the excavation volume during the construction period at a relatively low cost and relatively fast speed based on the analysis of the video stream data.

[0030] Method Example 1 refer to Figure 2 , shows a schematic flow chart of the steps of a method for real-time monitoring and analysis of excavation volume during construction based on action recognition according to an embodiment of the present application, the method specifically comprises the following steps: Step 201, obtaining video stream data of the construction area to be counted; the video stream data specifically includes: a plurality of frame images arranged in time; Step 202: performing target detection on the frame image to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image; the equipment category specifically includes: self-unloading equipment and excavating equipment; Step 203: Track the construction machinery in the plurality of frame images arranged in time according to the equipment category and the detection frame corresponding to the construction machinery in the frame image to obtain the equipment identification corresponding to the construction machinery; Step 204: determining a device matching pair in the plurality of frame images arranged in time according to the device category, device identification and detection frame corresponding to the construction machinery in the frame images; the device matching pair includes: an excavating device and a matching target dump device; Step 205: performing action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images, so as to obtain the action category corresponding to the construction machinery; Step 206: When the action category of the excavating equipment in the above equipment matching pair undergoes a preset change, it is considered that the excavating equipment in the above equipment matching pair completes one excavation, and the cumulative excavation volume corresponding to the construction area to be counted is determined according to the rated excavation capacity corresponding to the above excavating equipment.

[0031] Figure 2 The method shown can be used for action recognition in device matching pairs based on frame images, and real-time monitoring of the excavation volume corresponding to the excavation operation. At least one of the steps included can be executed by the server. It can be understood that the embodiment of the present application is for Figure 2 The specific execution subject of the method shown is not limited.

[0032] In step 201, the process of obtaining the video stream data of the construction area to be counted specifically includes: after receiving the construction video sent by the image acquisition end, the server extracts the video stream data from the construction video. The server can receive the video stream data from the image acquisition end according to a preset time period.

[0033] The multiple frame images in the video stream data are arranged in chronological order, which means that as time goes on, the pictures corresponding to each time point will form separate frame images and be sorted in the order of occurrence. In the specific implementation process, it is an effective approach to distinguish different frame images by using frame numbers. For example, the first frame image generated at the starting moment can be numbered 1, and the frame image corresponding to the next moment is numbered 2, and so on. Through such frame numbers, not only can the frame images at different moments be clearly distinguished, but also in subsequent analysis and processing links such as target detection and construction machinery tracking, the frame images at specific time points can be accurately and quickly located and called according to the frame number, so as to better calculate the corresponding excavation volume and other related indicators based on the video stream data.

[0034] In the hydropower engineering scenario, multiple video sources can be set, and different video sources can correspond to different video stream data. Among them, the video source can continuously provide continuous video stream data. The embodiment of the present application can process the continuous video stream data provided by any video source to obtain the excavation amount corresponding to the video source. The construction area to be counted can be the construction area represented by a certain video source.

[0035] In step 202, target detection technology may be used to perform target detection on the video stream data to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image.

[0036] Target detection technology is an important branch in the field of computer vision, which aims to identify target information corresponding to a target in an image. Specifically, in the embodiment of the present application, the frame image may include targets such as excavation equipment and dump equipment. Examples of excavation equipment may include excavators, etc. Examples of dump equipment may include dump trucks, etc. In actual applications, the dump truck drives into the excavation site and stops within the working radius of the excavator so that the excavator can easily load the material into the cargo box of the dump truck.

[0037] The target information may specifically include: the equipment category and detection frame corresponding to the construction machinery. The equipment category specifically includes: excavation equipment category and dump equipment. The detection frame is used to indicate the position and range of the target in the frame image. The detection frame is usually a rectangular frame, whose four sides are aligned with the outermost edges of the target, thereby surrounding the entire target. The information of the detection frame may include: the coordinates of the upper left corner of the rectangular frame, the width and the height; or, the information of the detection frame may include: the coordinates of the center point of the rectangular frame, the width and the height.

[0038] In the embodiment of the present application, the target detection model can be used to perform target detection on the frame image. In a specific implementation, the frame image can be input into the target detection model to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image output by the target detection model.

[0039] The target detection model is a deep learning model in the field of computer vision. It can identify targets in frame images and determine target information such as the target detection box and equipment category. The target detection model is usually implemented based on a convolutional neural network. The output result of the target detection model can be in the format of: {(machine class, xmin, xmax, ymin,ymax)}. Among them, "machine class" represents the category of the construction machinery in the frame image, such as "dump equipment" or "excavation equipment", which is used to distinguish different categories of construction machinery; "xmin" and "xmax" are the leftmost and rightmost coordinate values ​​of the detection box of the construction machinery in the horizontal direction (x-axis) of the frame image; "ymin" and "ymax" are the top and bottom coordinate values ​​of the detection box of the construction machinery in the vertical direction (y-axis) of the frame image. These four coordinate values ​​together determine a rectangular detection box.

[0040] The embodiments of the present application do not limit the specific target detection model. For example, examples of target detection models may include: YOLO (You Only Look Once) series, SSD (Single Shot PultiBox Detector), etc.

[0041] In one example, the target detection model specifically includes: a feature extraction unit, a feature fusion unit and a detection unit.

[0042] The feature extraction unit is used to extract image features from the frame image.

[0043] The feature fusion unit is used to effectively fuse image features of different levels and types. The obtained fused image features can make the feature information more comprehensive and accurate, so as to enhance the ability to express the characteristics of construction machinery.

[0044] The detection unit is used to determine the category and position of the construction machinery in the frame image according to the fused image features.

[0045] In step 203, target tracking technology may be used to track the construction machinery in the multiple frame images arranged in time. The format of the tracking result output in step 203 may be: {(machine class, machineID,xmin, xmax, ymin, ymax)}, where machineID represents the machine ID corresponding to the construction machinery.

[0046] Target tracking technology is a technology that uses a series of algorithms and models to determine the position of a specific target (the construction machinery in the embodiment of the present invention) in different frame images in a frame image sequence, and continuously estimates and updates the target state (including position, speed, direction of movement, etc.). Its core is to achieve accurate recognition and continuous tracking of the target in a complex image environment and constantly changing target state.

[0047] In a specific implementation, the process of tracking the construction machinery in the multiple frame images arranged in time specifically includes: Step A1: for the detection box in the first frame image, create a track ID (identity) in the track set; Step A2, extracting features from the area image within the detection frame in the k-th frame image to obtain a first appearance feature vector of the detection frame in the k-th frame image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature; k is a positive integer; Step A3, determining the predicted trajectory of the trajectory ID in the trajectory set in the kth frame image according to the historical trajectory information of the trajectory ID in the trajectory set; Step A4, determining the state of the track ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state; Step A5: Assign a machine ID to the construction machine corresponding to the trajectory ID of the confirmed status.

[0048] In steps A1 to A5 of the embodiment of the present application, feature matching and trajectory prediction are used to implement tracking of the construction machinery in multiple frame images.

[0049] In the hydropower engineering scenario, construction machinery of the same category is similar in appearance, shape or color, which can easily lead to errors in the matching process between the trajectory ID and the detection frame by the trajectory tracking means. In the process of calculating the first appearance feature vector of the image of the area within the detection frame, the embodiment of the present application controls the weight of the texture feature to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, and the color feature has a lower degree of distinction when the model and appearance color are similar, on this basis, the embodiment of the present application determines the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID. This can more accurately distinguish different construction machinery, reduce the probability of misjudgment due to similar appearance, and improve the distinction of construction machinery.

[0050] In step A1, a tracking track is initialized by creating a track ID for the detection box in the initial first frame image.

[0051] Step A2 extracts the appearance feature vector of the detection frame area in each frame image, with emphasis on texture features, because in construction scenes, the texture features of construction machinery are more stable and distinguishable than color features.

[0052] For example, for two excavators of the same model, the colors and appearances of the two are similar, but the two may be different in terms of wear and scratches, cleanliness, corrosion, warning signs and safety signs, etc. The texture features of the embodiments of the present application can precisely reflect the differences between the two excavators of the same model in terms of wear and scratches, cleanliness, corrosion, warning signs and safety signs, etc.

[0053] Step A2 is a process of extracting features from the area image within the detection frame in the k-th frame image, specifically comprising: Step A21, using a convolution module to extract a first eigenvector of the regional image; Step A22, extracting a second eigenvector of the regional image using a local binary pattern method; Step A23: merge the first feature vector and the second feature vector to obtain an appearance feature vector.

[0054] In step A21, the convolution module can be constructed based on models such as CNN (Convolutional Neural Network) and VGG (Visual Geometry Group).

[0055] The first feature vector extracted by the convolution module is high-dimensional, usually containing hundreds, thousands or even more dimensions. Examples of the dimensions of the first feature vector may include: spatial features, hierarchical features, scale features, directional features, color features, depth features, context features, etc. Among them, the spatial features represent the spatial structural information of the construction machinery, such as edges, corners, textures, etc. Therefore, the spatial features may include: texture features.

[0056] LBP (Local Binary Patterns) is a simple and effective algorithm for describing image texture features. The LBP algorithm generates a binary code by comparing the grayscale values ​​of each pixel in the image with its neighboring pixels, and then converts these binary codes into decimal numbers to form a texture descriptor. Since LBP can effectively describe the local texture features of the image and capture the subtle changes and patterns in the image; therefore, the second eigenvector can contain: texture features with strong distinguishing ability.

[0057] The process of fusing the first feature vector and the second feature vector in the embodiment of the present application specifically includes: weighting the features of the same dimension in the first feature vector and the second feature vector to obtain a weighted processing result; the features of the same dimension include: texture features; and increasing the corresponding numerical value for the weighted processing result corresponding to the texture feature.

[0058] Assuming that the weight of the first eigenvector is w1, the value of the texture feature in the first eigenvector is f1, the weight of the second eigenvector is w2, and the value of the texture feature in the second eigenvector is f2, the weighted processing result corresponding to the texture feature can be: w = w1*f1+w2*f2. In order to highlight the importance of the second eigenvector, w2 can be greater than w1. It can be understood that those skilled in the art can determine the specific values ​​of w1 and w2 according to actual application requirements.

[0059] The embodiment of the present application increases the corresponding value for the weighted processing result corresponding to the texture feature, which can achieve the technical effect that the weight of the texture feature is greater than the weight of the color feature. The increased value can be determined by those skilled in the art according to actual application requirements.

[0060] In step A3, based on the historical trajectory information of the trajectory ID in the trajectory set, the predicted trajectory of the trajectory ID in the trajectory set in the kth frame of the video stream image is determined. k can be a positive integer greater than 1.

[0061] The historical track information may include: the position information of the construction machine or the detection frame corresponding to the track ID in the past frame images, and the position information generally includes coordinates, timestamp, speed, direction and other information.

[0062] The embodiment of the present application can use the Kalman filter method to determine the predicted trajectory of the trajectory ID in the trajectory set in the kth frame of the video stream image. Kalman filtering is an efficient autoregressive filter that can estimate the state of a linear dynamic system from a series of incomplete and noisy measurements.

[0063] In step A4, the state of the track ID may include: unconfirmed state or confirmed state. The confirmed state may refer to the track ID having a corresponding target detection frame after matching calculation. The unconfirmed state may refer to the track ID being generated due to the initialization of the first frame image or the appearance of a new construction machinery target object, and not having a corresponding target detection frame after matching calculation.

[0064] Step A4 is a process of determining the state of the track ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID, specifically including: Step A41, performing a first match between the trajectory ID in the trajectory set whose status is confirmed and the detection frame in the detection frame set to obtain a corresponding first matching result; wherein, in the first matching process, determining the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID, and determining the first matching result according to the cosine distance; the first matching result includes: the trajectory ID for which the first matching succeeds, the trajectory ID for which the first matching fails, and the detection frame for which the first matching fails; Among them, the trajectory set is a data structure used to record the trajectory of the construction machinery in the frame image. It assigns a unique trajectory ID to each tracked construction machinery, through which the historical trajectory information such as the position and appearance features of the construction machinery in the past frames can be indexed. For example, when the Kalman filter is used to predict the position of a construction machinery in the kth frame image, the historical position information (such as coordinates, speed, direction, etc.) in the trajectory set is used as important input data, thereby realizing the continuous estimation and update of the target motion state.

[0065] The detection box set stores the location information of the construction machinery detected in each frame image (in the form of detection boxes). These detection boxes determine the specific range of the target in the frame image.

[0066] Step A42, performing a second match on the trajectory ID that failed the first match or the trajectory ID in the unconfirmed state and the detection frame that failed the first match, to obtain a corresponding second matching result; the second matching is an intersection-over-union matching; the second matching result includes: the trajectory ID that succeeded the second match, the trajectory ID that failed the second match, and the detection frame that failed the second match; Step A43: for the detection frame that fails the second matching, determine it as a new construction machinery target object, and create a track ID in a non-confirmed state in the track set; Step A44: if the second matching result of the trajectory ID in the unconfirmed state is a successful match, update the state of the trajectory ID to the confirmed state; and if the number of second matching failures of the trajectory ID in the confirmed state exceeds the first threshold, delete the trajectory ID.

[0067] In step A41, the embodiment of the present application can determine the state of the newly created trajectory ID according to the actual application requirements. For example, when creating a trajectory ID in the trajectory set for the detection frame in the first frame of the video stream image, the state of the trajectory ID corresponding to the detection frame in the first frame of the video stream image can be set to a non-confirmed state. For another example, for the detection frame that fails the second match, it is determined to be a new construction machinery target object, and a trajectory ID with a non-confirmed state is created.

[0068] It should be noted that the embodiment of the present application provides a switching method from the unconfirmed state to the confirmed state. Specifically, for the track ID in the unconfirmed state, the embodiment of the present application will verify the authenticity of the construction machinery corresponding to the track ID in the unconfirmed state based on the subsequent second matching result. The above verification means can identify and eliminate errors in the target detection results, thereby reducing the errors in the tracking results caused by errors in the target detection results.

[0069] In the first matching process, the embodiment of the present application determines the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the trajectory ID, and determines the first matching result according to the cosine distance.

[0070] The second appearance feature vector is an appearance feature vector associated with the track ID, and represents the appearance features of the detection box corresponding to the track ID in the historical video stream image.

[0071] In practical applications, the second appearance feature vector corresponding to the trajectory ID may be determined according to the first appearance feature vector of the detection frame corresponding to the trajectory ID in the historical video stream image.

[0072] For example, if the track ID1 is created based on the detection box in the first frame of the video stream image, then the second appearance feature vector B2 of the track ID1 in the second frame of the video stream image can be determined based on the first appearance vector A1 of the detection box A in the first frame of the video stream image. The second appearance feature vector B2 can be equal to the first appearance vector A1.

[0073] For example, if the track ID2 is created based on the detection box B in the first frame of the video stream image, then the track ID2 successfully matches the detection box C in the second frame of the video stream image, the first appearance vector C2 of the detection box C in the second frame of the video stream image, then the second appearance feature vector B3 of the track ID2 in the third frame of the video stream image can be jointly determined based on the first appearance vector B1 of the detection box B in the first frame of the video stream image of the track ID2, and the first appearance vector C2 of the detection box C in the second frame of the video stream image. Of course, the embodiment of the present application does not limit the specific process of jointly determining the second appearance feature vector B3 of the track ID2 in the third frame of the video stream image.

[0074] Since the texture features contained in the first appearance feature vector or the second appearance feature vector of the embodiment of the present application can better reflect the surface structure and details of the construction machinery, on this basis, the embodiment of the present application determines the first matching result according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID, which can effectively reduce the errors that occur in the first matching process between the track ID and the detection frame by the track tracking means. In other words, the first matching of the embodiment of the present application can correctly associate the detection frame of each construction machinery with its corresponding track ID based on accurate appearance feature matching, which can effectively reduce matching errors caused by similar appearances, thereby improving the accuracy of the tracking results.

[0075] In one implementation of the present application, step A41 is a process of first matching the track ID in the track set with the detection frame in the detection frame set, specifically including: Step A411, determining a cosine distance between a first appearance feature vector of the detection frame and a second appearance feature vector corresponding to the track ID whose state is the confirmation state; Step A412, determining the Mahalanobis distance between the detection frame and the track ID whose state is the confirmation state; Step A413, constructing a cost matrix; the elements in the cost matrix represent the matching cost between a detection box and a trajectory; the values ​​of the elements are determined according to the cosine distance and the Mahalanobis distance; Step A414: solving the cost matrix with the goal of minimizing the total matching cost to obtain a first matching result.

[0076] In step A411, cosine distance is a commonly used similarity measurement method, which measures the similarity between two vectors by calculating the cosine value of the angle between them. The smaller the cosine distance, the more similar the two vectors are, indicating that the detection box may correspond to the same construction machine as the track ID. The cosine distance reflects the similarity between the detection box and the track ID in terms of appearance features.

[0077] In step A412, the Mahalanobis distance is a statistic used to measure the similarity between two random vectors. The Mahalanobis distance takes into account the covariance structure in the data, so it can better handle the correlation and scale differences in the data. Generally speaking, the smaller the Mahalanobis distance, the higher the similarity between the two vectors. The Mahalanobis distance characterizes the similarity between the detection box and the trajectory ID in terms of position features.

[0078] In the process of determining the Mahalanobis distance between the detection box and the trajectory ID in the confirmed state, a first position vector corresponding to the detection box can be determined, a second position vector corresponding to the trajectory ID can be determined, and the Mahalanobis distance between the first position vector and the second position vector can be calculated.

[0079] The second position vector may be determined based on the aforementioned position information. The position information corresponding to the trajectory ID may be determined based on a Kalman filter method.

[0080] In step A413, the cost matrix is ​​a two-dimensional matrix, each element of which represents the matching cost between a detection box and a track ID. This matching cost is usually determined based on a combination of cosine distance and Mahalanobis distance. For example, the cosine distance and the Mahalanobis distance can be weighted summed, or other functions can be used to combine the cosine distance and the Mahalanobis distance to form the final matching cost.

[0081] It should be noted that before constructing the cost matrix, the cosine distance and the Mahalanobis distance may be filtered to filter out the trajectory IDs with larger cosine distance or Mahalanobis distance.

[0082] In step A414, an optimization method such as the Hungarian algorithm and the linear assignment problem solver can be used to find a matching solution that minimizes the total matching cost. The Hungarian algorithm is a classic combinatorial optimization algorithm that is suitable for solving the minimum weight matching problem of a bipartite graph. By solving the cost matrix, it can be determined which detection boxes should be matched with which trajectory IDs.

[0083] The first matching result is obtained by solving the cost matrix. Each element in the cost matrix represents the matching cost between a detection box and a track ID. This cost is obtained by combining the appearance feature (cosine distance) and the position feature (Mahalanobis distance). The purpose of solving the cost matrix is ​​to find a matching solution that minimizes the total matching cost.

[0084] For example, in the process of solving the cost matrix, the Hungarian algorithm will look for the optimal matching combination. If the matching cost between the detection box A and the trajectory ID1 is the smallest among all possible combinations, then the detection box A and the trajectory ID1 will be matched together in the solution result, and this matching relationship constitutes part of the first matching result. Therefore, the successfully matched detection box and trajectory ID in the first matching result are determined by the minimization of the cost matrix.

[0085] The first matching result of the embodiment of the present application specifically includes: a trajectory ID where the first matching is successful, a trajectory ID where the first matching fails, and a detection box where the first matching fails.

[0086] For the first successfully matched trajectory ID, the Kalman filter method can be used to update the fusion of historical trajectory information. Specifically, according to the position information of the successfully matched detection frame, the position information corresponding to the first successfully matched trajectory ID is updated, and according to the first appearance feature vector of the successfully matched detection frame, the second appearance feature vector of the first successfully matched trajectory ID is updated.

[0087] In the embodiment of the present application, step A42 may be performed for the first track ID that failed to match and the first detection frame that failed to match.

[0088] Step A42 can perform a second match on the track ID that failed the first match or the track ID that is in a non-confirmed state with the detection frame that failed the first match to obtain the corresponding second matching result. The second match, as a supplement to the first match, can handle those detection frames and track IDs that failed to match successfully in the appearance feature match, and is suitable for handling situations where the appearance feature match fails due to occlusion, temporary disappearance, etc. Therefore, the second match of the embodiment of the present application can further improve the accuracy of the tracking result.

[0089] The second match can be an intersection-and-union match. The intersection-and-union match process specifically includes: calculating the area intersection-and-union ratio between the prediction box and the detection box corresponding to the trajectory ID. If the area intersection-and-union ratio is greater than a set threshold, the match is considered successful, otherwise the match fails. The prediction box corresponding to the trajectory ID can also be a rectangular box, which can be determined by a Kalman filter.

[0090] The second matching result specifically includes: a trajectory ID where the second matching is successful, a trajectory ID where the second matching is unsuccessful, and a detection frame where the second matching is unsuccessful.

[0091] For the second successfully matched trajectory ID, the Kalman filter method can be used to update the fusion of historical trajectory information. Specifically, according to the position information of the successfully matched detection frame, the position information corresponding to the second successfully matched trajectory ID is updated, and according to the first appearance feature vector of the successfully matched detection frame, the second appearance feature vector of the second successfully matched trajectory ID is updated.

[0092] The embodiment of the present application can execute step A43 for the detection frame that fails the second matching. In step A43, the detection frame that fails the second matching is determined to be a new construction machinery target object, and a track ID with a non-confirmed state is created in the track set.

[0093] The reason for the existence of the detection frame that failed the second match may be: new construction machinery appeared in the construction area to be counted, or it may be: there was an error in the target detection result, and an object that was not a construction machinery was mistakenly identified as a construction machinery. The embodiment of the present application creates a track ID with a non-confirmed state for the detection frame that failed the second match, and verifies the authenticity of the construction machinery corresponding to the track ID in the non-confirmed state based on the subsequent second matching result. The above means can identify and eliminate errors in the target detection results, thereby reducing errors in the tracking results caused by errors in the target detection results.

[0094] When the second matching failure number of the trajectory ID in the confirmation state exceeds the first threshold, the trajectory ID is deleted. The first threshold can be determined by those skilled in the art according to actual application requirements, for example, the first threshold can be 60 or other values.

[0095] For the trajectory ID that failed the second match, it can be determined whether it is in a non-confirmed state. If so, the trajectory ID can be directly deleted; if not, it can be determined whether the loss age of the trajectory ID is greater than a second threshold value, which can be a value such as 60. If it is greater, the trajectory ID is directly deleted, and if it is less, the trajectory ID is retained. Therefore, the method of the embodiment of the present application can also include: deleting the trajectory ID that is in a non-confirmed state and whose second matching result is a second match failure in the trajectory set.

[0096] In step A5, a machine identifier is assigned to the construction machine corresponding to the confirmed track ID. The machine identifier uniquely identifies the construction machine in the construction area to be counted, and can be a string such as 0001, 0002, etc. A construction machine corresponds to the same machine identifier in different frame images.

[0097] In step 204, for earthwork open excavation, there is a functional complementary relationship between the excavation equipment and the dump equipment. The excavation equipment is responsible for excavating earth, and the dump equipment is responsible for transporting earth. The two work closely together in the construction process. Therefore, the embodiment of the present application can use the spatial proximity relationship between the excavation equipment and the dump equipment to determine the equipment matching pairs in the above multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the above frame images.

[0098] In a specific implementation, the process of determining the device matching pairs in the multiple frame images arranged in time according to the device category, device identification and detection frame corresponding to the construction machinery in the frame images specifically includes: Step B1: determining a set of self-unloading equipment for the excavating equipment in the frame image; the distance between the self-unloading equipment and the excavating equipment in the set of self-unloading equipment does not exceed a distance threshold; Step B2: taking the maximum distance intersection-to-joint ratio index value as a criterion, determining a target self-unloading device that matches the excavating device from the set of self-unloading devices; the excavating device and the target self-unloading device form a device matching pair.

[0099] In step B1, for each excavating device in the k-th frame image, in order to screen out the dump equipment that may match it from the numerous construction machines in the k-th frame image, it is necessary to construct a dump equipment set.

[0100] The embodiment of the present application can calculate the distance between an excavating device and multiple dumping devices, and record the dumping devices whose distance does not exceed the distance threshold into the dumping device set. The distance threshold can be determined by those skilled in the art according to actual application requirements, and the embodiment of the present application does not limit the specific distance threshold.

[0101] In step B2, the DIOU (Distance - Intersection over Union) metric not only considers the Intersection over Union itself, but also incorporates the distance factor between the excavator and the dumper.

[0102] Assuming that the frame image contains N detection frames of mining equipment and M detection frames of dump trucks, the calculation of the DIOU index is shown in formula (1).

[0103] (1) in, Indicates indivual( and is a positive integer) detection frame of the self-unloading equipment; Indicates indivual( and is a positive integer) to mine the detection box of the device. , Represents the detection box , The center point of Center point , The Euclidean distance between is the diagonal distance of the minimum enclosed area that contains the detection box of the mth dump equipment and the detection box of the nth excavator. is the DIOU index between the detection box of the m-th dump equipment and the detection box of the n-th mining equipment.

[0104] After calculating the distance intersection and union ratio index values ​​for all the dump unloading equipment in the dump unloading equipment set, compare the values ​​of these distance intersection and union ratio indexes, and find the dump unloading equipment with the largest distance intersection and union ratio index value as the target dump unloading equipment that best matches the excavation equipment.

[0105] After determining the device matching pair in the jth frame image, it can be determined whether the subsequent frame image reuses the device matching pair in the jth frame image according to the position change of the dump equipment in the device matching pair in the jth frame image.

[0106] Accordingly, the process of determining the device matching pairs in the multiple frame images arranged in time according to the device category, device identification and detection frame corresponding to the construction machinery in the frame image specifically includes: Determine the first device matching pair in the j-th frame image according to the device category, device identification and detection frame corresponding to the construction machinery in the j-th frame image; j and m can both be positive integers; If the position change of the dump device in the first device matching pair in the j+mth frame image is within the set range, the first device matching pair in the jth frame image is used for the j+mth frame image; If the position change of the dump equipment in the first equipment matching pair in the j+mth frame image is outside the set range, the second equipment matching pair in the j+mth frame image is determined based on the equipment category, equipment identification and detection box corresponding to the construction machinery in the j+mth frame image.

[0107] Because in normal earthwork excavation operations, the position of the dump equipment will not change much during the loading process. If the position change of the dump equipment in the first equipment matching pair in the j+mth frame image is within the set range, it means that the coordination relationship between the dump equipment and the excavation equipment has not changed, so the first equipment matching pair in the jth frame image can be reused. This is based on the working characteristics of the dump equipment in the construction scene. Its main task is to receive the materials loaded by the excavation equipment at a relatively fixed position, and its position is relatively stable.

[0108] If the position change of the self-unloading equipment in the j+mth frame image exceeds the set range, it means that the state of the self-unloading equipment has changed significantly, and it may have left the loading position or moved to other places for other operations. At this time, the previous equipment matching pair may no longer be applicable, so it is necessary to re-determine the second equipment matching pair based on the equipment category, equipment identification and detection frame of the construction machinery in the j+mth frame image to accurately reflect the actual coordination relationship between the construction machinery in the current frame image.

[0109] The above-mentioned calculation method of multiplexed device matching pairs can save computing resources and improve the efficiency of the entire construction machinery motion recognition and device matching process, especially when processing a large number of frame images, the effect is more obvious.

[0110] The position change of the self-unloading equipment in the first device matching pair in the j+mth frame image may specifically refer to the position change of the self-unloading equipment in the first device matching pair in the j+mth frame image relative to the position change of the self-unloading equipment in the first device matching pair in the jth frame image, that is, the position change of the self-unloading equipment in the same first device matching pair in the j+mth frame image and the jth frame image.

[0111] In practical applications, the position change can be judged whether it is within the set range based on the detection frame of a dump truck in different frame images. Specifically, the center coordinates of the detection frame of the dump truck in the jth and j+mth frame images are compared to determine the offset of the center coordinates of the detection frame. If the center coordinate offset is within the allowable coordinate difference range, the position change can be considered to be within the set range. Conversely, if the center coordinate offset is not within the allowable coordinate difference range, the position change can be considered to be not within the set range.

[0112] In a specific implementation, the format of the output result of step 204 may be: {(matchID, excavatorID, truckID)}. Wherein, excavatorID and truckID represent the excavator ID and dump device ID, respectively, which have a matching relationship. It should be noted that the excavator ID and dump device ID have corresponding mechanical identifiers, respectively. In other words, the mechanical identifier of the excavator in the equipment matching pair may be used as the excavator ID, and the mechanical identifier of the dump device in the equipment matching pair may be used as the dump device ID.

[0113] In step 205, motion recognition technology can be used to analyze the motion categories of the multiple frame images arranged in time based on the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images to obtain the motion category corresponding to the construction machinery.

[0114] Among them, the equipment category can pre-define the possible range of motion of the construction machinery according to the mechanical function. The above range of motion can include: the common excavation and rotation actions of excavation equipment, the general loading and unloading actions of dump equipment, etc. The equipment identification can accurately lock each construction machinery, improve the consistency and accuracy of motion recognition, and will not confuse the movement trajectories of different construction machinery. The detection frame can accurately extract the position, shape, posture and other changing features of the construction machinery in each frame image.

[0115] In a specific implementation, step 205 is a process of performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images, which specifically includes: Step C1, performing a first sampling on the multiple frame images arranged in time according to a first sampling rate, and using P first convolution modules to perform a first convolution processing on the detection frame area in the first sampling result, and the Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1; Step C2: performing a second sampling on the multiple frame images arranged in time according to a second sampling rate, and performing a second convolution processing on the detection frame area in the second sampling result using P second convolution modules, and the Pth second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel; Step C3, performing average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature; Step C4: Determine the action category corresponding to the construction machinery according to the average pooling feature.

[0116] Reference Figure 3 , shows a schematic diagram of the structure of an action recognition model of an embodiment of the present application, the input of the action recognition model specifically includes: a plurality of frame images arranged in time, and the result of target tracking.

[0117] The action recognition model is used to perform action recognition on the construction machinery in the multiple frame images arranged in time, and the output result can be in the format of: {(machine class, machineID, machine actionID,xmin, xmax, ymin, ymax)}. Among them, machine actionID represents the action category corresponding to a specific detection box (xmin, xmax, ymin, ymax), a specific machine class (equipment category) and a specific machineID (equipment identifier) ​​in a frame image.

[0118] Figure 3 In the example, the action recognition model specifically includes: a first processing branch 301, a second processing branch 302, an average pooling module 303 and a fully connected module 304.

[0119] The first processing branch 301 corresponds to step C1, and is used to perform a first sampling on the multiple frame images arranged in time according to a first sampling rate, and use P first convolution modules to perform a first convolution processing on the detection frame area in the first sampling result, and the Pth first convolution module outputs a first convolution feature. P is a positive integer greater than 1, Figure 3 In the embodiment, the value of P is 3. It can be understood that the embodiment of the present application does not limit the specific value of P.

[0120] The second processing branch 302 corresponds to step C2, and is used to perform a second sampling on the multiple frame images arranged in time according to a second sampling rate, and use P second convolution modules to perform a second convolution processing on the detection frame area in the second sampling result, and the Pth second convolution module outputs a second convolution feature.

[0121] The second sampling rate is greater than the first sampling rate, so that it can pay more attention to information with obvious differences in the time dimension. For example, the sampling rate of the second sampling module can be set to 8 times the sampling rate of the first sampling module, that is, under the same video stream data input conditions, the sampling frame number of the second sampling module is 8 times that of the first sampling module.

[0122] From the perspective of capturing action details, the high sampling rate of the second sampling module can obtain more frame images, which is critical for capturing rapid changes and subtle movements in construction machinery. For example, the instantaneous action of the excavation teeth cutting into the material when the excavator is digging or the brief action of the compartment door opening when the self-unloading equipment is unloading, these details that are easily overlooked at a lower sampling rate can be well captured through a high sampling rate. These detailed action features are important for accurately identifying action categories, which can improve the accuracy of action recognition and avoid misjudging similar actions.

[0123] From the perspective of utilizing information in the time dimension, a higher sampling rate can better focus on information with obvious differences in the time dimension. The movements of construction machinery are fast or slow in time series. A high sampling rate can obtain enough frames in a short period of time to analyze the changing trend of the movement. This helps to obtain more accurate movement features in the stage where the movement changes rapidly (such as fast unloading of self-unloading equipment, fast rotation of excavation equipment, etc.), so as to more accurately judge the movement category of construction machinery as a whole, especially for those with short movement cycles and large movement changes, which can effectively improve the accuracy and completeness of recognition.

[0124] The first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel. In a convolutional neural network, the number of output channels of a convolution kernel can be simply understood as the number of feature maps generated after the convolution operation.

[0125] The first convolution kernel with a large number of channels can capture the more macroscopic and overall features of the construction machinery movements as comprehensively as possible at a relatively low sampling rate. Because the frame interval is relatively large at a low sampling rate, it is necessary to comprehensively consider the movement from multiple dimensions. More channels can map the movement from different perspectives, such as the overall movement direction of the excavator arm of the excavator, the approximate angle change range, the change in relative position in the picture, etc. These are integrated to form coarse-grained features, which facilitates a rough judgment framework for the movement.

[0126] The second convolution kernel with a small number of channels focuses on extracting key details of construction machinery movements when processing a large number of dense frame images at a high sampling rate, such as the slight position change when the excavation teeth of the excavator cut into the material, and the precise change of a local angle when the dump truck unloads the material. It extracts fine-grained features in a concise and targeted manner, which can avoid interference from redundant information and meet the needs of quickly capturing details at high sampling rates.

[0127] The average pooling module 303 is used to perform average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature.

[0128] Average pooling is a downsampling operation. The first convolution feature and the second convolution feature may contain information from many different locations. The average pooling module calculates the average value of these features in a certain area (such as a small pixel area). This can reduce the amount of data while retaining the main feature information. For example, the convolution features corresponding to multiple pixels in a local area are averaged to obtain a more representative value, avoiding overfitting, and reducing the dimension of the data, making subsequent processing more efficient.

[0129] The fully connected module 304 is used to determine the action category corresponding to the construction machine according to the average pooling feature.

[0130] Each neuron in the fully connected module is connected to all neurons in the previous layer (average pooling layer). These connections have corresponding weights. By multiplying the average pooling features with these weights and summing them, and then passing through the activation function, the action category corresponding to the construction machinery can be determined based on the learned pattern. For example, if the value output by the average pooling feature after these operations meets the numerical range corresponding to the pattern of the excavation action, it is determined to be an excavation action.

[0131] In an optional embodiment of the present application, the process of performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images may also include: fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the i+1-th first convolution module. i can be a positive integer.

[0132] The above fusion can better combine the detail features at a high sampling rate contained in the second convolution result and the macro features at a low sampling rate contained in the first convolution result, so that each stage of convolution processing can make full use of the two convolution results. In addition, the above fusion can make the features subsequently output by the first convolution module integrate key information at different sampling rates, thereby improving the accuracy of construction machinery action recognition.

[0133] In a specific implementation, the process of fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module may specifically include: Step D1, sampling the second convolution result according to the time step of the first convolution result to obtain a third sampling result; Step D2, performing matrix reconstruction and transposition on the third sampling result according to the matrix structure of the first convolution result, so that the matrix structure of the obtained transposition result matches the matrix structure of the first convolution; Step D3, performing time stride convolution on the transposed result and the first convolution result to obtain a stride convolution result; Step D4: Fuse the stride convolution result with the first convolution result.

[0134] In step D1, the strides of the first convolution result and the second convolution result in the time dimension (which can be understood as the interval between data points) are different due to different sampling rates. Step D1 samples the time stride of the second convolution result based on the time stride of the first convolution result, in order to make the two preliminarily aligned in the time dimension.

[0135] Although step D1 adjusts the time step, the matrix structure (such as dimension, shape, etc.) of the second convolution result may still not match the first convolution result. Step D2 reconstructs and transposes the third sampling result according to the matrix structure of the first convolution result. The reconstruction and transposition operation of step D2 is used to adjust the third sampling result to the same matrix form as the first convolution result, so that the two can match in matrix structure, providing a data basis with consistent structure for subsequent fusion operations.

[0136] After matching the matrix structures of the two results in step D2, step D3 performs time-stride convolution on the transposed result (i.e., the third sampling result after processing in step D2) and the first convolution result. The purpose of this operation is to further fuse the transposed result and the first convolution result in the time dimension and the space dimension (matrix structure). Time-stride convolution can extract the joint features of the transposed result and the first convolution result in time and space, and find a more complex correlation between them. For example, the correlation between the action details and the overall action trend of the mining equipment at different time and space positions is very important for accurately identifying the action category of construction machinery.

[0137] Step D4 fuses the stride convolution result with the first convolution result. The fusion result will include: feature information obtained from different sampling rates (the first sampling rate corresponding to the first convolution module and the second sampling rate corresponding to the second convolution module). These feature information include both macroscopic overall action features and microscopic detailed action features. Through such fusion, it is possible to provide more comprehensive and richer feature inputs for subsequent action recognition, thereby improving the accuracy of construction machinery action category judgment and avoiding misjudgment caused by relying only on features at a single sampling rate.

[0138] In step 206, each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can accurately determine the actual occurrence of each effective excavation by combining the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair, and thus determine the cumulative excavation volume corresponding to the construction area to be counted by accumulating the rated excavation capacity.

[0139] In the construction scenario, the sequence of the action categories of the excavation equipment is a complete work cycle. The first is excavation, where the excavation equipment controls the bucket to insert into the material pile and fill it with materials. Next is lifting and rotation, where the mechanical arm raises the bucket filled with materials and rotates the body so that the bucket is aligned with the position of the dump equipment. Then comes unloading, where the materials in the bucket are poured into the dump equipment. After that, the empty bucket returns, and the empty bucket is put back to its original position or the position ready for the next excavation. Finally, excavation begins again, and the cycle continues. This sequence fully demonstrates the process of each action category working in sequence when the excavation equipment cooperates with the dump equipment.

[0140] The preset changes of mining device action categories in device matching refer to that the device actions are performed in a predetermined specific order. The starting point of this specific order can be determined by those skilled in the art according to actual application requirements.

[0141] For example, if the starting point of a specific sequence is the "digging" action, then the action category of the digging equipment in the equipment matching pair undergoes a preset change, specifically including: the action category of the digging equipment in the equipment matching pair changes in the order of digging, lifting and rotating, unloading, empty bucket return, and digging.

[0142] For another example, if the starting point of a specific sequence is the "lifting and rotating" action, then the action categories of the excavating equipment in the equipment matching pair will change in the order of lifting and rotating, unloading, empty bucket return, excavation, and lifting and rotating.

[0143] For another example, if the starting point of a specific sequence is the "unloading" action, the action categories of the excavating equipment in the equipment matching pair will change in the order of unloading, empty bucket return, excavation, lifting and rotating, and unloading.

[0144] Alternatively, if the starting point of a specific sequence is the "empty bucket return" action, the action category of the excavating equipment in the equipment matching pair changes in the order of empty bucket return, excavation, lifting and rotation, unloading, and empty bucket return.

[0145] When the excavation equipment leaves the factory, there will be a parameter setting of the rated excavation capacity, which represents the maximum volume of material that the excavation equipment can theoretically excavate during each normal excavation. In each actual effective excavation process, the excavation volume of the excavation equipment is in a fixed corresponding relationship with the rated excavation capacity. Therefore, the embodiment of the present application can determine the cumulative excavation volume corresponding to the construction area to be counted by accumulating the rated excavation capacity.

[0146] In practical applications, the process of determining the rated excavation capacity includes: Step E1, using the mining equipment model classification model, determining the target model corresponding to the mining equipment whose action category undergoes a preset change; Step E2: According to the target model, search in the mapping relationship between the search model and the rated mining capacity to determine the rated mining capacity corresponding to the mining equipment whose action category has undergone a preset change.

[0147] The mining device model classification model of the embodiment of the present application can have the ability to classify mining device models. In other words, it can determine the target model corresponding to the mining device in the local image based on the local image corresponding to the mining device with a preset change in action category.

[0148] In a specific implementation, the embodiment of the present application can label the model category for the mining device image data set, and train the mining device model classification model based on the mining device image data set.

[0149] Here, a process for acquiring an excavation equipment image dataset is provided. Specifically, based on a video surveillance system, excavation equipment images covering different construction backgrounds, different lighting conditions, different working postures, etc. can be collected. In addition, it is also necessary to collect a combined image of an excavation equipment and a dump truck in cooperation with the excavation operation to reflect the actual situation of the two types of construction machinery blocking each other. Therefore, the excavation equipment image dataset may include: excavation equipment images and combined images.

[0150] The embodiment of the present application does not limit the specific structure of the mining equipment model classification model. For example, the mining equipment model classification model may include: a convolutional neural network structure such as VGG (Visual GeoPetry Group).

[0151] The embodiment of the present application can save the mapping relationship between the model and the rated excavation capacity in advance, so that the target model corresponding to the excavation equipment with the preset change in action category can be searched in the above mapping relationship to obtain the rated excavation capacity corresponding to the excavation equipment with the preset change in action category. The above mapping relationship can be determined based on information sources such as the technical parameter manual of the excavation equipment or the equipment nameplate.

[0152] When the action category of the excavating equipment in the equipment matching pair changes by a preset value, it is considered that the excavating equipment in the equipment matching pair completes one excavation, and the process of determining the cumulative excavation volume corresponding to the construction area to be counted according to the rated excavation capacity corresponding to the excavating equipment specifically includes: Step F1: Determine the number of effective excavation times according to the number of preset changes in the action category of the excavation equipment in the equipment matching. For example, when the action of the excavation equipment is completely cycled in the order of excavation, lifting rotation, unloading, empty bucket return, and excavation, it is counted as an effective excavation. The number of such complete cycles is counted by real-time monitoring of video frame images or construction processes.

[0153] Step F2: obtaining the rated excavation capacity corresponding to the excavation equipment whose action category undergoes a preset change.

[0154] The embodiment of the present application can first use the mining equipment model classification model to determine the target model of the mining equipment; then, according to the target model, search in the mapping relationship between the model and the rated mining capacity to obtain the rated mining capacity corresponding to the mining equipment with a preset change in action category.

[0155] Step F3: Determine the cumulative excavation volume corresponding to the construction area to be counted according to the effective excavation times and the rated excavation capacity.

[0156] Previously, the target tracking technology has been used to assign equipment identification to the excavation equipment in the frame image of the construction area to be counted. Steps F1 and F2 can determine the target equipment identification corresponding to the excavation equipment whose action category has undergone a preset change, as well as the effective excavation times and rated excavation capacity corresponding to the target equipment identification. On this basis, the cumulative excavation volume corresponding to the target equipment identification can be determined according to the effective excavation times and rated excavation capacity corresponding to the target equipment identification. In addition, the cumulative excavation volumes corresponding to all target equipment identifications can be accumulated to obtain the total cumulative excavation volume corresponding to the entire construction area to be counted.

[0157] In one example of the present application, the total cumulative excavation volume corresponding to the construction area to be counted can be set to 0 at the beginning of a natural day. Subsequently, the total cumulative excavation volume can be updated according to the update of the effective excavation times identified by any target device to obtain a continuously updated total cumulative excavation volume. Therefore, the real-time monitoring and analysis process of the excavation volume during the construction period can be a process of continuously updating the total cumulative excavation volume according to the update of the effective excavation times identified by any target device.

[0158] In practical applications, if there are multiple construction areas to be counted in a hydropower project, the total cumulative excavation volume corresponding to each construction area to be counted can be monitored separately according to natural days. After a natural day, the total cumulative excavation volumes corresponding to multiple construction areas to be counted can be merged. It can be understood that the embodiment of the present application does not limit the specific fusion method.

[0159] In summary, the method for real-time monitoring and analysis of excavation volume during construction based on motion recognition in an embodiment of the present application provides a process for real-time monitoring and analysis of excavation volume during construction executed by a computer, the process first acquiring video stream data of the construction area to be counted, the video stream data specifically including: a plurality of frame images arranged in time; then, performing target detection on the frame images to obtain the equipment category (dumping equipment and excavation equipment) and the detection frame corresponding to the construction machinery; then, based on the equipment category and the detection frame, the construction machinery in the plurality of frame images is tracked to obtain the equipment identification corresponding to the construction machinery; thereafter, an equipment matching pair is determined based on the equipment category, the equipment identification and the detection frame, the equipment matching pair specifically representing an excavation equipment and a matching target dumping equipment thereof; then, based on the equipment category, the equipment identification and the detection frame, the construction machinery is motion recognized to obtain the corresponding action category; further, in the event that the action category of the excavation equipment in the equipment matching pair undergoes a preset change, the cumulative excavation volume corresponding to the construction area to be counted is determined based on the rated excavation capacity of the excavation equipment.

[0160] The above process of the embodiment of the present application performs real-time monitoring of the excavation volume during the construction period according to the objective law of the cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can combine the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair to accurately determine the actual occurrence of each effective excavation, thereby determining the cumulative excavation volume corresponding to the construction area to be counted by accumulating the rated excavation capacity.

[0161] Since the above-mentioned objective laws of the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period based on the intrinsic logical connection between the actual process of the excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters. Therefore, the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period at a lower cost and faster speed based on the analysis of the video stream data. In other words, the embodiments of the present application can save the manpower cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0162] Furthermore, since the above-mentioned objective laws of the embodiments of the present application do not involve subjective assumptions and complex manual measurement and estimation, the embodiments of the present application can effectively improve the accuracy of real-time monitoring of excavation volume during the construction period.

[0163] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0164] Based on the above embodiment, this embodiment also provides a real-time monitoring and analysis device for excavation volume during construction based on motion recognition. Figure 4 The device may specifically include: a video stream acquisition module 401, a target detection module 402, a target tracking module 403, a device matching module 404, an action recognition module 405 and an excavation amount determination module 406.

[0165] The video stream acquisition module 401 is used to acquire video stream data of the construction area to be counted; the video stream data includes: a plurality of frame images arranged in time; The target detection module 402 is used to perform target detection on the frame image to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image; the equipment category includes: self-unloading equipment and excavation equipment; The target tracking module 403 is used to track the construction machinery in the multiple frame images arranged in time according to the equipment category and detection frame corresponding to the construction machinery in the frame image, so as to obtain the equipment identification corresponding to the construction machinery; The equipment matching module 404 is used to determine the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image; the equipment matching pairs include: an excavating device and a matching target dump device; The action recognition module 405 is used to perform action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, so as to obtain the action category corresponding to the construction machinery; The excavation volume determination module 406 is used to determine the cumulative excavation volume corresponding to the construction area to be counted based on the rated excavation capacity corresponding to the excavation equipment, when the action category of the excavation equipment in the equipment matching pair undergoes a preset change.

[0166] Optionally, the preset change in the action category of the excavating equipment in the equipment matching pair includes: the action category of the excavating equipment in the equipment matching pair changes in the order of excavating, lifting and rotating, unloading, empty bucket return, and excavating.

[0167] Optionally, the action recognition module includes: A first branch processing module, configured to perform a first sampling on the plurality of frame images arranged in time according to a first sampling rate, and perform a first convolution processing on the detection frame area in the first sampling result using P first convolution modules, and the Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1; a second branch processing module, configured to perform a second sampling on the plurality of frame images arranged in time according to a second sampling rate, and perform a second convolution processing on the detection frame area in the second sampling result using P second convolution modules, and the Pth second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module adopts a first convolution kernel, the second convolution module adopts a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel; An average pooling module, used to perform average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature; The action category determination module is used to determine the action category corresponding to the construction machinery according to the average pooling feature.

[0168] Optionally, the action recognition module further includes: The convolution result fusion module is used to fuse the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the i+1-th first convolution module.

[0169] Optionally, the convolution result fusion module includes: A time step sampling module, used for performing time step sampling on the second convolution result according to the time step of the first convolution result to obtain a third sampling result; A matrix reconstruction transposition module, used for performing matrix reconstruction transposition on the third sampling result according to the matrix structure of the first convolution result, so that the matrix structure of the obtained transposition result matches the matrix structure of the first convolution; A time stride convolution module, used to perform time stride convolution on the transposed result and the first convolution result to obtain a stride convolution result; The fusion module is used to fuse the stride convolution result with the first convolution result.

[0170] Optionally, the device matching module includes: A set determination module, used for determining a set of self-unloading equipment for the excavating equipment in the frame image; the distance between the self-unloading equipment and the excavating equipment in the set of self-unloading equipment does not exceed a distance threshold; The target dump equipment determination module is used to determine the target dump equipment matching the excavation equipment from the dump equipment set based on the maximum distance intersection ratio index value; the excavation equipment and the target dump equipment form an equipment matching pair.

[0171] Optionally, the device matching module includes: A first device matching pair determination module, used to determine a first device matching pair in the j-th frame image according to the device category, device identification and detection frame corresponding to the construction machinery in the j-th frame image; A multiplexing module, configured to use the first device matching pair in the jth frame image for the j+mth frame image if the position change of the self-unloading device in the first device matching pair in the j+mth frame image is within a set range; The second device matching pair determination module is used to determine the second device matching pair in the j+mth frame image according to the device category, device identification and detection box corresponding to the construction machinery in the j+mth frame image if the position change of the dump equipment in the first device matching pair in the j+mth frame image is outside the set range.

[0172] Optionally, the target tracking module includes: A creation module is used to create a track ID in the track set for the detection box in the first frame image; A feature extraction module is used to extract features from the regional image within the detection frame in the k-th frame image to obtain a first appearance feature vector of the detection frame in the k-th frame image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature; k is a positive integer; A trajectory prediction module, used to determine a predicted trajectory of a trajectory ID in the trajectory set in the kth frame image according to historical trajectory information of the trajectory ID in the trajectory set; A trajectory state determination module, used to determine the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID; the state of the trajectory ID includes: a confirmed state or a non-confirmed state; The allocation module is used to allocate a machine identification to the construction machine corresponding to the trajectory ID of the confirmed status.

[0173] Optionally, the feature extraction module includes: A first extraction module, used for extracting a first feature vector of the regional image by using a convolution module; A second extraction module is used to extract a second feature vector of the regional image by using a local binary pattern method; The vector fusion module is used to fuse the first feature vector and the second feature vector to obtain an appearance feature vector.

[0174] Optionally, the vector fusion module includes: A weighted processing module is used to perform weighted processing on the same-dimensional features in the first feature vector and the second feature vector to obtain a weighted processing result; the same-dimensional features include: texture features; The value increasing module is used to increase the corresponding value according to the weighted processing result corresponding to the texture feature.

[0175] The embodiment of the present application further provides a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0176] The present application embodiment provides one or more machine-readable media, on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices, servers (clusters), etc.

[0177] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method for real-time monitoring and analysis of excavation volume during construction based on motion recognition as described in any of the above embodiments.

[0178] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Figure 5 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.

[0179] For one embodiment, Figure 5 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the processor(s) 1302, a memory 1306 coupled to the control module 1304, an NVP (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0180] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0181] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0182] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0183] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0184] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include a double data rate type four synchronous dynamic random access memory.

[0185] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the input / output device(s) 1310 .

[0186] For example, non-volatile memory / storage 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0187] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.

[0188] The input / output device(s) 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0189] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip.

[0190] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0191] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0192] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0193] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0194] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0195] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0197] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0198] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0199] The above is a detailed introduction to a method and device for real-time monitoring and analysis of excavation volume during construction based on motion recognition, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for real-time monitoring and analysis of excavation volume during construction based on motion recognition, characterized in that: The method comprises: Acquire video stream data of the construction area to be counted; the video stream data includes: a plurality of frame images arranged in time; Performing target detection on the frame image to obtain equipment categories and detection frames corresponding to the construction machinery in the frame image; the equipment categories include: self-unloading equipment and excavating equipment; Tracking the construction machinery in the multiple frame images arranged in time according to the equipment category and the detection frame corresponding to the construction machinery in the frame image to obtain the equipment identification corresponding to the construction machinery; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating equipment in the equipment matching pair, it is considered that the excavating equipment in the equipment matching pair completes one excavation, and the cumulative excavation volume corresponding to the construction area to be counted is determined according to the rated excavation capacity corresponding to the excavating equipment.

2. The method according to claim 1, characterized in that The preset change in the action category of the excavating equipment in the equipment matching pair includes: the action category of the excavating equipment in the equipment matching pair changes in the order of excavating, lifting and rotating, unloading, empty bucket returning, and excavating.

3. The method according to claim 1, characterized in that The step of performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images comprises: According to the first sampling rate, the plurality of frame images arranged in time are first sampled, and the detection frame area in the first sampling result is first convolutionally processed by using P first convolution modules, and the Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1; According to a second sampling rate, a second sampling is performed on the multiple frame images arranged in time, and a second convolution processing is performed on the detection frame area in the second sampling result using P second convolution modules, and the Pth second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel; Performing average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature; The action category corresponding to the construction machinery is determined according to the average pooling feature.

4. The method according to claim 3, characterized in that The step of performing action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images further includes: The second convolution result output by the i-th second convolution module is fused with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the i+1-th first convolution module.

5. The method according to claim 4, characterized in that The fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module includes: According to the time step of the first convolution result, the second convolution result is sampled in time step to obtain a third sampling result; According to the matrix structure of the first convolution result, the third sampling result is subjected to matrix reconstruction and transposition, and the matrix structure of the obtained transposition result matches the matrix structure of the first convolution; Perform time-stride convolution on the transposed result and the first convolution result to obtain the stride convolution result; The stride convolution result is fused with the first convolution result.

6. The method according to claim 1, characterized in that The determining of the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image comprises: Determine a set of self-unloading equipment for the excavating equipment in the frame image; the distance between the self-unloading equipment and the excavating equipment in the set of self-unloading equipment does not exceed a distance threshold; Taking the maximum distance intersection-to-combination ratio index value as a criterion, a target self-unloading device matching the excavating device is determined from the self-unloading device set; the excavating device and the target self-unloading device form a device matching pair.

7. The method according to claim 1, characterized in that The determining of the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image comprises: Determine a first device matching pair in the j-th frame image according to the device category, device identification, and detection frame corresponding to the construction machinery in the j-th frame image; If the position change of the dump device in the first device matching pair in the j+mth frame image is within the set range, the first device matching pair in the jth frame image is used for the j+mth frame image; If the position change of the dump equipment in the first equipment matching pair in the j+mth frame image is outside the set range, the second equipment matching pair in the j+mth frame image is determined based on the equipment category, equipment identification and detection box corresponding to the construction machinery in the j+mth frame image.

8. The method according to claim 1, characterized in that The step of tracking the construction machinery in the plurality of frame images arranged in time according to the equipment category and the detection frame corresponding to the construction machinery in the frame images comprises: For the detection box in the first frame image, create a track ID in the track set; Performing feature extraction on the area image within the detection frame in the k-th frame image to obtain a first appearance feature vector of the detection frame in the k-th frame image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature; k is a positive integer; Determine the predicted trajectory of the trajectory ID in the trajectory set in the kth frame image according to the historical trajectory information of the trajectory ID in the trajectory set; Determine the state of the track ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state; A machine ID is assigned to the construction machine corresponding to the trajectory ID of the confirmed status.

9. The method according to claim 8, characterized in that The feature extraction of the area image within the detection frame in the k-th frame image includes: Using the convolution module, extract the first eigenvector of the regional image; Using the local binary pattern method, extract the second eigenvector of the regional image; The first feature vector is fused with the second feature vector to obtain an appearance feature vector.

10. The method according to claim 9, characterized in that The fusing the first feature vector and the second feature vector comprises: Performing weighted processing on the same-dimensional features in the first feature vector and the second feature vector to obtain a weighted processing result; the same-dimensional features include: texture features; For the weighted processing result corresponding to the texture feature, increase the corresponding value.

11. A real-time monitoring and analysis device for excavation volume during construction based on motion recognition, characterized in that: The device comprises: The video stream acquisition module is used to acquire the video stream data of the construction area to be counted; the video stream data includes: a plurality of frame images arranged in time; A target detection module, used for performing target detection on the frame image to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image; the equipment category includes: self-unloading equipment and excavation equipment; A target tracking module, used for tracking the construction machinery in the plurality of frame images arranged in time according to the equipment category and the detection frame corresponding to the construction machinery in the frame image, so as to obtain the equipment identification corresponding to the construction machinery; The equipment matching module is used to determine the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: an excavating device and a matching target dump device; an action recognition module, configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, so as to obtain the action category corresponding to the construction machinery; The excavation volume determination module is used to determine the cumulative excavation volume corresponding to the construction area to be counted based on the rated excavation capacity corresponding to the excavation equipment, when the action category of the excavation equipment in the equipment matching pair undergoes a preset change.

12. The device according to claim 11, characterized in that The preset change in the action category of the excavating equipment in the equipment matching pair includes: the action category of the excavating equipment in the equipment matching pair changes in the order of excavating, lifting and rotating, unloading, empty bucket returning, and excavating.

13. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 10.

14. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images