A whole-process measurement method for excavation volume during the construction period based on multi-source data fusion

Abstract

An embodiment of the present application provides a method for calculating the whole-process excavation volume during the construction period based on multi-source data fusion. The method specifically includes: determining a three-dimensional model of the already excavated part of the excavation area according to the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; determining the first excavation volume of the excavation area within a preset time period according to the three-dimensional model of the already excavated part of the excavation area; the first moment is the termination moment of the preset time period; determining the second excavation volume of the excavation area within the preset time period according to the video stream data of the excavation area within the preset time period; determining a correction coefficient according to the first excavation volume and the second excavation volume; determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block; correcting the real-time cumulative excavation volume of the excavation block according to the correction coefficient. The embodiment of the present application can meet the requirements of engineering construction for the whole-process real-time calculation of the excavation volume during the construction period.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of open cut excavation of earth and rock, and particularly to a method for calculating the whole process of excavation volume during the construction period based on multi-source data fusion. Background Art

[0002] Open cut excavation of earth and rock is one of the common construction operations. The calculation of its excavation volume is crucial for closely following the on-site construction progress, and can provide strong support for the planning of excavation construction plans and the timely allocation of construction machinery.

[0003] In a related technology, the excavation volume during the design period is calculated based on 3D design software. Specifically, first, according to the engineering design drawings, topographic and geomorphic parameters and design planning parameters are obtained; then, using the topographic and geomorphic parameters and design planning parameters, the excavation area is divided into unit bodies, and the volume of the unit bodies is calculated; then, the volumes of all unit bodies are accumulated to obtain the total excavation volume of the entire excavation area.

[0004] In another related technology, the excavation volume during the construction period at stages is calculated based on UAV oblique photography. Specifically, before excavation, UAV oblique photography is used to obtain digital elevation model data to record the original topographic conditions; after excavation, the same technology is used again to obtain the corresponding digital elevation model data; by comparing the data before and after excavation, the volume corresponding to the difference between the two is calculated, and this volume is the excavation volume, thus realizing the calculation of the excavation volume.

[0005] However, whether it is calculating the excavation volume during the design period based on 3D design software or calculating the excavation volume during the construction period at stages based on UAV oblique photography, both have the problem of poor real-time performance and are difficult to meet the requirements of real-time calculation of the whole process of excavation volume during engineering construction. Summary of the Invention

[0006] The embodiments of the present application provide a method for calculating the whole process of excavation volume during the construction period based on multi-source data fusion, which can make the calculation of the excavation volume follow the construction progress in real time while significantly improving the accuracy and meeting the requirements of real-time calculation of the whole process of excavation volume during engineering construction.

[0007] Correspondingly, the embodiments of the present application also provide a device for calculating the whole process of excavation volume during the construction period based on multi-source data fusion, an electronic device, and a machine-readable medium to ensure the implementation and application of the above method.

[0008] To solve the above problems, the embodiments of the present application disclose a method for calculating the whole process of excavation volume during the construction period based on multi-source data fusion, including:

[0009] Determine the three-dimensional model of the completed excavated part of the excavation area based on the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the excavation area includes: at least one excavation block;

[0010] Determine the first excavation volume of the excavation area within a preset time period based on the three-dimensional model of the completed excavated part of the excavation area; the first moment is the end moment of the preset time period;

[0011] Determine the second excavation volume of the excavation area within a preset time period based on the video stream data of the excavation area within the preset time period;

[0012] Determine a correction coefficient based on the first excavation volume and the second excavation volume;

[0013] Determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block;

[0014] Correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the measurement result of the excavation volume of the excavation block;

[0015] Wherein, the video stream data includes: a plurality of frame images arranged in time sequence;

[0016] The determining the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block includes:

[0017] Determine the equipment matching pairs in the plurality of frame images arranged in time sequence according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: one excavating equipment and one target dump truck equipment that matches it;

[0018] Perform action recognition on the construction machinery in the plurality of frame images arranged in time sequence according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images to obtain the action category corresponding to the construction machinery;

[0019] When the action category of the excavating equipment in the equipment matching pair undergoes a preset change, it is considered that the excavating equipment in the equipment matching pair has completed one excavation, and determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavating equipment.

[0020] The embodiment of the present application also discloses a device for measuring the whole process of excavation volume during the construction period based on multi-source data fusion, and the device includes:

[0021] An excavated model determination module, configured to determine the three-dimensional model of the completed excavated part of the excavation area based on the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the excavation area includes: at least one excavation block;

[0022] The first excavation volume determination module is configured to determine the first excavation volume of the excavation part within a preset time period according to the three-dimensional model of the completed excavated part of the excavation part; the first moment is the end moment of the preset time period;

[0023] The second excavation volume determination module is configured to determine the second excavation volume of the excavation part within a preset time period according to the video stream data of the excavation part within the preset time period;

[0024] The correction coefficient determination module is configured to determine a correction coefficient according to the first excavation volume and the second excavation volume;

[0025] The real-time excavation volume determination module is configured to determine the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block;

[0026] The correction module is configured to correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the excavation volume measurement result of the excavation block;

[0027] Wherein, the video stream data includes: a plurality of frame images arranged in time sequence;

[0028] The real-time excavation volume determination module includes:

[0029] The equipment matching module is configured to determine the equipment matching pairs in the plurality of frame images arranged in time sequence according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: an excavating equipment and a target self-unloading equipment matched with it;

[0030] The action recognition module is configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time sequence according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images to obtain the action category corresponding to the construction machinery;

[0031] The excavation volume measurement module is configured to, when the action category of the excavating equipment in the equipment matching pair changes presetly, it is considered that the excavating equipment in the equipment matching pair has completed one excavation, and determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavating equipment.

[0032] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor is caused to execute the method as described in the embodiment of the present application.

[0033] An embodiment of the present application also discloses a machine-readable medium, on which executable code is stored, and when the executable code is executed, a processor is caused to execute the method as described in the embodiment of the present application.

[0034] The embodiments of the present application also disclose a computer program product, including computer programs / instructions, which implement the foregoing method when executed by a processor.

[0035] The embodiments of the present application include the following advantages:

[0036] The embodiments of the present application provide a method for calculating the whole-process excavation volume during the construction period based on multi-source data fusion. This method utilizes two types of multi-source data, namely point cloud data and video stream data, to achieve the whole-process calculation of the excavation volume during the construction period. Specifically, for the point cloud data, the point cloud data at the end moment of a preset time period for the excavation part is obtained and combined with the terrain surface model already constructed before the excavation of the excavation part, so as to determine the three-dimensional model of the already excavated part of the excavation part, and further determine the first excavation volume within this preset time period. For the video stream data, first, the frame images included therein are analyzed to identify the equipment category, equipment identifier, and detection frame of the construction machinery, thereby determining the equipment matching pairs (that is, an excavating equipment and a target self-unloading equipment that matches it), and at the same time identifying the action category of the construction machinery. Once the action category of the excavating equipment in the equipment matching pair undergoes a preset change, it is determined that one excavation has been completed, and the second excavation volume is determined in combination with the rated excavation capacity of this excavating equipment. Subsequently, a correction coefficient is determined based on the first and second excavation volumes, and this correction coefficient is used to correct the real-time cumulative excavation volume of the excavation blocks obtained based on the video stream data, and finally, an accurate and real-time excavation volume calculation result is obtained.

[0037] Among them, the point cloud data can accurately reflect the spatial position and shape of an object. Based on the three-dimensional model of the already excavated part of the excavation part constructed from the point cloud data, a relatively accurate first excavation volume calculation result can be obtained. Due to the fast real-time nature of the video stream data, it can be obtained and analyzed in real time, thereby obtaining a second excavation volume calculation result reflecting the real-time construction status.

[0038] The specific fusion processing method of the embodiments of the present application is to determine a correction coefficient according to the first excavation volume obtained based on the point cloud data and the second excavation volume obtained based on the video stream data. This correction coefficient can establish the correlation between the first excavation volume calculation result and the second excavation volume calculation result, and adjust the real-time cumulative excavation volume corresponding to the video stream data with the high accuracy of the first excavation volume calculation result corresponding to the point cloud data. Since the point cloud data has high accuracy, it can make up for the deficiency of the video stream data in terms of accuracy; while the video stream data has strong real-time performance, it can make up for the defect that the point cloud data is not updated in a timely manner. Therefore, the fusion processing of the embodiments of the present application can make the excavation volume calculation significantly improve the accuracy while following up the construction progress in real time, meeting the requirements of the whole-process real-time calculation of the excavation volume during the construction period in engineering construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the step flow of a method for calculating the whole-process excavation volume during the construction period based on multi-source data fusion according to an embodiment of the present application;

[0040] Figure 2 It is a schematic diagram of the step flow of a method for determining the real-time cumulative excavation volume of an excavation block according to the video stream data of the excavation block according to an embodiment of the present application;

[0041] Figure 3 It is a schematic diagram of the structure of an action recognition model according to an embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of the structure of a device for calculating the whole-process excavation volume during the construction period based on multi-source data fusion according to an embodiment of the present application;

[0043] Figure 5 It is a schematic diagram of the structure of a device provided according to an embodiment of the present application. Detailed implementation manners

[0044] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0045] The embodiments of the present application can be applied to engineering industries such as hydropower and civil engineering, and are used for real-time measurement of the corresponding excavation volume for the open-pit excavation operation in the engineering industry.

[0046] Taking the pumped storage power station project as an example, the project construction presents a situation where excavation operations are carried out simultaneously at multiple construction sites. At the same time, due to the continuous change of the on-site construction situation, there is a need for real-time measurement of the whole-process excavation volume during the construction period.

[0047] In the traditional technology, either the excavation volume during the design period is measured based on 3D design software, or the excavation volume during the construction period at stages is measured based on drone oblique photography. However, whether it is measuring the excavation volume during the design period based on 3D design software or measuring the excavation volume during the construction period at stages based on drone oblique photography, there are problems of poor real-time performance, and it is difficult to meet the need for real-time measurement of the whole-process excavation volume during the engineering construction.

[0048] In view of the technical problem that the traditional technology is difficult to meet the need for real-time measurement of the whole-process excavation volume during the engineering construction, the embodiments of the present application provide a method for calculating the whole-process excavation volume during the construction period based on multi-source data fusion. The method specifically includes the following steps:

[0049] Determine the 3D model of the completed excavated part of the excavation block based on the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the excavation area specifically includes: at least one excavation block;

[0050] Determine the first excavation volume of the excavation area within a preset time period based on the 3D model of the completed excavated part of the excavation area; the above first moment is the end moment of the preset time period;

[0051] Determine the second excavation volume of the excavation area within a preset time period based on the video stream data of the excavation area within the preset time period;

[0052] Determine the correction coefficient based on the first excavation volume and the second excavation volume;

[0053] Determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block;

[0054] Correct the real-time cumulative excavation volume of the excavation block according to the above correction coefficient to obtain the measurement result of the excavation volume of the excavation block;

[0055] Among them, the above video stream data includes: a plurality of frame images arranged in time;

[0056] The above determination of the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block includes:

[0057] Determine the equipment matching pairs in the above-mentioned plurality of frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the above frame images; the above equipment matching pairs include: an excavation equipment and a target dump truck equipment that matches it;

[0058] Perform action recognition on the construction machinery in the above-mentioned plurality of frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the above frame images to obtain the action category corresponding to the above construction machinery;

[0059] When the action category of the excavation equipment in the above equipment matching pair undergoes a preset change, it is considered that the excavation equipment in the above equipment matching pair has completed one excavation, and determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the above excavation equipment.

[0060] The embodiment of the present application provides a method for the whole-process measurement of the excavation volume during the construction period based on multi-source data fusion. This method utilizes two types of multi-source data, namely point cloud data and video stream data, to achieve the whole-process measurement of the excavation volume during the construction period. Specifically, for the point cloud data, the point cloud data of the excavation site at the end of the preset time period is obtained and combined with the terrain surface model that has been constructed before the excavation of the excavation site, so as to determine the three-dimensional model of the excavated part of the excavation site, and then determine the first excavation volume within the preset time period. For the video stream data, first, the frame images included in it are analyzed to identify the equipment category, equipment identifier, and detection frame of the construction machinery, thereby determining the equipment matching pair (that is, an excavating equipment and a target dump truck equipment that matches it), and at the same time identifying the action category of the construction machinery. Once the action category of the excavating equipment in the equipment matching pair undergoes a preset change, it is determined that one excavation has been completed, and the second excavation volume is determined in combination with the rated excavation capacity of the excavating equipment. Subsequently, a correction coefficient is determined based on the first and second excavation volumes, and this correction coefficient is used to correct the real-time cumulative excavation volume of the excavation blocks obtained based on the video stream data, and finally, an accurate and real-time excavation volume measurement result is obtained.

[0061] Among them, the point cloud data can accurately reflect the spatial position and shape of an object. Based on the three-dimensional model of the excavated part of the excavation site constructed from the point cloud data, a relatively accurate first excavation volume measurement result can be obtained. Due to the fast real-time nature of the video stream data, it can be obtained and analyzed in real time, thereby obtaining a second excavation volume measurement result that reflects the real-time construction status.

[0062] The specific fusion processing method in the embodiment of the present application is to determine a correction coefficient according to the first excavation volume obtained based on the point cloud data and the second excavation volume obtained based on the video stream data. This correction coefficient can establish a connection between the first excavation volume measurement result and the second excavation volume measurement result, and adjust the real-time cumulative excavation volume corresponding to the video stream data with the high accuracy of the first excavation volume measurement result corresponding to the point cloud data. Since the point cloud data has high accuracy, it can make up for the deficiency of the video stream data in terms of accuracy; while the video stream data has strong real-time performance, it can make up for the defect that the point cloud data is not updated in a timely manner. Therefore, the fusion processing in the embodiment of the present application can not only keep up with the construction progress in real time for the excavation volume measurement, but also significantly improve the accuracy, meeting the requirements of the whole-process real-time measurement of the excavation volume during the construction period in engineering construction.

[0063] Method Embodiment 1

[0064] Reference Figure 1 , which shows the schematic flow chart of the steps of the method for the whole-process measurement of the excavation volume during the construction period based on multi-source data fusion in an embodiment of the present application. This method specifically includes the following steps:

[0065] Step 101: Determine the 3D model of the completed excavated part of the excavation area based on the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the above-mentioned excavation area specifically includes: at least one excavation block.

[0066] Step 102: Determine the first excavation volume of the excavation area within a preset time period according to the 3D model of the completed excavated part of the excavation area; the above-mentioned first moment is the end moment of the preset time period.

[0067] Step 103: Determine the second excavation volume of the excavation area within a preset time period according to the video stream data of the excavation area within the preset time period.

[0068] Step 104: Determine the correction coefficient according to the first excavation volume and the second excavation volume.

[0069] Step 105: Determine the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block.

[0070] Step 106: Correct the real-time cumulative excavation volume of the excavation block according to the above-mentioned correction coefficient to obtain the measurement result of the excavation volume of the excavation block.

[0071] Wherein, the above-mentioned video stream data includes: a plurality of frame images arranged in time sequence.

[0072] The process of the above-mentioned Step 105 for determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block specifically includes:

[0073] Determine the equipment matching pairs in the above-mentioned plurality of frame images arranged in time sequence according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the above-mentioned frame images; the above-mentioned equipment matching pairs include: an excavating equipment and a target dump truck equipment that matches it.

[0074] Perform action recognition on the construction machinery in the above-mentioned plurality of frame images arranged in time sequence according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the above-mentioned frame images to obtain the action category corresponding to the above-mentioned construction machinery.

[0075] When the action category of the excavating equipment in the above-mentioned equipment matching pair undergoes a preset change, it is considered that the excavating equipment in the above-mentioned equipment matching pair has completed one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the above-mentioned excavating equipment.

[0076] Figure 1 The method shown can be used to realize the whole-process measurement of the excavation volume during the construction period by means of two types of multi-source data, namely point cloud data and video stream data. Figure 1 At least one of the steps included can be executed by the server. It can be understood that the embodiments of the present applicationFigure 1 The specific execution entity of the method shown is not restricted.

[0077] In the embodiments of the present application, the excavation area refers to the specific scope where excavation operations need to be carried out for engineering objects such as slopes and dams. It is a comprehensive concept that covers the area where excavation work is to be carried out, and its scale and boundaries are determined according to the actual planning and requirements of the project. The excavation area can be composed of one or more excavation regions.

[0078] An excavation region is a subdivided part of the excavation area and is further divided from the perspective of construction organization, process arrangement, etc. Each excavation region has its own independent construction plan and requirements and is generally composed of at least one excavation block.

[0079] An excavation block is the basic construction unit in the excavation operations of engineering objects such as slopes and dams and is also the smallest statistical unit when calculating the excavated volume.

[0080] In step 101, the point cloud data is a set of discrete points obtained through technologies such as laser scanning or drone oblique photography. This set of discrete points carries accurate spatial coordinate information and can reflect the spatial position and shape of an object with high precision.

[0081] The point cloud data of the excavation area at the first moment details the actual terrain surface information of the excavation area at the first moment, including various shapes and undulations formed after excavation.

[0082] The terrain surface model before excavation is a digital terrain model constructed before the excavation operations of the engineering object and represents the original terrain condition of the excavation area before any excavation operations. In practical applications, according to the design information of the engineering object corresponding to the excavation area, the terrain surface model before excavation of the excavation area can be established using contour lines and elevation points in the original terrain.

[0083] Among them, the design information is used to clarify the scope and accuracy requirements for constructing the terrain surface model and gives specific plans such as site boundaries and overall layouts according to the engineering characteristics. Contour lines are used to visually display the terrain undulations, reflect the slope changes through their density and shape, and can also extract elevation data points at a certain interval to assist in identifying various terrain features. Elevation points are used to provide the accurate elevation values of specific positions on the terrain surface. As basic data, with the help of interpolation algorithms, they cooperate with surrounding elevation points to calculate the elevation of unknown points, thereby generating continuous data to accurately construct the terrain surface model.

[0084] In the embodiments of the present application, the point cloud data of the excavation site at the first moment can be compared and analyzed with the data of the corresponding area in the pre-excavation terrain surface model. The above analysis can identify the spatial position difference between the two. In this way, based on the spatial position difference and with the help of three-dimensional modeling technology, the excavated part of the excavation site can be three-dimensionally reconstructed to obtain a three-dimensional model of the excavated part of the excavation site.

[0085] In a specific implementation, the process of determining the three-dimensional model of the excavated part of the excavation site according to the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation in step 101 specifically includes:

[0086] Step 111: Perform grid interpolation on the point cloud data of the excavation site at the first moment to obtain the actual excavation contour surface of the excavation site at the first moment;

[0087] Step 112: Determine the first envelope of the excavation site at the first moment according to the actual excavation contour surface of the excavation site at the first moment;

[0088] Step 113: Perform a Boolean subtraction operation on the first envelope of the excavation site at the first moment and the terrain surface model of the excavation site before excavation to obtain the operation result;

[0089] Step 114: Obtain the three-dimensional model of the excavated part of the excavation site from the operation result according to the spatial range of the excavation site.

[0090] After starting the excavation operation of the excavation site, at the first moment, the actual excavation contour surface formed by the excavation operation can be scanned to obtain the point cloud data at the first moment. The corresponding scanning method can be unmanned aerial vehicle oblique photography or three-dimensional laser scanning.

[0091] Optionally, the embodiments of the present application can perform smoothing processing on the scanned point cloud data. The above smoothing processing is used to eliminate abnormal noises such as sharp corners and protrusions in the point cloud data to obtain smooth point cloud data.

[0092] Optionally, the embodiments of the present application can perform point cloud segmentation on the scanned point cloud data or the smooth point cloud data according to the spatial range of the excavation site to obtain the point cloud data of the excavation site at the first moment.

[0093] In step 111, the point cloud data is a set of discrete points. These discrete points are usually discontinuous and cannot directly form a continuous surface for analysis. The purpose of grid interpolation is to convert these discrete points into a continuous surface. By estimating data between known points, a regular grid structure is generated. As a result of the network interpolation, uniform grid point cloud data is used to represent the actual excavation contour surface of the excavation site at the first moment.

[0094] Common grid interpolation methods include: inverse distance weighted interpolation, Kriging interpolation, spline interpolation, etc. For example, inverse distance weighted interpolation is based on the principle of inverse distance, believing that the interpolation point closer to the known point is more affected by the known point. It can be understood that the embodiments of the present application do not limit the specific grid interpolation method.

[0095] Use the interpolation algorithm to perform grid interpolation on the point cloud data, such as the Kriging interpolation algorithm, to obtain uniform grid point cloud data for characterizing the actual excavation contour surface.

[0096] In step 112, the actual excavation contour surface is a two-dimensional surface. To more comprehensively describe the spatial morphology of the excavation part, it is necessary to construct a three-dimensional space body that can completely contain the actual excavation contour surface, that is, the first envelope body.

[0097] In practical applications, the actual excavation contour surface can be stretched along the horizontal direction into the first envelope body. Specifically, first determine the boundary of the actual excavation contour surface; then, according to the boundary of the actual excavation contour surface, use the surface reconstruction algorithm to expand the actual excavation contour surface into the first envelope body.

[0098] In step 113, the Boolean difference operation is an operation used to calculate the difference between two geometric entities in three-dimensional space. By subtracting the first envelope body from the terrain surface model before excavation, the difference part between the terrain after excavation and before excavation can be obtained, and this part is the actual excavation area, providing key data for determining the three-dimensional model of the excavated completed part.

[0099] The operation result is used to characterize a new geometric entity, and this new entity represents the remaining part after removing the space occupied by the first envelope body from the original terrain, that is, it reflects the actual excavation area.

[0100] In step 114, the operation result obtained through the Boolean difference operation may contain relevant information of multiple excavation parts. The embodiments of the present application can screen out the excavated parts belonging to a specific excavation part from the operation result according to the spatial range of the excavation part, so as to obtain the three-dimensional model of the excavated completed part of the specific excavation part.

[0101] In three-dimensional space, the spatial range of the excavation part can be defined by the coordinate boundaries of the excavation part (such as the minimum and maximum X, Y, Z coordinate values).

[0102] In specific implementation, first, according to the spatial range of the excavation part, screen the geometric elements (such as points, lines, surfaces, etc.) in the operation result, retain the target geometric elements within the spatial range, and discard the geometric elements outside the spatial range. Then, generate the three-dimensional model of the excavated completed part of the excavation part according to the target geometric elements.

[0103] In step 102, the three-dimensional model is essentially a spatial entity composed of a series of geometric elements (points, lines, surfaces, etc.). Through mathematical methods, such as integral operations or voxel-based calculation methods based on discretization, etc., the spatial volume occupied by this three-dimensional model can be calculated.

[0104] Since the first moment is set as the end moment of the preset time period, the three-dimensional model of the completed excavated part of the excavation area obtained at the first moment represents the excavation result within the preset time period. By calculating the volume of this three-dimensional model, the excavation volume within this preset time period, that is, the first excavation volume, can be directly obtained.

[0105] Therefore, the process of determining the first excavation volume of the excavation area within the preset time period according to the three-dimensional model of the completed excavated part of the excavation area in step 102 specifically includes: dividing the three-dimensional model of the completed excavated part of the excavation area into grid cells; determining the unit volume of the grid cells in the three-dimensional model of the completed excavated part of the excavation area; and summing up the unit volumes of all grid cells in the three-dimensional model of the completed excavated part of the excavation area to obtain the first excavation volume of the excavation area within the preset time period.

[0106] In step 103, the first frame image in the video stream data within the preset time period can be analyzed in the order from front to back in time to obtain the first device matching pair in the first frame image and the action category corresponding to the construction machinery in the first frame image; and when a preset change occurs in the action category of the device in the first device matching pair, it is considered that the excavating device in the first device matching pair has completed one excavation, and according to the rated excavation capacity corresponding to the excavating device, the second excavation volume of the excavation area within the preset time period is determined.

[0107] The determination process of the second excavation volume monitors the excavation volume during the construction period in real time according to the objective law of the operation cooperation between the excavating device and the dump device during the construction process. The above objective law specifically includes: when the excavating device in the device matching pair completes a specific sequence of action categories each time, it is considered to have completed one effective excavation, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavating device. Therefore, the embodiment of the present application can accurately determine the actual occurrence of each effective excavation by combining the monitoring result of the preset change in the action category of the excavating device in the device matching pair, and thus determine the real-time cumulative excavation volume corresponding to the excavation area by accumulating the rated excavation capacity.

[0108] Since the above objective laws of the embodiments of the present application can be based on the internal logical connection between the actual process of the excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters, the real-time monitoring of the excavation volume during the construction period can be realized. Therefore, the embodiments of the present application can, based on the analysis of the video stream data, achieve the real-time monitoring of the excavation volume during the construction period at a relatively fast speed, and thus can improve the processing efficiency of the earthwork excavation volume.

[0109] In step 104, according to the first excavation volume and the second excavation volume, a correction coefficient is determined; this correction coefficient can establish the correlation between the measurement result of the first excavation volume and the measurement result of the second excavation volume, and with the high precision of the measurement result of the first excavation volume corresponding to the point cloud data, adjust the real-time cumulative excavation volume corresponding to the video stream data.

[0110] In a specific implementation, the correction coefficient can be determined based on a proportional relationship. If the relationship between the first excavation volume and the second excavation volume is linear, the ratio of the two can be calculated as the correction coefficient. For example, in multiple measurements, calculate the average value of the ratio of the measurement result of the first excavation volume to the measurement result of the second excavation volume, and use it as the correction coefficient. One measurement can correspond to a preset time period. The length of the preset time period can be determined by those skilled in the art according to actual application requirements. For example, the length of the preset time period can be N days, where N is a positive integer. For example, at the end of the Nth day of the excavation operation at the excavation site, the first measurement is carried out. At the end of the 2Nth day of the excavation operation at the excavation site, the second measurement is carried out.

[0111] During the process of multiple measurements, after each measurement is completed, the measurement results of the first excavation volume and the second excavation volume need to be recorded in detail. Subsequently, for each measurement result, calculate the ratio of the measurement result of the first excavation volume to the measurement result of the second excavation volume.

[0112] Record the ratio obtained each time in sequence, add up all these ratios, and then divide the sum of the addition by the total number of measurements. The average value finally obtained is the correction coefficient to be determined. This correction coefficient can effectively reflect the relatively stable proportional relationship between the first excavation volume and the second excavation volume.

[0113] Alternatively, the correction coefficient can be determined using the least squares method. When the relationship between the first excavation volume and the second excavation volume is relatively complex and not completely a simple linear relationship, the least squares method can be used to determine the correction coefficient.

[0114] Alternatively, the correction coefficient can be determined based on statistical analysis methods. A large amount of data on the first excavation volume and the second excavation volume can be statistically analyzed, such as plotting a scatter plot of the two and observing the distribution law of the data. If the data shows a certain clustering or distribution trend, the correction coefficient can be determined based on these clustering or distribution trends.

[0115] Alternatively, for more complex cases, regression algorithms in machine learning can be used to determine the correction coefficient. Taking the data of the first excavation volume and the second excavation volume as training samples, a suitable regression model is selected, such as linear regression, polynomial regression, support vector regression, etc. By training the model, the mapping relationship between the two is learned, and thus the correction coefficient is obtained. For example, using a linear regression model, during the training process, the linear regression model will automatically adjust the parameters to minimize the error between the predicted first excavation volume and the actual first excavation volume. The coefficient related to the second excavation volume in the finally obtained model parameters is the correction coefficient.

[0116] In step 105, the determination process of the real-time cumulative excavation volume can also monitor the excavation volume during the construction period in real time according to the objective laws of the operation cooperation between the excavation equipment and the dump truck equipment during the construction process. The above objective laws specifically include: for the equipment matching pair, each time the excavation equipment completes a specific sequence of action categories, it is considered to complete an effective excavation, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can accurately determine the actual occurrence of each effective excavation by combining the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair, and thus determine the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0117] In step 106, according to the above correction coefficient, the real-time cumulative excavation volume of the excavation block is corrected to obtain the measurement result of the excavation volume of the excavation block.

[0118] Assume that the correction coefficient is k, and it is known that the real-time cumulative excavation volume of the excavation block is V1. Then the method for correcting the real-time cumulative excavation volume of the excavation block can include: a linear correction method or a correction method based on machine learning, etc.

[0119] Among them, in the linear correction method, it can be considered that there is a linear relationship between the first excavation volume and the second excavation volume, and the correction coefficient k is used as the proportional factor between the two. Then, the product of the correction coefficient k and the real-time cumulative excavation volume V1 can be used as the measurement result V2 of the excavation volume of the excavation block.

[0120] The correction method based on machine learning can utilize the powerful non-linear fitting ability of machine learning algorithms to model complex data relationships. Taking a large amount of data of the first excavation volume and the second excavation volume as the training set, a suitable machine learning model (such as support vector machine regression, decision tree regression, neural network regression, etc.) is selected for training.

[0121] Taking support vector machine regression as an example, first, preprocess the training data, including operations such as data normalization. Then, use the training data to train the support vector machine model and adjust the parameters of the machine learning model to enable it to fit the relationship between the first excavation volume and the second excavation volume as accurately as possible. After training, use the real-time cumulative excavation volume V1 as the input of the machine learning model, and the output of the machine learning model can be used as the measurement result V2 of the excavation volume of the excavation block.

[0122] The embodiments of the present application can perform subsequent processing such as recording on the measurement result V2 of the excavation volume of the excavation block. For example, store the measurement result V2 of the excavation volume in a dedicated engineering database and classify and store it according to information such as the timestamp and the excavation block number for easy query and call at any time in the future. Another example is to generate a detailed excavation volume statistical report based on the measurement result V2 of the excavation volume. The report covers the comparative analysis of the excavation volumes of different time periods and different excavation blocks, providing intuitive data support for the control of the project progress. Or, by comparing the measurement result of the excavation volume with the excavation volume index in the project budget, if it is found that the deviation between the actual excavation volume and the budget exceeds the preset range, trigger the warning mechanism in time to remind the project management personnel to check and adjust aspects such as the construction process and equipment use.

[0123] In an alternative implementation manner of the present application, the above method may further include: determining the over-excavation or under-excavation situation of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the designed volume of the first excavation block; adjusting the remaining excavation volume of the second excavation block in the (s + 1)-th excavation area according to the over-excavation or under-excavation situation of the first excavation block. s can be a positive integer. The s-th excavation area and the (s + 1)-th excavation area may be two adjacent excavation areas in terms of geographical location.

[0124] The over-excavation or under-excavation situation specifically includes: over-excavation or under-excavation. Over-excavation means that the real-time cumulative excavation volume of the first excavation block exceeds the designed volume of the first excavation block, and under-excavation means that the real-time cumulative excavation volume of the first excavation block is less than the designed volume of the first excavation block.

[0125] Since the s-th excavation area and the (s + 1)-th excavation area are usually two adjacent excavation areas in terms of geographical location, this adjacent relationship makes it important to draw on experience and adjust data during the construction process.

[0126] The embodiments of the present application can compare the real-time cumulative excavation volume of the first excavation block with the designed volume of the first excavation block to obtain the specific over-excavation amount or under-excavation amount, and then the remaining excavation volume of the second excavation block is correspondingly reduced or increased on this basis.

[0127] For example, if the first excavation block is over-excavated, the adjusted remaining excavation volume of the second excavation block is specifically: the designed volume of the second excavation block minus the over-excavation volume of the first excavation block. Or, if the first excavation block is under-excavated, the adjusted remaining excavation volume of the second excavation block is specifically: the designed volume of the second excavation block plus the under-excavation volume of the first excavation block. Of course, in practical applications, factors such as soil loosening, construction errors, and the impact of subsequent processes also need to be considered to correct the adjusted remaining excavation volume.

[0128] In an implementation manner of the present application, the process of determining the designed volume of the excavation block specifically includes:

[0129] Step S1: According to the excavation contour line of the excavation area, establish an excavation surface model of the excavation area after excavation;

[0130] Step S2: Perform a Boolean operation on the second envelope corresponding to the terrain surface model and the third envelope corresponding to the excavation surface model to obtain a three-dimensional model of the excavation area;

[0131] Step S3: According to the spatial range of the excavation block, obtain the three-dimensional model of the excavation block from the three-dimensional model of the excavation area;

[0132] Step S4: Determine the designed volume of the excavation block according to the three-dimensional model of the excavation block.

[0133] In Step S1, the excavation contour line is the boundary defining the final shape of the excavation area. Through mathematical methods such as interpolation and fitting techniques, the discrete excavation contour line data is converted into a continuous surface. This surface can be called the excavation surface model, which represents the shape of the excavation area after the excavation operation is completed.

[0134] In Step S2, the Boolean operation is a basic operation method in 3D modeling. The envelope is a closed spatial body that can completely contain the corresponding surface model. The second envelope of the terrain surface model represents the spatial range of the original terrain, and the third envelope of the excavation surface model represents the spatial range of the planned excavation. By means of operations such as the difference set operation in the Boolean operation, the spatial range of the planned excavation can be subtracted from the spatial range of the original terrain, and the result is the three-dimensional model of the excavation area. The three-dimensional model of the excavation area presents the spatial form formed on the basis of the original terrain after the excavation operation of the excavation area is completed.

[0135] In Step S3, based on the obtained three-dimensional model of the excavation area, according to the pre-determined spatial range of the excavation block, through operations such as spatial clipping or extraction, the part of the three-dimensional model of the excavation area that meets the spatial range is extracted, and the three-dimensional model of the specific excavation block is obtained.

[0136] In step S4, for the obtained three-dimensional model of the excavation block, various methods can be used to calculate its volume. A common method is based on the principle of numerical integration. The three-dimensional model is discretized into multiple volume units, such as voxels (similar to the corresponding of pixels in a two-dimensional image in three-dimensional space), and then the volumes of multiple volume units are accumulated to obtain the volume of the entire excavation block.

[0137] In summary, the whole-process measurement method for the excavation volume during the construction period based on multi-source data fusion in the embodiments of the present application uses these two types of multi-source data, namely point cloud data and video stream data, to achieve the whole-process measurement of the excavation volume during the construction period. Specifically, for the point cloud data, the point cloud data at the end moment of the preset time period of the excavation part is obtained and combined with the terrain surface model that has been constructed before the excavation of the excavation part, so as to determine the three-dimensional model of the excavated part of the excavation part, and then determine the first excavation volume within the preset time period. For the video stream data, first analyze the frame images included in it, identify the equipment category, equipment identifier, and detection frame of the construction machinery therein, thereby determining the equipment matching pair (that is, an excavation equipment and a target self-unloading equipment that matches it), and at the same time identify the action category of the construction machinery. Once the action category of the excavation equipment in the equipment matching pair changes presetly, it is considered that an excavation is completed, and the second excavation volume is determined in combination with the rated excavation capacity of the excavation equipment. Subsequently, a correction coefficient is determined based on the first and second excavation volumes, and this correction coefficient is used to correct the real-time cumulative excavation volume of the excavation block obtained based on the video stream data, and finally an accurate and real-time excavation volume measurement result is obtained.

[0138] Among them, the point cloud data can accurately reflect the spatial position and shape of an object. Based on the three-dimensional model of the excavated part of the excavation part constructed by the point cloud data, a relatively accurate first excavation volume measurement result can be obtained. Due to the fast real-time performance of the video stream data, it can be obtained and analyzed in real time, so as to obtain a second excavation volume measurement result reflecting the real-time construction state.

[0139] The specific fusion processing method in the embodiments of the present application is to determine a correction coefficient according to the first excavation volume obtained based on the point cloud data and the second excavation volume obtained based on the video stream data. This correction coefficient can establish the correlation between the first excavation volume measurement result and the second excavation volume measurement result, and adjust the real-time cumulative excavation volume corresponding to the video stream data with the high accuracy of the first excavation volume measurement result corresponding to the point cloud data. Because the point cloud data has high accuracy, it can make up for the deficiency of the video stream data in terms of accuracy; while the video stream data has strong real-time performance, it can make up for the defect that the point cloud data is not updated in time. Therefore, the fusion processing in the embodiments of the present application can not only keep up with the construction progress in real time for the excavation volume measurement, but also significantly improve the accuracy, meeting the requirements of the whole-process real-time measurement of the excavation volume during the construction period in engineering construction.

[0140] Method Embodiment 2

[0141] Reference Figure 2 , which shows the step - flow schematic diagram of the method for determining the real - time cumulative excavation volume of an excavation block according to the video - stream data of the excavation block in an embodiment of the present application. The method specifically includes the following steps:

[0142] Step 201: Determine the device matching pairs in multiple frame images arranged in time according to the device category, device identifier, and detection frame corresponding to the construction machinery in the frame images of the video - stream data; the above - mentioned device matching pairs include: one excavation device and one target dump device that matches it;

[0143] Step 202: Perform action recognition on the construction machinery in the above - mentioned multiple frame images arranged in time according to the device category, device identifier, and detection frame corresponding to the construction machinery in the frame images, so as to obtain the action category corresponding to the construction machinery;

[0144] Step 203: When the action category of the excavation device in the above - mentioned device matching pair undergoes a preset change, it is considered that the excavation device in the above - mentioned device matching pair has completed one excavation. Determine the real - time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavation device.

[0145] In step 201, the process of obtaining the video - stream data of the excavation block specifically includes: after the server receives the construction video sent by the image acquisition end, extract the video - stream data from the construction video. Among them, the server can receive the video - stream data from the image acquisition end according to a preset time period.

[0146] The multiple frame images in the video - stream data are arranged in sequence according to time. This means that as time progresses, the picture corresponding to each time point will form a separate frame image and be sorted according to the occurrence order. In the specific implementation process, it is an effective method to use frame numbers to distinguish different frame images. For example, the first frame image generated at the starting moment can be numbered as No. 1, and the frame number corresponding to the next moment is No. 2, and so on. Through such frame numbers, not only can different frame images at different times be clearly distinguished, but also in subsequent analysis and processing links such as target detection and construction machinery tracking, the frame images at specific time points can be accurately and quickly located and called according to the frame numbers, so as to better calculate relevant indicators such as the corresponding excavation volume based on the video - stream data.

[0147] In the scenario of hydropower engineering, multiple video sources can be set, and different video sources can correspond to different excavation blocks. Among them, the video source can continuously provide continuous video - stream data.

[0148] The embodiments of the present application can utilize object detection technology to perform object detection on the above video stream data, so as to obtain the equipment category and detection frame corresponding to the construction machinery in the above frame image.

[0149] Object detection technology is an important branch in the field of computer vision, which aims to identify the object information corresponding to the objects in an image. Specifically in the embodiments of the present application, the frame image may include objects such as excavation equipment and dump equipment. Examples of excavation equipment may include: excavators, etc. Examples of dump equipment may include: dump trucks, etc. In practical applications, a dump truck drives into the excavation site and stops within the working radius of the excavator so that the excavator can easily load materials into the cargo box of the dump truck.

[0150] The object information may specifically include: the equipment category and detection frame corresponding to the construction machinery. The equipment category specifically includes: excavation equipment category and dump equipment. The detection frame is used to represent the position and range of the object in the frame image. The detection frame is usually a rectangular frame, and its four sides are respectively aligned with the outermost edges of the object, thus enclosing the entire object. The information of the detection frame may include: the upper left corner coordinates, width, and height of the rectangular frame; or, the information of the detection frame may include: the center point coordinates, width, and height of the rectangular frame.

[0151] In the embodiments of the present application, an object detection model can be utilized to perform object detection on the above frame image. In a specific implementation, the frame image can be input into the object detection model to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image output by the object detection model.

[0152] The object detection model is a deep learning model in the field of computer vision, which can identify the objects in the frame image and determine object information such as the detection frame and equipment category of the objects. The object detection model is usually implemented based on a convolutional neural network. The output result format of the object detection model can be: {(machine class, xmin, xmax, ymin, ymax)}. Among them, "machine class" represents the category to which the construction machinery in the frame image belongs, such as "dump equipment" or "excavation equipment", etc., which is used to distinguish different categories of construction machinery; "xmin" and "xmax" are respectively the leftmost and rightmost coordinate values of the detection frame of the construction machinery in the horizontal direction (x-axis) of the frame image; "ymin" and "ymax" are respectively the uppermost and lowermost coordinate values of the detection frame of the construction machinery in the vertical direction (y-axis) of the frame image, and these four coordinate values jointly determine a rectangular detection frame.

[0153] The embodiments of the present application do not limit the specific object detection model. For example, examples of the object detection model may include: the YOLO (You Only Look Once) series, SSD (Single Shot MultiBox Detector), etc.

[0154] In one example, the object detection model specifically includes: a feature extraction unit, a feature fusion unit, and a detection unit.

[0155] Among them, the feature extraction unit is used to extract image features from the frame image.

[0156] The feature fusion unit is used to effectively fuse image features of different levels and different types, and the obtained fused image features can make the feature information more comprehensive and accurate, so as to enhance the expression ability of construction machinery features.

[0157] The detection unit is used to judge the category and determine the position of the construction machinery in the frame image according to the fused image features.

[0158] The embodiments of the present application can use object tracking technology to track the construction machinery in the above-mentioned multiple frame images arranged in time. The output result format of the object tracking technology can be: {(machine class, machineID, xmin, xmax, ymin, ymax)}, where machineID represents the machine identifier corresponding to the construction machinery.

[0159] Object tracking technology is a technology that, in a sequence of frame images, for a specific object (the construction machinery in the embodiments of the present invention), determines the position of the object in different frame images through a series of algorithms and models, and continuously estimates and updates the object state (including position, speed, motion direction, etc.). Its core is to achieve accurate recognition and continuous tracking of the object in a complex image environment and changing object states.

[0160] In specific implementation, the process of tracking the construction machinery in the above-mentioned multiple frame images arranged in time specifically includes:

[0161] Step A1: Create a track ID (Identity) in the track set for the detection box in the first frame image;

[0162] Step A2: Extract the feature of the region image within the detection box in the kth frame image to obtain the first appearance feature vector of the detection box in the kth frame image; the first appearance feature vector includes: texture feature and color feature; the weight of the texture feature is greater than the weight of the color feature; k is a positive integer;

[0163] Step A3: Determine the predicted trajectory of the trajectory ID in the k-th frame image according to the historical trajectory information of the trajectory ID in the trajectory set.

[0164] Step A4: Determine the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the trajectory ID; the states of the trajectory ID include: confirmed state or unconfirmed state.

[0165] Step A5: Assign a mechanical identifier to the construction machinery corresponding to the trajectory ID in the confirmed state.

[0166] Steps A1 to A5 of the embodiments of the present application use feature matching and trajectory prediction to achieve the tracking of construction machinery in multiple frame images.

[0167] In the scenario of hydropower engineering, construction machinery of the same category is similar in appearance, shape, color, etc., which easily leads to errors in the matching process between the trajectory ID and the detection box by the trajectory tracking means. In the process of calculating the first appearance feature vector of the region image within the detection box in the embodiments of the present application, the weight of the texture feature is controlled to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, and the color feature has a lower discrimination degree when the models and appearance colors are similar, on this basis, the embodiments of the present application determine the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the trajectory ID, so as to more accurately distinguish different construction machinery, reduce the misjudgment probability caused by similar appearance, and improve the discrimination degree of construction machinery.

[0168] Among them, step A1 initializes the tracking trajectory by creating a trajectory ID for the detection box in the initial first frame image.

[0169] Step A2 extracts the appearance feature vectors of the detection box regions in each frame image, with particular consideration given to the texture feature because in the construction scenario, the texture feature of the construction machinery is more stable and discriminatory than the color feature.

[0170] For example, for two excavators of the same model, their colors and appearances are similar, but they may be different in terms of wear and scratches, cleanliness, corrosion, warning signs, and safety signs. The texture feature of the embodiments of the present application can precisely reflect the differences between two excavators of the same model in terms of wear and scratches, cleanliness, corrosion, warning signs, and safety signs.

[0171] The process of step A2 extracting features from the region image within the detection box in the k-th frame image specifically includes:

[0172] Step A21: Use the convolutional module to extract the first feature vector of the regional image;

[0173] Step A22: Use the local binary pattern method to extract the second feature vector of the regional image;

[0174] Step A23: Fuse the first feature vector and the second feature vector to obtain the appearance feature vector.

[0175] In Step A21, the convolutional module can be constructed based on models such as CNN (Convolutional Neural Network) and VGG (Visual Geometry Group).

[0176] The first feature vector extracted by the convolutional module is high-dimensional, usually containing hundreds or thousands or even more dimensions. Examples of the dimensions of the first feature vector can include: spatial features, hierarchical features, scale features, orientation features, color features, depth features, context features, etc. Among them, the spatial features represent the spatial structure information of the construction machinery, such as edges, corners, textures, etc. Therefore, the spatial features can include: texture features.

[0177] LBP (Local Binary Patterns) is a simple and effective algorithm for describing image texture features. The LBP algorithm generates a binary code by comparing the gray value of each pixel in the image with the gray values of its neighboring pixels, and then converts these binary codes into decimal numbers to form a texture descriptor. Since LBP can effectively describe the local texture features of the image and capture the tiny changes and patterns in the image; therefore, the second feature vector can contain: texture features with strong discrimination ability.

[0178] The process of fusing the first feature vector and the second feature vector in the embodiment of the present application specifically includes: performing weighted processing on the features of the same dimension in the first feature vector and the second feature vector to obtain a weighted processing result; the features of the same dimension include: texture features; for the weighted processing result corresponding to the texture features, increase the corresponding value.

[0179] Assume that the weight of the first feature vector is w1, the value of the texture feature in the first feature vector is f1, the weight of the second feature vector is w2, and the value of the texture feature in the second feature vector is f2. Then the weighted processing result corresponding to the texture feature can be: w = w1 * f1 + w2 * f2. In order to highlight the importance of the second feature vector, w2 can be greater than w1. It can be understood that those skilled in the art can determine the specific values of w1 and w2 according to actual application requirements.

[0180] In the embodiment of the present application, for the weighted processing result corresponding to the texture feature, adding a corresponding value can achieve the technical effect that the weight of the texture feature is greater than the weight of the color feature. The added value can be determined by those skilled in the art according to actual application requirements.

[0181] In step A3, according to the historical trajectory information of the trajectory ID in the trajectory set, determine the predicted trajectory of the trajectory ID in the k-th frame of the video stream image. k can be a positive integer greater than 1.

[0182] The historical trajectory information may include: the position information of the construction machinery or detection frame corresponding to the trajectory ID in the past frame images, and these position information usually include information such as coordinates, timestamps, speeds, directions, etc.

[0183] The embodiment of the present application can use the Kalman filtering method to determine the predicted trajectory of the trajectory ID in the k-th frame of the video stream image. Kalman filtering is an efficient autoregressive filter that can estimate the state of a linear dynamic system from a series of incomplete and noisy measurements.

[0184] In step A4, the state of the trajectory ID may include: an unconfirmed state or a confirmed state. Among them, the confirmed state may mean that the trajectory ID has a corresponding target detection frame after matching calculation. The unconfirmed state may mean that the trajectory ID is generated due to the initialization of the first frame image or the appearance of a new construction machinery target object, and does not have a corresponding target detection frame after matching calculation.

[0185] The process of step A4 for determining the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID specifically includes:

[0186] Step A41: Perform a first match on the trajectory IDs with the confirmed state in the trajectory set and the detection frames in the detection frame set to obtain a corresponding first match result; among them, in the first match process, determine the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID, and determine the first match result according to the cosine distance; the first match result includes: the trajectory IDs with the first match successful, the trajectory IDs with the first match failed, and the detection frames with the first match failed;

[0187] Among them, the trajectory set is a data structure for recording the trajectories of construction machinery in frame images. It assigns a unique trajectory ID to each tracked construction machinery, and through this trajectory ID, historical trajectory information such as the positions and appearance features of the construction machinery in past frame images can be indexed. For example, when predicting the position of a certain construction machinery in the k-th frame image using Kalman filtering, the historical position information (such as coordinates, speed, direction, etc.) in the trajectory set serves as important input data, thereby realizing the continuous estimation and update of the target motion state.

[0188] The detection box set stores the position information (in the form of detection boxes) of the construction machinery detected in each frame image. These detection boxes determine the specific range of the target in the frame image.

[0189] Step A42: Perform a second matching between the trajectory ID with the first matching failure or the trajectory ID with a non - confirmed state and the detection box with the first matching failure to obtain the corresponding second matching result; the second matching is an intersection - over - union matching; the second matching result includes: the trajectory ID with the second matching success, the trajectory ID with the second matching failure, and the detection box with the second matching failure;

[0190] Step A43: For the detection box with the second matching failure, determine it as a new construction machinery target object, and create a trajectory ID with a non - confirmed state in the trajectory set.

[0191] Step A44: When the second matching result of the trajectory ID with a non - confirmed state is a matching success, update the state of the trajectory ID to a confirmed state; and when the number of second matching failures of the trajectory ID with a confirmed state exceeds the first threshold, delete the trajectory ID.

[0192] In step A41, the embodiments of the present application can determine the state of the newly created trajectory ID according to actual application requirements. For example, when creating a trajectory ID in the trajectory set for the detection box in the first - frame video stream image, the state of the trajectory ID corresponding to the detection box in the first - frame video stream image can be set to a non - confirmed state. Another example is that for the detection box with the second matching failure, it is determined as a new construction machinery target object, and a trajectory ID with a non - confirmed state is created.

[0193] It should be noted that the embodiments of the present application provide a switching method from a non - confirmed state to a confirmed state. Specifically, for the trajectory ID in the non - confirmed state, the embodiments of the present application will verify the authenticity of the construction machinery corresponding to the trajectory ID in the non - confirmed state based on the subsequent second matching results. The above verification means can identify and exclude errors in the target detection results, thereby reducing errors in the construction machinery counting results caused by errors in the target detection results.

[0194] In the first matching process of the embodiments of the present application, the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the trajectory ID is determined, and the first matching result is determined according to the above cosine distance.

[0195] The second appearance feature vector is an appearance feature vector associated with the trajectory ID, which represents the appearance features of the detection box corresponding to the trajectory ID in the historical video stream images.

[0196] In practical applications, the second appearance feature vector corresponding to the trajectory ID can be determined according to the first appearance feature vector of the detection box corresponding to the trajectory ID in the historical video stream images.

[0197] For example, if the trajectory ID1 is created according to the detection box in the first-frame video stream image, then the second appearance feature vector B2 of the trajectory ID1 in the case of the second-frame video stream image can be determined according to the first appearance vector A1 of the detection box A of the trajectory ID1 in the first-frame video stream image. The second appearance feature vector B2 can be equal to the first appearance vector A1.

[0198] For example, if the trajectory ID2 is created according to the detection box B in the first-frame video stream image, and the trajectory ID2 successfully matches the detection box C in the second-frame video stream image, and the first appearance vector C2 of the detection box C in the second-frame video stream image, then the second appearance feature vector B3 of the trajectory ID2 in the case of the third-frame video stream image can be jointly determined according to the first appearance vector B1 of the detection box B of the trajectory ID2 in the first-frame video stream image and the first appearance vector C2 of the detection box C in the second-frame video stream image. Of course, the embodiments of the present application do not limit the specific process of jointly determining the second appearance feature vector B3 of the trajectory ID2 in the case of the third-frame video stream image.

[0199] Since the texture features included in the first appearance feature vector or the second appearance feature vector of the embodiments of the present application can better reflect the surface structure and details of the construction machinery, on this basis, the embodiments of the present application determine the first matching result according to the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the trajectory ID, which can effectively reduce the errors that occur in the first matching process between the trajectory ID and the detection box by the trajectory tracking means. In other words, the first matching of the embodiments of the present application can correctly associate the detection box of each construction machinery with its corresponding trajectory ID based on accurate appearance feature matching, which can effectively reduce the matching errors caused by similar appearances, and thus can improve the accuracy of the construction machinery counting analysis in the hydropower project scenario.

[0200] In an implementation of the present application, the process of the first matching of the track IDs with the confirmed status in the track set and the detection boxes in the detection box set in step A41 specifically includes:

[0201] Step A411: Determine the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the track ID with the confirmed status.

[0202] Step A412: Determine the Mahalanobis distance between the detection box and the track ID with the confirmed status.

[0203] Step A413: Construct a cost matrix; the elements in the cost matrix represent the matching cost between a detection box and a track; the value of the element is determined according to the cosine distance and the Mahalanobis distance.

[0204] Step A414: Solve the cost matrix with the goal of minimizing the total matching cost to obtain the first matching result.

[0205] In step A411, the cosine distance is a commonly used similarity measurement method. It measures the similarity between two vectors by calculating the cosine value of the angle between them. The smaller the cosine distance, the more similar the two vectors are, indicating that the detection box may correspond to the same construction machinery as the track ID. The cosine distance reflects the similarity in appearance features between the detection box and the track ID.

[0206] In step A412, the Mahalanobis distance is a statistic used to measure the similarity between two random vectors. The Mahalanobis distance takes into account the covariance structure in the data, so it can better handle the correlation and scale differences in the data. Generally speaking, the smaller the Mahalanobis distance, the higher the similarity between the two vectors. The Mahalanobis distance characterizes the similarity in position features between the detection box and the track ID.

[0207] In the process of determining the Mahalanobis distance between the detection box and the track ID with the confirmed status, the first position vector corresponding to the detection box can be determined, the second position vector corresponding to the track ID can be determined, and the Mahalanobis distance between the first position vector and the second position vector can be calculated.

[0208] Among them, the second position vector can be determined according to the aforementioned position information. The position information corresponding to the track ID can be determined by the Kalman filtering method.

[0209] In step A413, the cost matrix is a two-dimensional matrix, and each element in it represents the matching cost between a detection box and a track ID. This matching cost is usually determined based on a combination of cosine distance and Mahalanobis distance. For example, the cosine distance and Mahalanobis distance can be weighted and summed, or other functions can be used to combine the cosine distance and Mahalanobis distance to form the final matching cost.

[0210] It should be noted that before constructing the cost matrix, the cosine distance and Mahalanobis distance can also be filtered to filter out track IDs with relatively large cosine distance or Mahalanobis distance.

[0211] In step A414, optimization methods such as the Hungarian algorithm and linear assignment problem solver can be adopted to find the matching scheme that minimizes the total matching cost. Among them, the Hungarian algorithm is a classic combinatorial optimization algorithm suitable for solving the minimum weight matching problem of bipartite graphs. By solving the cost matrix, it can be determined which detection boxes should be matched with which track IDs.

[0212] The first matching result is obtained by solving the cost matrix. Each element in the cost matrix represents the matching cost between a detection box and a track ID, and this cost is obtained by integrating appearance features (cosine distance) and position features (Mahalanobis distance). The purpose of solving the cost matrix is to find a matching scheme that minimizes the total matching cost.

[0213] For example, during the process of solving the cost matrix, the Hungarian algorithm will search for the optimal matching combination. If the matching cost between detection box A and track ID1 is the smallest among all possible combinations, then in the solution result, detection box A and track ID1 will be matched together, and this matching relationship constitutes a part of the first matching result. Therefore, the successfully matched detection boxes and track IDs in the first matching result are determined by the minimization solution of the cost matrix.

[0214] The first matching result of the embodiment of the present application specifically includes: the first successfully matched track IDs, the first failed track IDs, and the first failed detection boxes.

[0215] For the first successfully matched track IDs, the Kalman filtering method can be used to fuse and update their historical track information. Specifically, according to the position information of the successfully matched detection boxes, the position information corresponding to the first successfully matched track IDs is updated, and according to the first appearance feature vector of the successfully matched detection boxes, the second appearance feature vector of the first successfully matched track IDs is updated.

[0216] The embodiment of the present application can execute step A42 for the first failed track IDs and the first failed detection boxes.

[0217] Step A42 can perform a second matching on the trajectory ID with the first matching failure or the trajectory ID with a non - confirmed status and the detection box with the first matching failure to obtain the corresponding second matching result. As a supplement to the first matching, the second matching can handle those detection boxes and trajectory IDs that failed to be successfully matched in the appearance feature matching, and is applicable to the situation where the appearance feature matching fails due to occlusion, short - term disappearance, etc. Therefore, the second matching in the embodiments of the present application can further improve the accuracy of the construction machinery counting analysis in the hydropower engineering scenario.

[0218] The above - mentioned second matching can be intersection - over - union (IoU) matching. The IoU matching process specifically includes: calculating the intersection - over - union ratio of the area between the predicted box corresponding to the trajectory ID and the detection box. If the intersection - over - union ratio is greater than the set threshold, it is considered a successful match; otherwise, it is a failed match. The predicted box corresponding to the trajectory ID can also be a rectangular box, which can be determined by the Kalman filter.

[0219] The above - mentioned second matching result specifically includes: the trajectory ID with a successful second matching, the trajectory ID with a failed second matching, and the detection box with a failed second matching.

[0220] For the trajectory ID with a successful second matching, the Kalman filtering method can be used to fuse and update its historical trajectory information. Specifically, according to the position information of the detection box with a successful match, the position information corresponding to the trajectory ID with a successful second matching is updated, and according to the first appearance feature vector of the detection box with a successful match, the second appearance feature vector of the trajectory ID with a successful second matching is updated.

[0221] The embodiments of the present application can execute step A43 for the detection box with a failed second matching. Step A43 determines the detection box with a failed second matching as a new construction machinery target object and creates a trajectory ID with a non - confirmed status in the trajectory set.

[0222] The reasons for the existence of the detection box with a failed second matching may be: a new construction machinery appears within the excavation block, or: there is an error in the target detection result, and an object that is not a construction machinery is wrongly identified as a construction machinery. The embodiments of the present application create a trajectory ID with a non - confirmed status for the detection box with a failed second matching and verify the authenticity of the construction machinery corresponding to the trajectory ID with a non - confirmed status based on the subsequent second matching results. The above - mentioned means can identify and eliminate the errors in the target detection results, thereby reducing the errors in the construction machinery counting results caused by the errors in the target detection results.

[0223] When the number of second matching failures of a trajectory ID in the confirmed state exceeds the first threshold, the trajectory ID is deleted. The first threshold can be determined by those skilled in the art according to actual application requirements. For example, the first threshold can be a value such as 60.

[0224] For a trajectory ID with a second matching failure, it can be determined whether it is in an unconfirmed state. If so, the trajectory ID can be directly deleted; if not, it can be determined whether the loss age of the trajectory ID is greater than the second threshold. The second threshold can be a value such as 60. If it is greater, the trajectory ID is directly deleted; if it is less, the trajectory ID is retained. Therefore, the method of the embodiment of the present application may further include: deleting a trajectory ID in the trajectory set that is in an unconfirmed state and whose second matching result is a second matching failure.

[0225] In step A5, for the construction machinery corresponding to the trajectory ID in the confirmed state, a machinery identifier is assigned. The machinery identifier uniquely identifies the construction machinery within the excavation block, and it can be a string such as 0001, 0002, etc. A construction machinery corresponds to the same machinery identifier in different frame images.

[0226] In the embodiment of the present application, for the open cut excavation operation, there is a relationship of functional complementarity between the excavation equipment and the dump truck equipment. The excavation equipment is responsible for excavating the soil, and the dump truck equipment is responsible for transporting the soil. The two cooperate closely in the construction process. Therefore, the embodiment of the present application can utilize the spatial proximity relationship between the excavation equipment and the dump truck equipment, and determine the equipment matching pairs in the above-mentioned multiple frame images arranged in time according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the above frame images.

[0227] In a specific implementation, the process of determining the equipment matching pairs in the above-mentioned multiple frame images arranged in time according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the above frame images specifically includes:

[0228] Step B1: For the excavation equipment in the frame image, determine a set of dump truck equipment; the distance between the dump truck equipment in the set of dump truck equipment and the excavation equipment does not exceed the distance threshold;

[0229] Step B2: Taking the maximum value of the distance intersection-over-union index value as the criterion, determine the target dump truck equipment that matches the excavation equipment from the set of dump truck equipment; the excavation equipment and the target dump truck equipment form an equipment matching pair.

[0230] In step B1, for each excavation equipment in the kth frame image, in order to screen out the dump truck equipment that may match it among the numerous construction machineries in the kth frame image, it is necessary to construct a set of dump truck equipment.

[0231] Embodiments of the present application can calculate the distances between an excavation device and multiple dump trucks, and record the dump trucks with distances not exceeding the distance threshold into the dump truck set. The distance threshold can be determined by those skilled in the art according to actual application requirements, and the embodiments of the present application do not limit the specific distance threshold.

[0232] In step B2, the DIOU (Distance - Intersection over Union) metric not only considers the intersection over union itself, but also incorporates the distance factor between the excavation device and the dump truck.

[0233] Assume that the frame image contains N detection frames of excavation devices and M detection frames of dump trucks. Then the calculation of the DIOU metric is shown in formula (1).

[0234] (1)

[0235] Wherein, represents the detection frame of the th ( and a positive integer) dump truck; represents the detection frame of the th ( and a positive integer) excavation device. 、 respectively represent the center points of the detection frames 、 , is the Euclidean distance between the center points 、 , is the diagonal distance of the smallest closed region that simultaneously contains the detection frame of the mth dump truck and the detection frame of the nth excavation device. is the DIOU metric between the detection frame of the mth dump truck and the detection frame of the nth excavation device.

[0236] After calculating the distance intersection over union metric values for all dump trucks in the dump truck set, compare the magnitudes of these distance intersection over union metric values, and find the dump truck with the largest distance intersection over union metric value as the target dump truck that best matches the excavation device.

[0237] After determining the device matching pairs in the jth frame image, it is possible to determine whether to reuse the device matching pairs in the jth frame image based on the position change of the dump trucks in the device matching pairs in the jth frame image.

[0238] Correspondingly, the process of determining the device matching pairs in the multiple frame images arranged in time sequence according to the device category, device identifier, and detection frame of the construction machinery in the frame image specifically includes:

[0239] Determine the first equipment matching pair in the j-th frame image according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the j-th frame image; both j and m are positive integers;

[0240] If the position change of the dump truck in the first equipment matching pair in the (j + m)-th frame image is within the set range, then use the first equipment matching pair in the j-th frame image for the (j + m)-th frame image;

[0241] If the position change of the dump truck in the first equipment matching pair in the (j + m)-th frame image is outside the set range, then determine the second equipment matching pair in the (j + m)-th frame image according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the (j + m)-th frame image.

[0242] Because in normal earthwork excavation operations, the position change of the dump truck during the loading process is not too large. If in the (j + m)-th frame image, the position change of the dump truck in the first equipment matching pair is within the set range, it indicates that the cooperation relationship between the dump truck and the excavation equipment has not changed. Therefore, the first equipment matching pair in the j-th frame image can be reused. This is based on the working characteristics of the dump truck in the construction scenario. Its main task is to receive the materials loaded by the excavation equipment at a relatively fixed position, and the position is relatively stable.

[0243] If the position change of the dump truck in the (j + m)-th frame image exceeds the set range, this means that the state of the dump truck has changed significantly. It may have left the loading position or moved to other places for other operations. At this time, the previous equipment matching pair may no longer be applicable. Therefore, it is necessary to re-determine the second equipment matching pair according to the equipment category, equipment identifier, and detection frame of the construction machinery in the (j + m)-th frame image to accurately reflect the actual cooperation relationship between the construction machinery in the current frame image.

[0244] The above method of reusing the equipment matching pair can save computing resources and improve the efficiency of the entire construction machinery motion recognition and equipment matching process. Especially when dealing with a large number of frame images, the effect is more obvious.

[0245] The position change of the dump truck in the first equipment matching pair in the (j + m)-th frame image can specifically refer to the position change of the dump truck in the first equipment matching pair in the (j + m)-th frame image relative to the dump truck in the first equipment matching pair in the j-th frame image, that is, the position change of the dump truck in the same first equipment matching pair between the (j + m)-th frame image and the j-th frame image.

[0246] In practical applications, the position change can be determined to be within the set range based on the detection boxes of a dump device in different frame images. Specifically, by comparing the center coordinates of the detection boxes of the dump device in the j-th and (j + m)-th frame images, the offset of the center coordinates of the detection box is determined. If the offset of the center coordinates is within the allowable coordinate difference range, it can be considered that the position change is within the set range. Conversely, if the offset of the center coordinates is not within the allowable coordinate difference range, it can be considered that the position change is not within the set range.

[0247] In a specific implementation, the output result format of step 201 can be: {(matchID, excavatorID, truckID)}. Where excavatorID and truckID respectively represent the ID of the excavating device and the ID of the dump device with a matching relationship. It should be noted that the excavating device ID and the dump device ID each have corresponding mechanical identifiers. In other words, the mechanical identifier of the excavating device in the device matching pair can be used as the excavating device ID, and the mechanical identifier of the dump device in the device matching pair can be used as the dump device ID.

[0248] In step 202, action recognition technology can be used to analyze the action categories of the above-mentioned multiple frame images arranged in time according to the device category, device identifier, and detection box of the construction machinery in the above-mentioned frame images, so as to obtain the action category corresponding to the above-mentioned construction machinery.

[0249] Among them, the device category can pre-limit the possible action range of the construction machinery according to the mechanical function. The above-mentioned action range can include: common excavation and slewing actions of the excavating device, general loading and unloading actions of the dump device, etc. The device identifier can accurately lock each construction machinery, improve the coherence and accuracy of action recognition, and will not confuse the action trajectories of different construction machineries. The detection box can accurately extract the position, shape, posture and other change features of the construction machinery in each frame image.

[0250] In a specific implementation, the process of step 202 for performing action recognition on the construction machinery in the above-mentioned multiple frame images arranged in time according to the device category, device identifier, and detection box of the construction machinery in the above-mentioned frame images specifically includes:

[0251] Step C1: Sample the above-mentioned multiple frame images arranged in time at a first sampling rate, and perform first convolution processing on the detection box area in the first sampling result using P first convolution modules. The P-th first convolution module outputs first convolution features; P is a positive integer greater than 1;

[0252] Step C2: Second sample the multiple frame images arranged in time according to the second sampling rate, and use P second convolutional modules to perform second convolutional processing on the detection box regions in the second sampling result. The P-th second convolutional module outputs second convolutional features. The second sampling rate is greater than the first sampling rate. The first convolutional module uses a first convolutional kernel, and the second convolutional module uses a second convolutional kernel. The number of output channels of the first convolutional kernel is greater than the number of output channels of the second convolutional kernel.

[0253] Step C3: Perform average pooling processing on the first convolutional features and the second convolutional features to obtain average pooling features.

[0254] Step C4: Determine the action category corresponding to the construction machinery according to the average pooling features.

[0255] Referring to Figure 3 , a schematic structural diagram of an action recognition model according to an embodiment of the present application is shown. The input of the action recognition model specifically includes: multiple frame images arranged in time, and the result of target tracking.

[0256] The action recognition model is used to recognize the actions of construction machinery in the above-mentioned multiple frame images arranged in time. The output result can be in the format of: {(machine class, machineID, machine actionID,xmin, xmax, ymin, ymax)}. Wherein, machine actionID represents the action category corresponding to a specific detection box (xmin, xmax, ymin, ymax), a specific machine class (equipment category), and a specific machineID (equipment identifier) in a certain frame image.

[0257] Figure 3 In

[0258] Among them, the first processing branch 301 corresponds to Step C1, and is used to first sample the multiple frame images arranged in time according to the first sampling rate, and use P first convolutional modules to perform first convolutional processing on the detection box regions in the first sampling result. The P-th first convolutional module outputs first convolutional features. P is a positive integer greater than 1. Figure 3 In

[0259] The second processing branch 302 corresponds to step C2, and is used to perform second sampling on the plurality of frame images arranged in time according to a second sampling rate, and use P second convolution modules to perform second convolution processing on the detection box regions in the second sampling result. The P-th second convolution module outputs second convolution features.

[0260] The second sampling rate is greater than the first sampling rate, which can make it pay more attention to the information with obvious differences in the time dimension. For example, the sampling rate of the second sampling module can be set to 8 times that of the first sampling module, that is, under the condition of the same input of video stream data, the number of sampled frames of the second sampling module is 8 times that of the first sampling module.

[0261] From the aspect of capturing action details, the second sampling module with a high sampling rate can obtain more frame images, which is very crucial for capturing the rapid changes and subtle actions in the actions of construction machinery. For example, the instantaneous action when the digging teeth of the excavating equipment cut into the material during excavation or the brief action when the carriage door of the dump truck opens during unloading. These details that are easily overlooked at a lower sampling rate can be well captured through a high sampling rate. These detailed action features are important for accurately identifying the action category, which can improve the accuracy of action recognition and avoid misjudging similar actions.

[0262] From the perspective of using time dimension information, a higher sampling rate can better focus on the information with obvious differences in the time dimension. The actions of construction machinery are fast and slow in the time series. Through a high sampling rate, enough frames can be obtained in a short time to analyze the change trend of the action. This helps to obtain more accurate action features during the rapid change stage of the action (such as the rapid unloading of the dump truck and the rapid rotation of the excavating equipment), so as to more accurately judge the action category of the construction machinery as a whole, especially for those situations with a short action cycle and a large change amplitude of the action, which can effectively improve the accuracy and integrity of recognition.

[0263] The first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel. In a convolutional neural network, the number of output channels of the convolution kernel can be simply understood as the number of feature maps generated after the convolution operation.

[0264] The first convolutional kernel with a large number of channels can capture the relatively macroscopic and overall features of the construction machinery actions as comprehensively as possible under a relatively low sampling rate. Since the frame interval is relatively large under a low sampling rate, it is more necessary to comprehensively consider the action situation from multiple dimensions. More channels can map the actions from different perspectives, such as the overall moving direction of the excavating arm of the excavating equipment, the approximate angle change range, and the change in the relative position in the picture. Integrating these can form coarse-grained features, which is convenient for initially forming a general judgment framework for the actions.

[0265] The second convolutional kernel with a small number of channels focuses on extracting the key details of the construction machinery actions when processing a large number of fine-grained frame images under a high sampling rate, such as the slight position change at the moment when the excavating teeth of the excavating equipment cut into the material, and the very precise change in a local angle when the carriage of the self-unloading equipment discharges materials. It simply and pertinently extracts fine-grained features, which can avoid interference from redundant information and also meet the requirement of quickly capturing details under a high sampling rate.

[0266] The average pooling module 303 is used to perform average pooling processing on the first convolutional feature and the second convolutional feature to obtain an average pooling feature.

[0267] Average pooling is a downsampling operation. The first convolutional feature and the second convolutional feature may contain a lot of information at different positions. The average pooling module calculates the average value of these features within a certain area (such as a small pixel area). This can reduce the amount of data while retaining the main feature information. For example, by averaging the convolutional features corresponding to multiple pixel points within a local area, a more representative value is obtained, avoiding overfitting and reducing the dimension of the data, making subsequent processing more efficient.

[0268] The fully connected module 304 is used to determine the action category corresponding to the construction machinery according to the average pooling feature.

[0269] Each neuron in the fully connected module is connected to all neurons in the previous layer (average pooling layer). These connections have corresponding weights. By multiplying the average pooling feature by these weights and summing them, and then passing through the activation function, the action category corresponding to the construction machinery can be judged according to the learned pattern. For example, if the value output after these operations on the average pooling feature falls within the numerical range corresponding to the pattern of the excavating action, it is determined as the excavating action.

[0270] In an alternative embodiment of the present application, the process of performing action recognition on construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the frame image may further include: fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the (i + 1)-th first convolution module.

[0271] The above fusion can better combine the detailed features at a high sampling rate included in the second convolution result and the macroscopic features at a low sampling rate included in the first convolution result, enabling each stage of convolution processing to make full use of both convolution results. Moreover, the above fusion can enable the features subsequently output by the first convolution module to incorporate key information at different sampling rates, thereby improving the accuracy of action recognition for construction machinery.

[0272] In a specific implementation, the process of fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module may specifically include:

[0273] Step D1: Sampling the second convolution result according to the time stride of the first convolution result to obtain a third sampling result;

[0274] Step D2: Reconstructing and transposing the third sampling result according to the matrix structure of the first convolution result, so that the matrix structure of the obtained transposed result matches the matrix structure of the first convolution;

[0275] Step D3: Performing time-stride convolution on the transposed result and the first convolution result to obtain a stride convolution result;

[0276] Step D4: Fusing the stride convolution result and the first convolution result.

[0277] In Step D1, due to different sampling rates, the time strides (which can be understood as the intervals between data points) of the first convolution result and the second convolution result in the time dimension are different. Step D1 samples the second convolution result according to the time stride of the first convolution result, aiming to initially align the two in the time dimension.

[0278] Although Step D1 adjusts the time stride, the matrix structure (such as dimensions, shape, etc.) of the second convolution result may still not match that of the first convolution result. Step D2 reconstructs and transposes the third sampling result according to the matrix structure of the first convolution result. The reconstruction and transposition operation in Step D2 is used to adjust the third sampling result to the same matrix form as the first convolution result, enabling the two to match in matrix structure and providing a data basis with consistent structure for subsequent fusion operations.

[0279] After the matrix structures of the two results are matched in step D2, in step D3, a temporal stride convolution is performed on the transposed result (i.e., the third sampling result after being processed in step D2) and the first convolution result. The purpose of this operation is to further fuse the transposed result and the first convolution result in the temporal dimension and the spatial dimension (matrix structure). Temporal stride convolution can extract the joint features of the transposed result and the first convolution result in time and space, and find more complex correlation relationships between them. For example, it can mine the correlation between the action details of the excavation equipment at different time and spatial positions and the overall action trend. Such correlation features are very important for accurately identifying the action categories of construction machinery.

[0280] In step D4, the stride convolution result and the first convolution result are fused. The fused result will include: feature information obtained from different sampling rates (the first sampling rate corresponding to the first convolution module and the second sampling rate corresponding to the second convolution module). These feature information have both macroscopic overall action features and microscopic detailed action features. Through such fusion, it can provide a more comprehensive and richer feature input for subsequent action recognition, thereby improving the accuracy of judging the action categories of construction machinery and avoiding misjudgment caused by relying only on the features at a single sampling rate.

[0281] In step 203, each time the excavation equipment in the equipment matching pair completes the action categories in a specific order, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the amount of each excavation and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can accurately determine the actual occurrence of each effective excavation by combining the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair, and thus determine the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0282] In one implementation, the preset change in the action category of the excavation equipment in the equipment matching pair specifically includes: the action categories of the excavation equipment in the equipment matching pair change in the order of excavation, lift and swing, unloading, return of empty bucket, and excavation in sequence.

[0283] In a construction scenario, the order of the action categories of the excavation equipment is a complete working cycle. First is excavation, where the excavation equipment controls the bucket to insert into the material pile and fill it with materials. Then is lift and swing, where the mechanical arm raises the bucket filled with materials and rotates the body of the machine to align the bucket with the position of the dump truck. Next is unloading, where the materials in the bucket are poured into the dump truck. After that is the return of the empty bucket, where the empty bucket is placed back to its original position or the preparation position for the next excavation. Finally, excavation starts again, and so on. This order fully demonstrates the process of each action category of the excavation equipment working in sequence when cooperating with the dump truck.

[0284] When the excavation equipment leaves the factory, a parameter setting of the rated excavation capacity will be provided. This parameter represents the maximum volume of materials that the excavation equipment can theoretically excavate during each normal excavation. During each actual effective excavation process, there is a fixed corresponding relationship between the excavation volume of the excavation equipment and this rated excavation capacity. Therefore, in the embodiments of the present application, the real-time cumulative excavation volume corresponding to the excavation block can be determined by accumulating the rated excavation capacity.

[0285] In practical applications, the process of determining the rated excavation capacity specifically includes:

[0286] Step E1: Use the excavation equipment model classification model to determine the target model corresponding to the excavation equipment with a preset change in the action category;

[0287] Step E2: According to the target model, search in the mapping relationship between the search model and the rated excavation capacity to determine the rated excavation capacity corresponding to the excavation equipment with a preset change in the action category.

[0288] The excavation equipment model classification model in the embodiments of the present application can have the classification ability of the excavation equipment model. In other words, it can determine the target model corresponding to the excavation equipment in the local image according to the local image corresponding to the excavation equipment with a preset change in the action category.

[0289] In a specific implementation, the embodiments of the present application can label the model categories for the excavation equipment image dataset and train the excavation equipment model classification model according to the excavation equipment image dataset.

[0290] Here, a process for obtaining the excavation equipment image dataset is provided. Specifically, based on the video monitoring system, excavation equipment images covering different construction scene backgrounds, different lighting conditions, different working postures, etc. can be collected. In addition, combined images of the excavation equipment and the dump truck during the excavation operation need to be collected to reflect the actual situation of mutual occlusion between the two types of construction machinery. Therefore, the excavation equipment image dataset can include: excavation equipment images and combined images.

[0291] The embodiments of the present application do not limit the specific structure of the excavation equipment model classification model. For example, the excavation equipment model classification model can include structures such as VGG (Visual Geometry Group) convolutional neural networks.

[0292] In the embodiments of the present application, the mapping relationship between the model and the rated excavation capacity can be pre-saved. In this way, the rated excavation capacity corresponding to the excavation equipment with a preset change in the action category can be found in the above mapping relationship according to the target model corresponding to the excavation equipment with a preset change in the action category. The above mapping relationship can be determined according to information sources such as the technical parameter manual or the equipment nameplate of the excavation equipment.

[0293] In the embodiments of the present application, when it is monitored that the excavation operation state of the current target area is updated from the loading state to the non-excavated state or the excavating state, it can be considered that the current target area has completed the current loading, and the excavation volume corresponding to the current loading of the current target area can be determined according to the rated excavation capacity corresponding to the dump truck equipment in the current target area. The excavation volume corresponding to the current loading can be equal to the rated excavation capacity corresponding to the dump truck equipment in the current target area.

[0294] In the case where a preset change occurs in the action category of the excavation equipment in the equipment matching pair, it is considered that the excavation equipment in the equipment matching pair has completed one excavation. The process of determining the real-time cumulative excavation volume corresponding to the excavation block according to the rated excavation capacity corresponding to the excavation equipment specifically includes:

[0295] Step F1: Determine the effective excavation times according to the number of times of preset changes in the action category of the excavation equipment in the equipment matching pair. For example, when the actions of the excavation equipment complete a full cycle in the order of excavation, lifting and slewing, unloading, returning with an empty bucket, and excavation, it is counted as one effective excavation. By real-time monitoring of video frame images or the construction process, the number of times of such full cycles is counted.

[0296] Step F2: Obtain the rated excavation capacity corresponding to the excavation equipment with a preset change in the action category.

[0297] In the embodiments of the present application, the target model of the excavation equipment can be determined first by using the excavation equipment model classification model; then, according to the target model, the rated excavation capacity corresponding to the excavation equipment with a preset change in the action category can be found in the mapping relationship between the model and the rated excavation capacity.

[0298] Step F3: Determine the real-time cumulative excavation volume corresponding to the excavation block according to the effective excavation times and the rated excavation capacity.

[0299] Previously, the target tracking technology has been used to assign device identifiers to the excavation equipment in the frame images of the excavation blocks. Steps F1 and F2 can determine the target device identifier corresponding to the excavation equipment with a preset change in the action category, as well as the effective excavation times and rated excavation capacity corresponding to the target device identifier. On this basis, the real-time cumulative excavation volume corresponding to the target device identifier can be determined according to the effective excavation times and rated excavation capacity corresponding to the target device identifier. Moreover, the real-time cumulative excavation volumes corresponding to all target device identifiers can be accumulated to obtain the real-time cumulative excavation volume corresponding to the entire excavation block.

[0300] In an example of the present application, at a set start time, the real-time cumulative excavation volume corresponding to the excavation block can be set to 0. Subsequently, according to the update of the effective excavation times of any target device identifier, the real-time cumulative excavation volume can be updated to obtain an ever-updated real-time cumulative excavation volume. Therefore, the process of real-time monitoring and analysis of the excavation volume during the construction period can be a process of continuously updating the real-time cumulative excavation volume according to the update of the effective excavation times of any target device identifier. The set start time can be determined by those skilled in the art according to actual application requirements. For example, the set start time can be the start time of the excavation operation of the excavation block, or the set start time can be the start time of a natural day.

[0301] In practical applications, if there are multiple excavation blocks in a hydropower project, the real-time cumulative excavation volume corresponding to each individual excavation block can be monitored separately according to natural days. After the end of a natural day, the real-time cumulative excavation volumes corresponding to multiple excavation blocks can be fused. It can be understood that the embodiments of the present application do not limit the specific fusion method.

[0302] For the process of determining the second excavation volume of the excavation area within a preset time period based on the video stream data of the excavation area within the preset time period, since it is similar to the process of determining the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block, it will not be elaborated here and can be referred to each other.

[0303] In an example, the start time and end time of the preset time period can be determined, and the data segments within the preset time period can be screened out from the recorded data of the real-time cumulative excavation volume of the excavation area. The real-time cumulative excavation volumes of all excavation blocks in the excavation area can be summarized to obtain the recorded data of the real-time cumulative excavation volume of the excavation area.

[0304] If the real-time cumulative excavation volume is recorded on a natural day basis (starting from zero for each natural day to record the real-time cumulative excavation volume of that day), the data segments within this time period are integrated and calculated. For example, if the preset time period is from the 3rd day to the 5th day, first obtain the real-time cumulative excavation volumes of the 3rd, 4th, and 5th natural days respectively, and then sum up the real-time cumulative excavation volumes of the 3rd, 4th, and 5th natural days. The sum result is the second excavation volume within this preset time period.

[0305] If the real-time cumulative excavation volume is recorded in the form of cumulative addition at time intervals (setting the real-time cumulative excavation volume to zero at the start time of the excavation operation for each excavation block), it can be obtained by calculating the difference in cumulative values for the corresponding time intervals within this time period. For example, given that the real-time cumulative excavation volume is recorded cumulatively from the start of the project, if the preset time period is from the 3rd day to the 5th day, first obtain the real-time cumulative excavation volume at the end of the 5th day, and then subtract the real-time cumulative excavation volume at the end of the 2nd day. The resulting difference is the second excavation volume within this preset time period.

[0306] In summary, the embodiments of the present application provide a real-time monitoring and analysis process for the excavation volume during the construction period executed by a computer. This process first obtains the video stream data of the excavation block, and the video stream data specifically includes: a plurality of frame images arranged in time sequence; then, performs object detection on the above frame images to obtain the equipment categories (dump equipment and excavation equipment) corresponding to the construction machinery and the detection frames; next, tracks the construction machinery in the plurality of frame images based on the equipment categories and detection frames to obtain the equipment identifiers corresponding to the construction machinery; after that, determines the equipment matching pairs according to the equipment categories, equipment identifiers, and detection frames. The above equipment matching pairs specifically represent an excavation equipment and a target dump equipment that matches it; then, according to the equipment categories, equipment identifiers, and detection frames, performs action recognition on the construction machinery to obtain the corresponding action categories; furthermore, in the case where the action category of the excavation equipment in the equipment matching pair undergoes a preset change, determines the real-time cumulative excavation volume corresponding to the excavation block according to the rated excavation capacity of the excavation equipment.

[0307] The above process of the embodiments of the present application performs real-time monitoring of the excavation volume during the construction period according to the objective law of the operation cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered to have completed an effective excavation, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiments of the present application can accurately determine the actual occurrence of each effective excavation by combining the monitoring result that the action category of the excavation equipment in the equipment matching pair undergoes a preset change, and thus determine the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0308] Since the above objective laws of the embodiments of the present application can be based on the internal logical relationship between the actual process of the excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters, the real-time monitoring of the excavation volume during the construction period can be realized. Therefore, the embodiments of the present application can, based on the analysis of the video stream data, achieve the real-time monitoring of the excavation volume during the construction period at a relatively low cost and at a relatively fast speed. In other words, the embodiments of the present application can save the labor cost, data acquisition cost, and data processing time cost consumed by using the unmanned aerial vehicle oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0309] Moreover, since the above objective laws of the embodiments of the present application do not involve subjective speculation and complex manual measurement and estimation, the embodiments of the present application can effectively improve the accuracy of the real-time calculation of the excavation volume during the construction period.

[0310] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be carried out in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present application.

[0311] Based on the above embodiments, the present embodiment further provides a device for calculating the whole process of the excavation volume during the construction period based on multi-source data fusion. Referring to Figure 4 , the device may specifically include: an excavated model determination module 401, a first excavation volume determination module 402, a second excavation volume determination module 403, a correction coefficient determination module 404, a real-time excavation volume determination module 405, and a correction module 406.

[0312] Among them, the excavated model determination module 401 is used to determine the three-dimensional model of the excavated completed part of the excavation area according to the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the excavation area includes: at least one excavation block;

[0313] The first excavation volume determination module 402 is used to determine the first excavation volume of the excavation area within a preset time period according to the three-dimensional model of the excavated completed part of the excavation area; the first moment is the end moment of the preset time period;

[0314] The second excavation volume determination module 403 is used to determine the second excavation volume of the excavation area within a preset time period according to the video stream data of the excavation area within the preset time period;

[0315] A correction coefficient determination module 404, configured to determine a correction coefficient according to a first excavation volume and a second excavation volume;

[0316] A real-time excavation volume determination module 405, configured to determine a real-time cumulative excavation volume of an excavation block according to video stream data of the excavation block;

[0317] A correction module 406, configured to correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain a measurement result of the excavation volume of the excavation block;

[0318] Wherein, the video stream data includes: a plurality of frame images arranged in time sequence;

[0319] The real-time excavation volume determination module includes:

[0320] An equipment matching module, configured to determine an equipment matching pair in the plurality of frame images arranged in time sequence according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the frame images; the equipment matching pair includes: an excavation equipment and a target self-unloading equipment matched with it;

[0321] An action recognition module, configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time sequence according to the equipment category, equipment identifier, and detection frame corresponding to the construction machinery in the frame images to obtain an action category corresponding to the construction machinery;

[0322] An excavation volume measurement module, configured to, when a preset change occurs in the action category of the excavation equipment in the equipment matching pair, consider that the excavation equipment in the equipment matching pair has completed one excavation, and determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavation equipment.

[0323] Optionally, the excavated model determination module includes:

[0324] A grid interpolation module, configured to perform grid interpolation on the point cloud data of the excavation part at the first moment to obtain an actual excavation contour surface of the excavation part at the first moment;

[0325] A first envelope determination module, configured to determine a first envelope of the excavation part at the first moment according to the actual excavation contour surface of the excavation part at the first moment;

[0326] A first Boolean operation module, configured to perform a Boolean subtraction operation on the first envelope of the excavation part at the first moment and the terrain surface model of the excavation part before excavation to obtain an operation result;

[0327] A model determination module, configured to obtain a three-dimensional model of the excavated completed part of the excavation part from the operation result according to the spatial range of the excavation part.

[0328] Optionally, the first excavation volume determination module includes:

[0329] A mesh division module for dividing the three-dimensional model of the completed excavated part of the excavation site into mesh cells;

[0330] A unit volume determination module for determining the unit volume of the mesh cells in the three-dimensional model of the completed excavated part of the excavation site;

[0331] A unit volume summary module for summarizing the unit volumes of all the mesh cells in the three-dimensional model of the completed excavated part of the excavation site to obtain the first excavation volume of the excavation site within a preset time period.

[0332] Optionally, the second excavation volume determination module includes:

[0333] An image analysis module for analyzing the first frame image in the video stream data within a preset time period in the order from front to back in time to obtain the first device matching pair in the first frame image and the action category corresponding to the construction machinery in the first frame image;

[0334] A second excavation volume measurement module for, in the case where a preset change occurs in the action category of the excavation equipment in the first device matching pair, considering that the excavation equipment in the first device matching pair has completed one excavation, and determining the second excavation volume of the excavation site within a preset time period according to the rated excavation capacity corresponding to the excavation equipment.

[0335] Optionally, the device further includes:

[0336] A terrain surface model establishment module for establishing a terrain surface model of the excavation site before excavation according to the design information of the engineering object corresponding to the excavation site and using the contour lines and elevation points in the original terrain.

[0337] Optionally, the device further includes:

[0338] An over-excavation and under-excavation analysis module for determining the over-excavation and under-excavation conditions of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the designed volume of the first excavation block;

[0339] An update module for adjusting the remaining excavation volume of the second excavation block in the (s + 1)-th excavation area according to the over-excavation and under-excavation conditions of the first excavation block; s is a positive integer; the s-th excavation area and the (s + 1)-th excavation area are two geographically adjacent excavation areas.

[0340] Optionally, the process of determining the designed volume of the excavation block includes:

[0341] Establishing an excavation surface model of the excavation site after excavation according to the excavation contour line of the excavation site;

[0342] Perform a Boolean operation on the second envelope corresponding to the terrain surface model and the third envelope corresponding to the excavation surface model to obtain a 3D model of the excavation area;

[0343] Obtain the 3D model of the excavation block from the 3D model of the excavation area according to the spatial range of the excavation block;

[0344] Determine the designed volume of the excavation block according to the 3D model of the excavation block.

[0345] Optionally, the preset change in the action category of the excavation equipment in the equipment matching pair includes: the action category of the excavation equipment in the equipment matching pair changes in the order of excavation, lifting and slewing, unloading, empty bucket return, and excavation in sequence.

[0346] Optionally, the action recognition of the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the frame image includes:

[0347] Perform a first sampling on the multiple frame images arranged in time according to the first sampling rate, and use P first convolution modules to perform a first convolution process on the detection frame area in the first sampling result. The Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1;

[0348] Perform a second sampling on the multiple frame images arranged in time according to the second sampling rate, and use P second convolution modules to perform a second convolution process on the detection frame area in the second sampling result. The Pth second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel;

[0349] Perform an average pooling process on the first convolution feature and the second convolution feature to obtain an average pooling feature;

[0350] Determine the action category corresponding to the construction machinery according to the average pooling feature.

[0351] Optionally, the action recognition of the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the frame image further includes:

[0352] Fuse the second convolution result output by the ith second convolution module and the first convolution result output by the ith first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the (i + 1)th first convolution module.

[0353] The embodiments of the present application also provide a non - volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can be caused to execute the instructions of the method steps in the embodiments of the present application.

[0354] The embodiments of the present application provide one or more machine - readable media, on which instructions are stored. When executed by one or more processors, the electronic device is caused to execute one or more of the methods as described in the above embodiments. In the embodiments of the present application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0355] The embodiments of the present application provide a computer program product containing instructions. When it runs on a computer, the computer is caused to execute any one of the methods for the whole - process measurement of the excavation volume during the construction period based on multi - source data fusion as described in the above embodiments.

[0356] The embodiments of the present disclosure can be implemented as a device configured as desired using any suitable hardware, firmware, software, or any combination thereof. The device may include: electronic devices such as terminal devices and servers (clusters). Figure 5 Exemplary device 1300 that can be used to implement the various embodiments described in the present application is schematically shown.

[0357] For one embodiment, Figure 5 Exemplary device 1300 is shown, which has one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, an NVP (non - volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0358] Processor 1302 may include one or more single - core or multi - core processors. Processor 1302 may include any combination of general - purpose processors or special - purpose processors (such as graphics processors, application processors, base - band processors, etc.). In some embodiments, device 1300 can act as the terminal device, server (cluster), etc. described in the embodiments of the present application.

[0359] In some embodiments, device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314, and one or more processors 1302 coupled to the one or more computer-readable media and configured to execute the instructions 1314 to implement modules to perform the actions described in this disclosure.

[0360] For one embodiment, control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of (one or more) processors 1302 and / or any suitable device or component in communication with control module 1304.

[0361] Control module 1304 may include a memory controller module to provide an interface to memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0362] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. For one embodiment, memory 1306 may include any suitable volatile memory, e.g., suitable DRAP (Dynamic Random Access Memory). In some embodiments, memory 1306 may include double data rate type four synchronous dynamic random access memory.

[0363] For one embodiment, control module 1304 may include one or more input / output controllers to provide an interface to non-volatile memory / storage device 1308 and (one or more) input / output devices 1310.

[0364] For example, non-volatile memory / storage device 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile disk drives).

[0365] Non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which device 1300 is mounted, or it may be accessible by the device without being part of the device. For example, non-volatile memory / storage device 1308 may be accessed via network through (one or more) input / output devices 1310.

[0366] (One or more) Input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable devices. The input / output devices 1310 may include communication components, audio components, sensor components, etc. The network interface 1312 may provide an interface for device 1300 to communicate through one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2-Generation wireless telephone technology), 3G (3-Generation wireless telephone technology), 4G (4-Generation wireless telephone technology), 5G (5-Generation wireless telephone technology), etc., or a combination thereof for wireless communication.

[0367] For one embodiment, at least one of (one or more) processors 1302 may be logically encapsulated with one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of (one or more) processors 1302 may be logically encapsulated with one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of (one or more) processors 1302 may be logically integrated with one or more controllers of the control module 1304 on the same die. For one embodiment, at least one of (one or more) processors 1302 may be logically integrated with one or more controllers of the control module 1304 on the same die to form a system-on-chip.

[0368] In various embodiments, device 1300 may be, but is not limited to: a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touchscreen device, a netbook, etc.) and other terminal devices. In various embodiments, device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit, and speakers.

[0369] Among them, a main control chip can be used as a processor or a control module in the detection device. Sensor data, location information, etc. are stored in a memory or a non-volatile memory / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0370] For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the corresponding descriptions in the method embodiments.

[0371] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0372] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0373] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0374] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0375] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0376] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the said element.

[0377] The above has introduced in detail a method and device for the whole-process measurement of excavation volume during the construction period based on multi-source data fusion, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scenarios. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for calculating the whole process of excavation volume during the construction period based on multi-source data fusion, characterized in that, The method includes: Determining a three-dimensional model of the already-excavated completed part of the excavation area according to the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the excavation area includes at least one excavation block; Determining the first excavation volume of the excavation area within a preset time period according to the three-dimensional model of the already-excavated completed part of the excavation area; the first moment is the end moment of the preset time period; Determining the second excavation volume of the excavation area within a preset time period according to the video stream data of the excavation area within the preset time period; Determining a correction coefficient according to the first excavation volume and the second excavation volume; Determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block; Correcting the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the measurement result of the excavation volume of the excavation block; Wherein, the video stream data includes a plurality of frame images arranged in time sequence; The determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block includes: Determining equipment matching pairs in the plurality of frame images arranged in time sequence according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include an excavation equipment and a target dump truck equipment matched with it; Performing action recognition on the construction machinery in the plurality of frame images arranged in time sequence according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images to obtain the action category corresponding to the construction machinery; In the case that the action category of the excavation equipment in the equipment matching pair undergoes a preset change, it is considered that the excavation equipment in the equipment matching pair has completed one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the excavation equipment.

2. The method according to claim 1, wherein The determining the three-dimensional model of the already-excavated completed part of the excavation area according to the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation includes: Performing grid interpolation on the point cloud data of the excavation area at the first moment to obtain the actual excavation contour surface of the excavation area at the first moment; Determining the first envelope of the excavation area at the first moment according to the actual excavation contour surface of the excavation area at the first moment; Performing a Boolean difference operation on the first envelope of the excavation area at the first moment and the terrain surface model of the excavation area before excavation to obtain an operation result; Obtaining the three-dimensional model of the already-excavated completed part of the excavation area from the operation result according to the spatial range of the excavation area.

3. The method according to claim 1, characterized in that, The determining the first excavation volume of the excavation area within a preset time period according to the three-dimensional model of the already-excavated completed part of the excavation area includes: Dividing the three-dimensional model of the already-excavated completed part of the excavation area into grid cells; Determining the cell volume of the grid cells in the three-dimensional model of the already-excavated completed part of the excavation area; Summarizing the cell volumes of all grid cells in the three-dimensional model of the already-excavated completed part of the excavation area to obtain the first excavation volume of the excavation block within the preset time period.

4. The method according to claim 1, characterized in that, The determining the second excavation volume of the excavation area within a preset time period according to the video stream data of the excavation area within the preset time period includes: Analyze the first frame image in the video stream data within a preset time period in chronological order from front to back to obtain the first device matching pair in the first frame image and the action category corresponding to the construction machinery in the first frame image; When a preset change occurs in the action category of the device in the first device matching pair, it is considered that the excavation device in the first device matching pair has completed one excavation, and according to the rated excavation capacity corresponding to the excavation device, determine the second excavation volume of the excavation site within the preset time period.

5. The method according to claim 1, characterized in that, The method further includes: According to the design information of the engineering object corresponding to the excavation site, use the contour lines and elevation points in the original terrain to establish a terrain surface model of the excavation site before excavation.

6. The method according to claim 1, characterized in that The method further includes: Determine the over-excavation and under-excavation conditions of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the designed volume of the first excavation block; Adjust the remaining excavation volume of the second excavation block in the (s + 1)-th excavation area according to the over-excavation and under-excavation conditions of the first excavation block; s is a positive integer; the s-th excavation area and the (s + 1)-th excavation area are two adjacent excavation areas in terms of geographical location.

7. The method according to any one of claims 1 to 6, characterized in that, The process of determining the designed volume of the excavation block includes: According to the excavation contour line of the excavation site, establish an excavation surface model of the excavation site after excavation; Perform a Boolean operation on the second envelope corresponding to the terrain surface model and the third envelope corresponding to the excavation surface model to obtain a three-dimensional model of the excavation site; According to the spatial range of the excavation block, obtain the three-dimensional model of the excavation block from the three-dimensional model of the excavation site; Determine the designed volume of the excavation block according to the three-dimensional model of the excavation block.

8. The method according to any one of claims 1 to 6, characterized in that, The preset change in the action category of the excavation device in the device matching pair includes: the action category of the excavation device in the device matching pair changes in sequence according to the order of excavation, lifting and slewing, unloading, empty bucket return, and excavation.

9. The method according to any one of claims 1 to 6, characterized in that, The action recognition of the construction machinery in the multiple frame images arranged in time according to the device category, device identifier, and detection frame corresponding to the construction machinery in the frame image includes: Perform a first sampling on the multiple frame images arranged in time according to the first sampling rate, and use P first convolutional modules to perform a first convolutional process on the detection frame area in the first sampling result. The P-th first convolutional module outputs a first convolutional feature; P is a positive integer greater than 1; Perform a second sampling on the multiple frame images arranged in time according to the second sampling rate, and use P second convolutional modules to perform a second convolutional process on the detection frame area in the second sampling result. The P-th second convolutional module outputs a second convolutional feature; the second sampling rate is greater than the first sampling rate; the first convolutional module uses a first convolutional kernel, the second convolutional module uses a second convolutional kernel, and the number of output channels of the first convolutional kernel is greater than the number of output channels of the second convolutional kernel; Perform an average pooling process on the first convolutional feature and the second convolutional feature to obtain an average pooling feature; Determine the action category corresponding to the construction machinery according to the average pooling feature.

10. The method according to claim 9, wherein Performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the frame images further includes: Fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the (i + 1)-th first convolution module.

11. A device for calculating the excavation volume during the construction period based on multi-source data fusion, characterized in that, The device includes: An excavated model determination module, configured to determine a three-dimensional model of the excavated completed part of the excavation area according to the point cloud data of the excavation area at the first moment and the terrain surface model of the excavation area before excavation; the excavation area includes at least one excavation block; A first excavation volume determination module, configured to determine the first excavation volume of the excavation area within a preset time period according to the three-dimensional model of the excavated completed part of the excavation area; the first moment is the end moment of the preset time period; A second excavation volume determination module, configured to determine the second excavation volume of the excavation area within a preset time period according to the video stream data of the excavation area within the preset time period; A correction coefficient determination module, configured to determine a correction coefficient according to the first excavation volume and the second excavation volume; A real-time excavation volume determination module, configured to determine the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block; A correction module, configured to correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain a measurement result of the excavation volume of the excavation block; Wherein, the video stream data includes multiple frame images arranged in time; The real-time excavation volume determination module includes: An equipment matching module, configured to determine an equipment matching pair in the multiple frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the frame images; the equipment matching pair includes a digging equipment and a target dump truck matching it; An action recognition module, configured to perform action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification, and detection frame corresponding to the construction machinery in the frame images to obtain the action category corresponding to the construction machinery; An excavation volume measurement module, configured to, when the action category of the digging equipment in the equipment matching pair changes presetly, consider that the digging equipment in the equipment matching pair has completed one excavation, and determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the digging equipment.

12. The device according to claim 11, wherein The excavated model determination module includes: A grid interpolation module, configured to perform grid interpolation on the point cloud data of the excavation area at the first moment to obtain the actual excavation contour surface of the excavation area at the first moment; A first envelope determination module, configured to determine the first envelope of the excavation area at the first moment according to the actual excavation contour surface of the excavation area at the first moment; A first Boolean operation module, configured to perform a Boolean subtraction operation on the first envelope of the excavation area at the first moment and the terrain surface model of the excavation area before excavation to obtain an operation result; A model determination module, configured to obtain a three-dimensional model of the excavated completed part of the excavation area from the operation result according to the spatial range of the excavation area.

13. The device according to claim 11, characterized in that, The first excavation volume determination module includes: A grid division module for dividing the 3D model of the excavated part of the excavation area into grid cells; A cell volume determination module for determining the cell volume of the grid cells in the 3D model of the excavated part of the excavation area; A cell volume summation module for summing up the cell volumes of all the grid cells in the 3D model of the excavated part of the excavation area to obtain the first excavation volume of the excavation area within a preset time period.

14. The device according to claim 11, characterized in that, The second excavation volume determination module includes: An image analysis module for analyzing the first frame image in the video stream data within a preset time period in the order from front to back in time to obtain the first device matching pair in the first frame image and the action category corresponding to the construction machinery in the first frame image; A second excavation volume measurement module for, when a preset change occurs in the action category of the excavation equipment in the first device matching pair, considering that the excavation equipment in the first device matching pair has completed one excavation, and determining the second excavation volume of the excavation area within a preset time period according to the rated excavation capacity corresponding to the excavation equipment.

15. The device according to claim 11, characterized in that, The device further includes: An over-excavation and under-excavation analysis module for determining the over-excavation and under-excavation conditions of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the designed volume of the first excavation block; An update module for adjusting the remaining excavation volume of the second excavation block in the (s + 1)-th excavation area according to the over-excavation and under-excavation conditions of the first excavation block; s is a positive integer; the s-th excavation area and the (s + 1)-th excavation area are two geographically adjacent excavation areas.

16. An electronic device, characterized in that, It includes: A processor; And A memory storing executable code, which when executed, causes the processor to execute the method according to any one of claims 1-10.

17. A machine-readable medium storing executable code, which when executed, causes a processor to execute the method according to any one of claims 1-10.

18. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the method according to any one of claims 1-10 is implemented.

Images