Multi-source data fusion-based whole-process measurement and calculation method for excavation volume in construction period

Abstract

The embodiment of the invention provides a multi-source data fusion-based construction period excavation volume whole-process measurement and calculation method, which specifically comprises the following steps of: determining a three-dimensional model of an excavated part of an excavated part according to point cloud data of the excavated part at a first moment and a terrain curved surface model of the excavated part before excavation; according to the three-dimensional model of the excavated part of the excavated part, the first excavated volume of the excavated part in the preset time period is determined; the first moment is a termination moment of the preset time period; determining a second excavation volume of the excavation part in the preset time period according to the video stream data of the excavation part in the preset time period; determining a correction coefficient according to the first excavation volume and the second excavation volume; according to the video stream data of the excavation blocks, the real-time accumulative excavation volume of the excavation blocks is determined; and correcting the real-time accumulative excavation volume of the excavation blocks according to the correction coefficient. According to the embodiment of the invention, the requirement of engineering construction on real-time measurement and calculation of the whole process of the excavation volume in the construction period can be met.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of open excavation of earth and stone, and in particular to a method for calculating the entire process of excavation volume during construction based on multi-source data fusion. Background Art

[0002] Open excavation of earth and stone is one of the common types of construction operations. The calculation of excavation volume is crucial for closely following the progress of on-site construction. It can provide strong support for the planning of excavation construction plans and the timely deployment of the number of construction machinery.

[0003] A related technology is to calculate the excavation volume during the design period based on 3D design software. Specifically, first, the topographic parameters and design planning parameters are obtained according to the engineering design drawings; then, the excavation area is divided into units using the topographic parameters and design planning parameters, and the unit volume is calculated; then, all unit volumes are accumulated to obtain the total excavation volume of the entire excavation area.

[0004] Another related technology is to use drone oblique photography to calculate the amount of excavation during construction. Specifically, before excavation, drone oblique photography is used to obtain digital elevation model data to record the original terrain conditions; after excavation, the same technology is used again to obtain the corresponding digital elevation model data; by comparing the data before and after excavation, the volume corresponding to the difference between the two is calculated, and this volume is the excavation amount, thereby realizing the calculation of the excavation amount.

[0005] However, whether it is based on 3D design software to calculate the excavation volume during the design period, or based on drone oblique photography to achieve phased excavation volume calculation during the construction period, both have the problem of poor real-time performance, and it is difficult to meet the demand for real-time calculation of the excavation volume during the entire construction period in engineering construction. Summary of the invention

[0006] The embodiment of the present application provides a method for calculating the excavation volume during the entire construction period based on multi-source data fusion, which can significantly improve the accuracy of excavation volume calculation while following up the construction progress in real time, thereby meeting the needs of engineering construction for real-time calculation of the excavation volume during the entire construction period.

[0007] Correspondingly, the embodiment of the present application also provides a device for measuring the entire process of excavation volume during construction based on multi-source data fusion, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0008] In order to solve the above problems, the embodiment of the present application discloses a method for calculating the whole process of excavation volume during construction period based on multi-source data fusion, including: Determine a three-dimensional model of the excavated portion of the excavation site based on the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation; the excavation site includes: at least one excavation block; Determining a first excavation amount of the excavation site within a preset time period according to the three-dimensional model of the excavated portion of the excavation site; the first moment is the end moment of the preset time period; Determining a second excavation amount of the excavation portion within the preset time period according to the video stream data of the excavation portion within the preset time period; Determining a correction coefficient according to the first excavation amount and the second excavation amount; Determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block; According to the correction coefficient, the real-time cumulative excavation volume of the excavation block is corrected to obtain the excavation volume measurement result of the excavation block; Wherein, the video stream data includes: a plurality of frame images arranged in time; Determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block includes: According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating device in the equipment matching pair, it is considered that the excavating device in the equipment matching pair completes one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the excavating device.

[0009] The present application also discloses a device for calculating the whole process of excavation volume during construction based on multi-source data fusion, the device comprising: The excavated model determination module is used to determine the three-dimensional model of the excavated part of the excavation site based on the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation; the excavation site includes: at least one excavation block; A first excavation amount determination module is used to determine a first excavation amount of the excavation site within a preset time period according to a three-dimensional model of the excavated portion of the excavation site; the first moment is the end moment of the preset time period; A second excavation amount determination module, used to determine a second excavation amount of the excavation part within a preset time period according to the video stream data of the excavation part within a preset time period; A correction coefficient determination module, used to determine the correction coefficient according to the first excavation amount and the second excavation amount; A real-time excavation volume determination module is used to determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block; A correction module, used to correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the excavation volume measurement result of the excavation block; Wherein, the video stream data includes: a plurality of frame images arranged in time; The real-time excavation amount determination module includes: The equipment matching module is used to determine the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: an excavating device and a matching target dump device; an action recognition module, configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, so as to obtain the action category corresponding to the construction machinery; The excavation volume calculation module is used to determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavation equipment when the action category of the excavation equipment in the equipment matching pair undergoes a preset change.

[0010] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0011] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0012] The embodiment of the present application also discloses a computer program product, including a computer program / instruction, which implements the aforementioned method when executed by a processor.

[0013] The embodiments of the present application include the following advantages: The embodiment of the present application provides a method for calculating the excavation volume during the construction period based on multi-source data fusion. The method uses two types of multi-source data, point cloud data and video stream data, to achieve the calculation of the excavation volume during the construction period. Specifically, for the point cloud data, the point cloud data of the excavation site at the end of the preset time period is obtained, and it is combined with the terrain surface model that has been constructed before the excavation of the excavation site, so as to determine the three-dimensional model of the excavated part of the excavation site, and then determine the first excavation volume within the preset time period. For the video stream data, the frame image contained therein is first analyzed to identify the equipment category, equipment identification and detection frame of the construction machinery therein, thereby determining the equipment matching pair (that is, an excavation device and a matching target dump device), and at the same time identifying the action category of the construction machinery. Once the action category of the excavation equipment in the equipment matching pair changes by a preset change, it is determined that it has completed one excavation, and the second excavation volume is determined in combination with the rated excavation capacity of the excavation equipment. Subsequently, a correction coefficient is determined based on the first and second excavation volumes, and the correction coefficient is used to correct the real-time cumulative excavation volume of the excavation block obtained based on the video stream data, ultimately obtaining an accurate and real-time excavation volume measurement result.

[0014] Among them, point cloud data can accurately reflect the spatial position and shape of objects. The three-dimensional model of the excavated part of the excavation site constructed based on point cloud data can obtain a first excavation volume measurement result with high accuracy. Video stream data can be acquired and analyzed in real time due to its fast real-time performance, thereby obtaining a second excavation volume measurement result that reflects the real-time construction status.

[0015] The fusion processing method of the embodiment of the present application is specifically to determine a correction coefficient based on the first excavation volume obtained based on the point cloud data and the second excavation volume obtained based on the video stream data. The correction coefficient can establish a correlation between the first excavation volume measurement result and the second excavation volume measurement result, and with the help of the high precision of the first excavation volume measurement result corresponding to the point cloud data, adjust the real-time cumulative excavation volume corresponding to the video stream data. Due to the high precision of the point cloud data, it can make up for the shortcomings of the video stream data in terms of precision; and the strong real-time nature of the video stream data can make up for the defect of untimely update of the point cloud data. Therefore, the fusion processing of the embodiment of the present application can significantly improve the accuracy of the excavation volume measurement while following up the construction progress in real time, meeting the needs of engineering construction for real-time measurement of the excavation volume during the entire construction period. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the steps of a method for calculating the entire process of excavation volume during construction based on multi-source data fusion according to an embodiment of the present application; Figure 2 It is a schematic flow chart of the steps of a method for determining the real-time cumulative excavation amount of an excavation block according to video stream data of the excavation block in one embodiment of the present application; Figure 3 is a structural diagram of an action recognition model according to an embodiment of the present application; Figure 4 It is a structural schematic diagram of a device for calculating the whole process of excavation volume during construction period based on multi-source data fusion according to an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] The embodiments of the present application can be applied to engineering industries such as hydropower and civil engineering, and can be used for open excavation operations in the engineering industry to measure the corresponding excavation volume in real time.

[0019] Taking the pumped-storage power station project as an example, the construction project presents a situation where excavation operations are carried out simultaneously at multiple construction sites. At the same time, due to the continuous changes in the on-site construction situation, there is a need for real-time measurement of the excavation volume throughout the construction period.

[0020] In traditional technologies, excavation volume calculation is either performed based on 3D design software during the design period or performed based on drone oblique photography during the construction period. However, both methods have poor real-time performance and are difficult to meet the demand for real-time calculation of excavation volume during the entire construction period in engineering construction.

[0021] In view of the technical problem that traditional technologies are difficult to meet the demand for real-time measurement of the entire process of excavation during construction, the embodiment of the present application provides a method for measuring the entire process of excavation during construction based on multi-source data fusion, which specifically includes the following steps: Determine a three-dimensional model of an excavated portion of an excavation block according to point cloud data of the excavation site at a first moment and a terrain surface model of the excavation site before excavation; the excavation site specifically includes: at least one excavation block; Determine a first excavation amount of the excavation site within a preset time period according to the three-dimensional model of the excavated portion of the excavation site; the first moment is the end moment of the preset time period; Determining a second excavation amount of the excavation portion within the preset time period according to the video stream data of the excavation portion within the preset time period; Determining a correction coefficient according to the first excavation amount and the second excavation amount; Determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block; According to the correction coefficient, the real-time cumulative excavation volume of the excavation block is corrected to obtain the excavation volume measurement result of the excavation block; The video stream data includes: a plurality of frame images arranged in time; The above-mentioned method of determining the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block includes: According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating equipment in the above-mentioned equipment matching pair, it is considered that the excavating equipment in the above-mentioned equipment matching pair completes one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the above-mentioned excavating equipment.

[0022] The embodiment of the present application provides a method for calculating the excavation volume during the construction period based on multi-source data fusion. The method uses two types of multi-source data, point cloud data and video stream data, to achieve the calculation of the excavation volume during the construction period. Specifically, for the point cloud data, the point cloud data of the excavation site at the end of the preset time period is obtained, and it is combined with the terrain surface model that has been constructed before the excavation of the excavation site, so as to determine the three-dimensional model of the excavated part of the excavation site, and then determine the first excavation volume within the preset time period. For the video stream data, the frame image contained therein is first analyzed to identify the equipment category, equipment identification and detection frame of the construction machinery therein, thereby determining the equipment matching pair (that is, an excavation device and a matching target dump device), and at the same time identifying the action category of the construction machinery. Once the action category of the excavation equipment in the equipment matching pair changes by a preset change, it is determined that it has completed one excavation, and the second excavation volume is determined in combination with the rated excavation capacity of the excavation equipment. Subsequently, a correction coefficient is determined based on the first and second excavation volumes, and the correction coefficient is used to correct the real-time cumulative excavation volume of the excavation block obtained based on the video stream data, ultimately obtaining an accurate and real-time excavation volume measurement result.

[0023] Among them, point cloud data can accurately reflect the spatial position and shape of objects. The three-dimensional model of the excavated part of the excavation site constructed based on point cloud data can obtain a first excavation volume measurement result with high accuracy. Video stream data can be acquired and analyzed in real time due to its fast real-time performance, thereby obtaining a second excavation volume measurement result that reflects the real-time construction status.

[0024] The fusion processing method of the embodiment of the present application is specifically to determine a correction coefficient based on the first excavation volume obtained based on the point cloud data and the second excavation volume obtained based on the video stream data. The correction coefficient can establish a correlation between the first excavation volume measurement result and the second excavation volume measurement result, and with the help of the high precision of the first excavation volume measurement result corresponding to the point cloud data, adjust the real-time cumulative excavation volume corresponding to the video stream data. Due to the high precision of the point cloud data, it can make up for the shortcomings of the video stream data in terms of precision; and the video stream data has strong real-time performance, which can make up for the defect of untimely update of the point cloud data. Therefore, the fusion processing of the embodiment of the present application can significantly improve the accuracy of the excavation volume measurement while following up the construction progress in real time, meeting the needs of engineering construction for real-time measurement of the excavation volume during the entire construction period.

[0025] Method Example 1 refer to Figure 1 , shows a schematic flow chart of the steps of a method for calculating the whole process of excavation volume during construction period based on multi-source data fusion according to an embodiment of the present application, the method specifically comprises the following steps: Step 101, determining a three-dimensional model of an excavated portion of the excavation site based on point cloud data of the excavation site at a first moment and a terrain surface model of the excavation site before excavation; the excavation site specifically includes: at least one excavation block; Step 102: determining a first excavation amount of the excavation site within a preset time period according to the three-dimensional model of the excavated portion of the excavation site; the first moment is the end moment of the preset time period; Step 103: determining a second excavation amount of the excavation part within the preset time period according to the video stream data of the excavation part within the preset time period; Step 104: determining a correction coefficient according to the first excavation amount and the second excavation amount; Step 105: Determine the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block; Step 106: Correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the excavation volume measurement result of the excavation block; The video stream data includes: a plurality of frame images arranged in time; The above step 105 is a process of determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block, which specifically includes: According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating equipment in the above-mentioned equipment matching pair, it is considered that the excavating equipment in the above-mentioned equipment matching pair completes one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the above-mentioned excavating equipment.

[0026] Figure 1 The method shown can be used to achieve full-process measurement of excavation volume during the construction period with the help of two types of multi-source data: point cloud data and video stream data. Figure 1 At least one of the steps included can be executed by the server. It can be understood that the embodiment of the present application is for Figure 1 The specific execution subject of the method shown is not limited.

[0027] In the embodiments of the present application, the excavation site is the specific scope where excavation work needs to be carried out for engineering objects such as slopes and dams. It is a comprehensive concept that covers the area where excavation work is to be carried out, and its scale and boundaries are determined according to the actual planning and needs of the project. The excavation site can be composed of one or more excavation areas.

[0028] The excavation area is a subdivision of the excavation site, which is further divided from the perspective of construction organization, process arrangement, etc. Each excavation area has its own independent construction plan and requirements, and is generally composed of at least one excavation block.

[0029] Excavation block is the basic construction unit in the excavation work of engineering objects such as slopes and dams, and is also the smallest statistical unit when calculating the excavation volume.

[0030] In step 101, point cloud data is a collection of discrete points obtained through technologies such as laser scanning or drone oblique photography. These collections of discrete points carry precise spatial coordinate information and can reflect the spatial position and shape of objects with high precision.

[0031] The point cloud data of the excavation site at the first moment records in detail the actual terrain surface information of the excavation site at the first moment, including various shapes and undulations formed after excavation.

[0032] The terrain surface model before excavation is a digital terrain model constructed before the excavation of the engineering object, which represents the original terrain conditions before any excavation. In practical applications, the terrain surface model of the excavation site before excavation can be established based on the design information of the engineering object corresponding to the excavation site, using the contour lines and elevation points in the original terrain.

[0033] Among them, design information is used to clarify the scope and accuracy requirements of terrain surface model construction, and specific plans such as site boundaries and overall layout are given according to engineering characteristics. Contour lines are used to intuitively display terrain undulations, reflect slope changes through their density and form, and can also extract elevation data points at a certain interval to assist in identifying various terrain features. Elevation points are used to provide accurate elevation values ​​at specific locations on the terrain surface. As basic data, with the help of interpolation algorithms, the elevation of unknown points is calculated in collaboration with surrounding elevation points, thereby generating continuous data to accurately construct terrain surface models.

[0034] The embodiment of the present application can compare and analyze the point cloud data of the excavation site at the first moment with the data of the corresponding area in the terrain surface model before excavation. The above analysis can identify the spatial position difference between the two. In this way, based on the spatial position difference, the excavated part of the excavation site can be reconstructed in three dimensions with the help of three-dimensional modeling technology to obtain a three-dimensional model of the excavated part of the excavation site.

[0035] In a specific implementation, step 101 is a process of determining a three-dimensional model of the excavated portion of the excavation site based on the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation, which specifically includes: Step 111, grid interpolation is performed on the point cloud data of the excavation site at the first moment to obtain an actual excavation contour surface of the excavation site at the first moment; Step 112, determining a first envelope of the excavation site at the first moment according to an actual excavation contour surface of the excavation site at the first moment; Step 113, performing a Boolean difference operation on the first envelope of the excavation site at the first moment and the terrain surface model of the excavation site before excavation to obtain an operation result; Step 114: According to the spatial scope of the excavation site, a three-dimensional model of the excavated portion of the excavation site is obtained from the calculation results.

[0036] After the excavation work of the excavation site is started, the actual excavation contour surface formed by the excavation work can be scanned at the first moment to obtain the point cloud data at the first moment. The corresponding scanning method can be drone tilt photography or three-dimensional laser scanning.

[0037] Optionally, the embodiment of the present application can perform smoothing processing on the scanned point cloud data. The above-mentioned smoothing processing is used to eliminate abnormal noise such as sharp corners and protrusions in the point cloud data to obtain smooth point cloud data.

[0038] Optionally, the embodiment of the present application can perform point cloud segmentation on the scanned point cloud data or the smoothed point cloud data according to the spatial scope of the excavation site, so as to obtain the point cloud data of the excavation site at the first moment.

[0039] In step 111, the point cloud data is a collection of discrete points, which are usually discontinuous and cannot directly form a continuous surface for analysis. The purpose of grid interpolation is to convert these discrete points into a continuous surface by estimating data between known points to generate a regular grid structure. As a result of network interpolation, uniform grid point cloud data is used to characterize the actual excavation contour surface of the excavation site at the first moment.

[0040] Common grid interpolation methods include: inverse distance weighted interpolation, Kriging interpolation, spline interpolation, etc. For example, inverse distance weighted interpolation is based on the principle of inverse distance, and it is believed that the closer the interpolation point is to a known point, the greater the influence of the known point. It can be understood that the embodiments of the present application do not limit the specific grid interpolation method.

[0041] The point cloud data is grid interpolated using an interpolation algorithm, such as the Kriging interpolation algorithm, to obtain uniform grid point cloud data for characterizing the actual excavation contour surface.

[0042] In step 112, the actual excavation contour surface is a two-dimensional surface. In order to more comprehensively describe the spatial form of the excavation site, it is necessary to construct a three-dimensional space body that can completely contain the actual excavation contour surface, that is, the first envelope body.

[0043] In practical applications, the actual excavation contour surface can be stretched horizontally into a first envelope. Specifically, the boundary of the actual excavation contour surface is first determined; then, according to the boundary of the actual excavation contour surface, a surface reconstruction algorithm is used to expand the actual excavation contour surface into the first envelope.

[0044] In step 113, the Boolean difference operation is an operation used to calculate the difference between two geometric entities in three-dimensional space. By subtracting the first envelope from the terrain surface model before excavation, the difference between the terrain before and after excavation can be obtained. This part is the actual excavation area, which provides key data for determining the three-dimensional model of the excavated part.

[0045] The calculation result is used to represent a new geometric entity, which represents the remaining part after removing the space occupied by the first envelope from the original terrain, that is, reflects the actual excavation area.

[0046] In step 114, the calculation result obtained through the Boolean difference operation may include relevant information of multiple excavation locations. The embodiment of the present application can filter out the excavated part belonging to a specific excavation location from the calculation result according to the spatial scope of the excavation location, thereby obtaining a three-dimensional model of the excavated part of the specific excavation location.

[0047] In three-dimensional space, the coordinate boundaries of the excavation site (such as the minimum and maximum X, Y, and Z coordinate values) can be used to define its spatial extent.

[0048] In the specific implementation, the geometric elements (such as points, lines, and surfaces) in the calculation results can be first screened according to the spatial range of the excavation site, and the target geometric elements within the spatial range can be retained, and the geometric elements outside the spatial range can be discarded. Then, based on the target geometric elements, a three-dimensional model of the excavated part of the excavation site is generated.

[0049] In step 102, the three-dimensional model is essentially a spatial entity composed of a series of geometric elements (points, lines, surfaces, etc.). The spatial volume occupied by the three-dimensional model can be calculated by mathematical methods, such as integral operations or voxel calculation methods based on discretization.

[0050] Since the first moment is set as the end moment of the preset time period, the three-dimensional model of the excavated part of the excavation site obtained at the first moment represents the excavation results within the preset time period. By calculating the volume of this three-dimensional model, the excavation volume within the preset time period, i.e., the first excavation volume, can be directly obtained.

[0051] Therefore, step 102 is a process of determining the first excavation amount of the excavation site within a preset time period based on the three-dimensional model of the excavated portion of the excavation site, which specifically includes: dividing the three-dimensional model of the excavated portion of the excavation site into grid units; determining the unit volume of the grid units in the three-dimensional model of the excavated portion of the excavation site; and summarizing the unit volumes of all grid units in the three-dimensional model of the excavated portion of the excavation site to obtain the first excavation amount of the excavation site within the preset time period.

[0052] In step 103, the first frame image in the video stream data within the preset time period can be analyzed in chronological order from front to back to obtain the first device matching pair in the first frame image and the action category corresponding to the construction machinery in the first frame image; and when the action category of the excavating device in the first device matching pair undergoes a preset change, it is considered that the excavating device in the first device matching pair completes an excavation, and the second excavation amount of the excavation part within the preset time period is determined according to the rated excavation capacity corresponding to the excavation device.

[0053] The second excavation volume determination process performs real-time monitoring of the excavation volume during the construction period according to the objective law of the cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can combine the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair to accurately determine the actual occurrence of each effective excavation, thereby determining the real-time cumulative excavation volume corresponding to the excavation site by accumulating the rated excavation capacity.

[0054] Since the above objective laws of the embodiment of the present application can realize real-time monitoring of the excavation volume during the construction period based on the intrinsic logical connection between the actual process of excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters, the embodiment of the present application can realize real-time monitoring of the excavation volume during the construction period at a relatively fast speed based on the analysis of the video stream data, thereby improving the processing efficiency of the earthwork excavation volume.

[0055] In step 104, a correction coefficient is determined based on the first excavation volume and the second excavation volume. The correction coefficient can establish a correlation between the first excavation volume calculation result and the second excavation volume calculation result, and with the help of the high precision of the first excavation volume calculation result corresponding to the point cloud data, the real-time cumulative excavation volume corresponding to the video stream data is adjusted.

[0056] In a specific implementation, a correction coefficient can be determined based on a proportional relationship. If the relationship between the first excavation amount and the second excavation amount is linear, the ratio of the two can be calculated as a correction coefficient. For example, in multiple measurements, the average value of the ratio of the first excavation amount measurement result to the second excavation amount measurement result is calculated and used as the correction coefficient. One measurement can correspond to a preset time period. The length of the preset time period can be determined by those skilled in the art according to actual application requirements. For example, the length of the preset time period can be N days, where N is a positive integer. For example, at the end of the Nth day of excavation work at the excavation site, a first measurement is performed. At the end of the 2Nth day of excavation work at the excavation site, a second measurement is performed.

[0057] During the multiple measurement processes, after each measurement is completed, the measurement results of the first excavation amount and the second excavation amount need to be recorded in detail. Then, for each measurement result, the ratio of the first excavation amount measurement result to the second excavation amount measurement result is calculated.

[0058] Record the ratios obtained each time in sequence, add up all these ratios, and then divide the accumulated sum by the total number of measurements. The final average value is the correction factor that needs to be determined. The correction factor can effectively reflect the relatively stable proportional relationship between the first excavation volume and the second excavation volume.

[0059] Alternatively, the correction coefficient may be determined using the least square method. When the relationship between the first excavation amount and the second excavation amount is relatively complex and is not a completely simple linear relationship, the least square method may be used to determine the correction coefficient.

[0060] Alternatively, the correction coefficient may be determined based on a statistical analysis method. A large amount of first excavation volume data and second excavation volume data may be statistically analyzed, for example, a scatter plot of the two may be drawn to observe the distribution pattern of the data. If the data exhibits a certain clustering or distribution trend, the correction coefficient may be determined based on the clustering or distribution trend.

[0061] Alternatively, for more complex situations, the regression algorithm in machine learning can be used to determine the correction coefficient. The data of the first excavation volume and the second excavation volume are used as training samples, and a suitable regression model is selected, such as linear regression, polynomial regression, support vector regression, etc. The mapping relationship between the two is learned through the training model to obtain the correction coefficient. For example, when using a linear regression model, during the training process, the linear regression model will automatically adjust the parameters so that the error between the predicted first excavation volume and the actual first excavation volume is minimized. The coefficient related to the second excavation volume in the final model parameters is the correction coefficient.

[0062] In step 105, the process of determining the real-time cumulative excavation volume can also monitor the excavation volume during the construction period in real time according to the objective law of the cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can combine the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair to accurately determine the actual occurrence of each effective excavation, thereby determining the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0063] In step 106, the real-time cumulative excavation volume of the excavation block is corrected according to the correction coefficient to obtain the excavation volume measurement result of the excavation block.

[0064] Assuming that the correction coefficient is k, and the real-time cumulative excavation volume of the excavation block is known to be V1, the method for correcting the real-time cumulative excavation volume of the excavation block may include: a linear correction method or a correction method based on machine learning, etc.

[0065] Among them, the linear correction method can assume that there is a linear relationship between the first excavation amount and the second excavation amount, and the correction coefficient k is used as a proportional factor between the two. Then, the product of the correction coefficient k and the real-time cumulative excavation amount V1 can be used as the excavation amount measurement result V2 of the excavation block.

[0066] The correction method based on machine learning can use the powerful nonlinear fitting ability of machine learning algorithms to model complex data relationships. A large amount of first excavation volume and second excavation volume data is used as a training set, and a suitable machine learning model (such as support vector machine regression, decision tree regression, neural network regression, etc.) is selected for training.

[0067] Taking support vector machine regression as an example, the training data is first preprocessed, including data normalization and other operations, and then the support vector machine model is trained using the training data to adjust the parameters of the machine learning model so that it can fit the relationship between the first excavation volume and the second excavation volume as accurately as possible. After the training is completed, the real-time cumulative excavation volume V1 is used as the input of the machine learning model, and the output of the machine learning model can be used as the excavation volume measurement result V2 of the excavation block.

[0068] The embodiment of the present application can record and perform other subsequent processing on the excavation volume measurement result V2 of the excavation block. For example, the excavation volume measurement result V2 is stored in a special engineering database, and is classified and stored according to information such as timestamp and excavation block number, so as to facilitate subsequent query and call at any time. For another example, based on the excavation volume measurement result V2, a detailed excavation volume statistical report is generated, and the report covers the comparative analysis of the excavation volume of different time periods and different excavation blocks, providing intuitive data support for the control of the project progress. Alternatively, by comparing the excavation volume measurement result with the excavation volume indicator in the project budget, if it is found that the actual excavation volume deviation from the budget exceeds the preset range, the early warning mechanism is triggered in time to remind the project management personnel to check and adjust the construction process, equipment use and other aspects.

[0069] In an optional implementation of the present application, the method may further include: determining the over-excavation and under-excavation of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the design volume of the first excavation block; and adjusting the remaining excavation volume of the second excavation block in the (s+1)-th excavation area according to the over-excavation and under-excavation of the first excavation block. s may be a positive integer. The s-th excavation area and the (s+1)-th excavation area may be two excavation areas that are geographically adjacent.

[0070] The over-excavation and under-excavation situations specifically include: over-excavation or under-excavation. Over-excavation means that the real-time cumulative excavation volume of the first excavation block exceeds the design volume of the first excavation block, and under-excavation means that the real-time cumulative excavation volume of the first excavation block is less than the design volume of the first excavation block.

[0071] Since the sth excavation area and the (s + 1)th excavation area are usually two geographically adjacent excavation areas, this adjacent relationship makes experience reference and data adjustment in the construction process of great significance.

[0072] The embodiment of the present application can compare the real-time cumulative excavation volume of the first excavation block with the designed volume of the first excavation block to obtain the specific over-excavation or under-excavation, and the remaining excavation volume of the second excavation block can be reduced or increased accordingly on this basis.

[0073] For example, if the first excavation block is over-excavated, the adjusted remaining excavation volume of the second excavation block is specifically: the design volume of the second excavation block minus the over-excavation volume of the first excavation block. Alternatively, if the first excavation block is under-excavated, the adjusted remaining excavation volume of the second excavation block is specifically: the design volume of the second excavation block plus the under-excavation volume of the first excavation block. Of course, in actual applications, it is also necessary to comprehensively consider factors such as soil looseness, construction errors, and the impact of subsequent processes to correct the adjusted remaining excavation volume.

[0074] In one implementation of the present application, the process of determining the design volume of the excavation block specifically includes: Step S1, establishing an excavation surface model of the excavation site after excavation according to the excavation contour line of the excavation site; Step S2, performing Boolean operation on the second envelope corresponding to the terrain surface model and the third envelope corresponding to the excavation surface model to obtain a three-dimensional model of the excavation site; Step S3, obtaining a three-dimensional model of the excavation block from the three-dimensional model of the excavation site according to the spatial range of the excavation block; Step S4: Determine the design volume of the excavation block according to the three-dimensional model of the excavation block.

[0075] In step S1, the excavation contour is the boundary definition of the final shape of the excavation site. Through mathematical methods, such as interpolation, fitting and other techniques, these discrete excavation contour data are converted into a continuous surface. This surface can be called an excavation surface model, which represents the shape of the excavation site after the excavation operation is completed.

[0076] In step S2, Boolean operation is a basic operation method in three-dimensional modeling. An envelope is a closed space that can completely contain the corresponding surface model. The second envelope of the terrain surface model represents the spatial range of the original terrain, and the third envelope of the excavation surface model represents the spatial range of the planned excavation. Through methods such as difference operations in Boolean operations, the spatial range of the planned excavation can be subtracted from the spatial range of the original terrain, and the result is a three-dimensional model of the excavation site. The three-dimensional model of the excavation site presents the spatial form formed on the basis of the original terrain after the excavation work of the excavation site is completed.

[0077] In step S3, based on the obtained three-dimensional model of the excavation site, according to the predetermined spatial range of the excavation block, spatial clipping or extraction operations are used to extract the part of the three-dimensional model of the excavation site that conforms to the spatial range, thereby obtaining a three-dimensional model of the specific excavation block.

[0078] In step S4, for the obtained three-dimensional model of the excavation block, a variety of methods can be used to calculate its volume. A common method is to discretize the three-dimensional model into multiple volume units, such as voxels (similar to the correspondence of pixels in a two-dimensional image in three-dimensional space) based on the principle of numerical integration, and then accumulate the volumes of multiple volume units to obtain the volume of the entire excavation block.

[0079] In summary, the method for calculating the excavation volume during the construction period based on multi-source data fusion in the embodiment of the present application realizes the calculation of the excavation volume during the construction period by means of two types of multi-source data, namely, point cloud data and video stream data. Specifically, for the point cloud data, the point cloud data of the excavation site at the end of the preset time period is obtained, and it is combined with the terrain surface model that has been constructed before the excavation of the excavation site, so as to determine the three-dimensional model of the excavated part of the excavation site, and then determine the first excavation volume within the preset time period. For the video stream data, the frame images contained therein are first analyzed to identify the equipment category, equipment identification and detection frame of the construction machinery therein, thereby determining the equipment matching pair (that is, an excavation device and a matching target dump device), and at the same time identifying the action category of the construction machinery. Once the action category of the excavation equipment in the equipment matching pair changes by a preset change, it is determined that it has completed one excavation, and the second excavation volume is determined in combination with the rated excavation capacity of the excavation equipment. Subsequently, a correction coefficient is determined based on the first and second excavation volumes, and the correction coefficient is used to correct the real-time cumulative excavation volume of the excavation block obtained based on the video stream data, ultimately obtaining an accurate and real-time excavation volume measurement result.

[0080] Among them, point cloud data can accurately reflect the spatial position and shape of objects. The three-dimensional model of the excavated part of the excavation site constructed based on point cloud data can obtain a first excavation volume measurement result with high accuracy. Video stream data can be acquired and analyzed in real time due to its fast real-time performance, thereby obtaining a second excavation volume measurement result that reflects the real-time construction status.

[0081] The fusion processing method of the embodiment of the present application is specifically to determine a correction coefficient based on the first excavation volume obtained based on the point cloud data and the second excavation volume obtained based on the video stream data. The correction coefficient can establish a correlation between the first excavation volume measurement result and the second excavation volume measurement result, and with the help of the high precision of the first excavation volume measurement result corresponding to the point cloud data, adjust the real-time cumulative excavation volume corresponding to the video stream data. Due to the high precision of the point cloud data, it can make up for the shortcomings of the video stream data in terms of precision; and the video stream data has strong real-time performance, which can make up for the defect of untimely update of the point cloud data. Therefore, the fusion processing of the embodiment of the present application can significantly improve the accuracy of the excavation volume measurement while following up the construction progress in real time, meeting the needs of engineering construction for real-time measurement of the excavation volume during the entire construction period.

[0082] Method Example 2 refer to Figure 2 , shows a schematic flow chart of the steps of a method for determining the real-time cumulative excavation amount of an excavation block according to video stream data of the excavation block according to an embodiment of the present application, and the method specifically includes the following steps: Step 201: Determine a device matching pair in a plurality of frame images arranged in time according to the device category, device identification and detection frame corresponding to the construction machinery in the frame image in the video stream data; the device matching pair includes: an excavating device and a matching target dump device; Step 202: performing action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images, so as to obtain the action category corresponding to the construction machinery; Step 203: When a preset change occurs to the action category of the excavating device in the above-mentioned equipment matching pair, it is considered that the excavating device in the above-mentioned equipment matching pair completes one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the above-mentioned excavating device.

[0083] In step 201, the process of obtaining the video stream data of the excavation block specifically includes: after receiving the construction video sent by the image acquisition end, the server extracts the video stream data from the construction video. The server can receive the video stream data from the image acquisition end according to a preset time period.

[0084] The multiple frame images in the video stream data are arranged in chronological order, which means that as time goes on, the pictures corresponding to each time point will form separate frame images and be sorted in the order of occurrence. In the specific implementation process, it is an effective approach to distinguish different frame images by using frame numbers. For example, the first frame image generated at the starting moment can be numbered 1, and the frame number corresponding to the next moment is numbered 2, and so on. Through such frame numbers, not only can the frame images at different moments be clearly distinguished, but also in subsequent analysis and processing links such as target detection and construction machinery tracking, the frame images at specific time points can be accurately and quickly located and called according to the frame number, so as to better calculate the corresponding excavation volume and other related indicators based on the video stream data.

[0085] In the hydropower engineering scenario, multiple video sources can be set, and different video sources can correspond to different excavation blocks. Among them, the video source can continuously provide continuous video stream data.

[0086] The embodiment of the present application can utilize target detection technology to perform target detection on the above-mentioned video stream data to obtain the equipment category and detection frame corresponding to the construction machinery in the above-mentioned frame image.

[0087] Target detection technology is an important branch in the field of computer vision, which aims to identify target information corresponding to a target in an image. Specifically, in the embodiment of the present application, the frame image may include targets such as excavation equipment and dump equipment. Examples of excavation equipment may include excavators, etc. Examples of dump equipment may include dump trucks, etc. In actual applications, the dump truck drives into the excavation site and stops within the working radius of the excavator so that the excavator can easily load the material into the cargo box of the dump truck.

[0088] The target information may specifically include: the equipment category and detection frame corresponding to the construction machinery. The equipment category specifically includes: excavation equipment category and dump equipment. The detection frame is used to indicate the position and range of the target in the frame image. The detection frame is usually a rectangular frame, whose four sides are aligned with the outermost edges of the target, thereby surrounding the entire target. The information of the detection frame may include: the coordinates of the upper left corner of the rectangular frame, the width and the height; or, the information of the detection frame may include: the coordinates of the center point of the rectangular frame, the width and the height.

[0089] In the embodiment of the present application, the target detection model can be used to perform target detection on the frame image. In a specific implementation, the frame image can be input into the target detection model to obtain the equipment category and detection frame corresponding to the construction machinery in the frame image output by the target detection model.

[0090] The target detection model is a deep learning model in the field of computer vision. It can identify targets in frame images and determine target information such as the target detection box and equipment category. The target detection model is usually implemented based on a convolutional neural network. The output result of the target detection model can be in the format of: {(machine class, xmin, xmax, ymin,ymax)}. Among them, "machine class" represents the category of the construction machinery in the frame image, such as "dump equipment" or "excavation equipment", which is used to distinguish different categories of construction machinery; "xmin" and "xmax" are the leftmost and rightmost coordinate values ​​of the detection box of the construction machinery in the horizontal direction (x-axis) of the frame image; "ymin" and "ymax" are the top and bottom coordinate values ​​of the detection box of the construction machinery in the vertical direction (y-axis) of the frame image. These four coordinate values ​​together determine a rectangular detection box.

[0091] The embodiments of the present application do not limit the specific target detection model. For example, examples of target detection models may include: YOLO (You Only Look Once) series, SSD (Single Shot PultiBox Detector), etc.

[0092] In one example, the target detection model specifically includes: a feature extraction unit, a feature fusion unit and a detection unit.

[0093] The feature extraction unit is used to extract image features from the frame image.

[0094] The feature fusion unit is used to effectively fuse image features of different levels and types. The obtained fused image features can make the feature information more comprehensive and accurate, so as to enhance the ability to express the characteristics of construction machinery.

[0095] The detection unit is used to determine the category and position of the construction machinery in the frame image according to the fused image features. The embodiment of the present application can use the target tracking technology to track the construction machinery in the multiple frame images arranged in time. The output result of the target tracking technology can be in the format of: {(machine class, machineID,xmin, xmax, ymin, ymax)}, where machineID represents the machine ID corresponding to the construction machinery.

[0096] Target tracking technology is a technology that uses a series of algorithms and models to determine the position of a specific target (the construction machinery in the embodiment of the present invention) in different frame images in a frame image sequence, and continuously estimates and updates the target state (including position, speed, direction of movement, etc.). Its core is to achieve accurate recognition and continuous tracking of the target in a complex image environment and constantly changing target state.

[0097] In a specific implementation, the process of tracking the construction machinery in the multiple frame images arranged in time specifically includes: Step A1: for the detection box in the first frame image, create a track ID (identity) in the track set; Step A2, extracting features from the area image within the detection frame in the k-th frame image to obtain a first appearance feature vector of the detection frame in the k-th frame image; the first appearance feature vector includes: a texture feature and a color feature; the weight of the texture feature is greater than the weight of the color feature; k is a positive integer; Step A3, determining the predicted trajectory of the trajectory ID in the trajectory set in the kth frame image according to the historical trajectory information of the trajectory ID in the trajectory set; Step A4, determining the state of the track ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID; the state of the track ID includes: a confirmed state or a non-confirmed state; Step A5: Assign a machine ID to the construction machine corresponding to the trajectory ID of the confirmed status.

[0098] In steps A1 to A5 of the embodiment of the present application, feature matching and trajectory prediction are used to implement tracking of the construction machinery in multiple frame images.

[0099] In the hydropower engineering scenario, construction machinery of the same category is similar in appearance, shape or color, which can easily lead to errors in the matching process between the trajectory ID and the detection frame by the trajectory tracking means. In the process of calculating the first appearance feature vector of the image of the area within the detection frame, the embodiment of the present application controls the weight of the texture feature to be greater than the weight of the color feature. Since the texture feature can better reflect the surface structure and details of the construction machinery, and the color feature has a lower degree of distinction when the model and appearance color are similar, on this basis, the embodiment of the present application determines the state of the trajectory ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID. This can more accurately distinguish different construction machinery, reduce the probability of misjudgment due to similar appearance, and improve the distinction of construction machinery.

[0100] In step A1, a tracking track is initialized by creating a track ID for the detection box in the initial first frame image.

[0101] Step A2 extracts the appearance feature vector of the detection frame area in each frame image, with emphasis on texture features, because in construction scenes, the texture features of construction machinery are more stable and distinguishable than color features.

[0102] For example, for two excavators of the same model, the colors and appearances of the two are similar, but the two may be different in terms of wear and scratches, cleanliness, corrosion, warning signs and safety signs, etc. The texture features of the embodiments of the present application can precisely reflect the differences between the two excavators of the same model in terms of wear and scratches, cleanliness, corrosion, warning signs and safety signs, etc.

[0103] Step A2 is a process of extracting features from the area image within the detection frame in the k-th frame image, specifically comprising: Step A21, using a convolution module to extract a first eigenvector of the regional image; Step A22, extracting a second eigenvector of the regional image using a local binary pattern method; Step A23: merge the first feature vector and the second feature vector to obtain an appearance feature vector.

[0104] In step A21, the convolution module can be constructed based on models such as CNN (Convolutional Neural Network) and VGG (Visual Geometry Group).

[0105] The first feature vector extracted by the convolution module is high-dimensional, usually containing hundreds, thousands or even more dimensions. Examples of the dimensions of the first feature vector may include: spatial features, hierarchical features, scale features, directional features, color features, depth features, context features, etc. Among them, the spatial features represent the spatial structural information of the construction machinery, such as edges, corners, textures, etc. Therefore, the spatial features may include: texture features.

[0106] LBP (Local Binary Patterns) is a simple and effective algorithm for describing image texture features. The LBP algorithm generates a binary code by comparing the grayscale values ​​of each pixel in the image with its neighboring pixels, and then converts these binary codes into decimal numbers to form a texture descriptor. Since LBP can effectively describe the local texture features of the image and capture the subtle changes and patterns in the image; therefore, the second eigenvector can contain: texture features with strong distinguishing ability.

[0107] The process of fusing the first feature vector and the second feature vector in the embodiment of the present application specifically includes: weighting the features of the same dimension in the first feature vector and the second feature vector to obtain a weighted processing result; the features of the same dimension include: texture features; and increasing the corresponding numerical value for the weighted processing result corresponding to the texture feature.

[0108] Assuming that the weight of the first eigenvector is w1, the value of the texture feature in the first eigenvector is f1, the weight of the second eigenvector is w2, and the value of the texture feature in the second eigenvector is f2, the weighted processing result corresponding to the texture feature can be: w = w1*f1+w2*f2. In order to highlight the importance of the second eigenvector, w2 can be greater than w1. It can be understood that those skilled in the art can determine the specific values ​​of w1 and w2 according to actual application requirements.

[0109] The embodiment of the present application increases the corresponding value for the weighted processing result corresponding to the texture feature, which can achieve the technical effect that the weight of the texture feature is greater than the weight of the color feature. The increased value can be determined by those skilled in the art according to actual application requirements.

[0110] In step A3, based on the historical trajectory information of the trajectory ID in the trajectory set, the predicted trajectory of the trajectory ID in the trajectory set in the kth frame of the video stream image is determined. k can be a positive integer greater than 1.

[0111] The historical track information may include: the position information of the construction machine or the detection frame corresponding to the track ID in the past frame images, and the position information generally includes coordinates, timestamp, speed, direction and other information.

[0112] The embodiment of the present application can use the Kalman filter method to determine the predicted trajectory of the trajectory ID in the trajectory set in the kth frame of the video stream image. Kalman filtering is an efficient autoregressive filter that can estimate the state of a linear dynamic system from a series of incomplete and noisy measurements.

[0113] In step A4, the state of the track ID may include: unconfirmed state or confirmed state. The confirmed state may refer to the track ID having a corresponding target detection frame after matching calculation. The unconfirmed state may refer to the track ID being generated due to the initialization of the first frame image or the appearance of a new construction machinery target object, and not having a corresponding target detection frame after matching calculation.

[0114] Step A4 is a process of determining the state of the track ID according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID, specifically including: Step A41, performing a first match between the trajectory ID in the trajectory set whose status is confirmed and the detection frame in the detection frame set to obtain a corresponding first matching result; wherein, in the first matching process, determining the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the trajectory ID, and determining the first matching result according to the cosine distance; the first matching result includes: the trajectory ID for which the first matching succeeds, the trajectory ID for which the first matching fails, and the detection frame for which the first matching fails; Among them, the trajectory set is a data structure used to record the trajectory of the construction machinery in the frame image. It assigns a unique trajectory ID to each tracked construction machinery, through which the historical trajectory information such as the position and appearance features of the construction machinery in the past frames can be indexed. For example, when the Kalman filter is used to predict the position of a construction machinery in the kth frame image, the historical position information (such as coordinates, speed, direction, etc.) in the trajectory set is used as important input data, thereby realizing the continuous estimation and update of the target motion state.

[0115] The detection box set stores the location information of the construction machinery detected in each frame image (in the form of detection boxes). These detection boxes determine the specific range of the target in the frame image.

[0116] Step A42, performing a second match on the trajectory ID that failed the first match or the trajectory ID in the unconfirmed state and the detection frame that failed the first match, to obtain a corresponding second matching result; the second matching is an intersection-over-union matching; the second matching result includes: the trajectory ID that succeeded the second match, the trajectory ID that failed the second match, and the detection frame that failed the second match; Step A43: for the detection frame that fails the second matching, determine it as a new construction machinery target object, and create a track ID in a non-confirmed state in the track set; Step A44: if the second matching result of the trajectory ID in the unconfirmed state is a successful match, update the state of the trajectory ID to the confirmed state; and if the number of second matching failures of the trajectory ID in the confirmed state exceeds the first threshold, delete the trajectory ID.

[0117] In step A41, the embodiment of the present application can determine the state of the newly created trajectory ID according to the actual application requirements. For example, when creating a trajectory ID in the trajectory set for the detection frame in the first frame of the video stream image, the state of the trajectory ID corresponding to the detection frame in the first frame of the video stream image can be set to a non-confirmed state. For another example, for the detection frame that fails the second match, it is determined to be a new construction machinery target object, and a trajectory ID with a non-confirmed state is created.

[0118] It should be noted that the embodiment of the present application provides a switching method from the non-confirmed state to the confirmed state. Specifically, for the track ID in the non-confirmed state, the embodiment of the present application will verify the authenticity of the construction machinery corresponding to the track ID in the non-confirmed state based on the subsequent second matching result. The above verification means can identify and eliminate errors in the target detection results, thereby reducing errors in the construction machinery counting results caused by errors in the target detection results.

[0119] In the first matching process, the embodiment of the present application determines the cosine distance between the first appearance feature vector of the detection box and the second appearance feature vector corresponding to the trajectory ID, and determines the first matching result according to the cosine distance.

[0120] The second appearance feature vector is an appearance feature vector associated with the track ID, and represents the appearance features of the detection box corresponding to the track ID in the historical video stream image.

[0121] In practical applications, the second appearance feature vector corresponding to the trajectory ID may be determined according to the first appearance feature vector of the detection frame corresponding to the trajectory ID in the historical video stream image.

[0122] For example, if the track ID1 is created based on the detection box in the first frame of the video stream image, then the second appearance feature vector B2 of the track ID1 in the second frame of the video stream image can be determined based on the first appearance vector A1 of the detection box A in the first frame of the video stream image. The second appearance feature vector B2 can be equal to the first appearance vector A1.

[0123] For example, if the track ID2 is created based on the detection box B in the first frame of the video stream image, then the track ID2 successfully matches the detection box C in the second frame of the video stream image, the first appearance vector C2 of the detection box C in the second frame of the video stream image, then the second appearance feature vector B3 of the track ID2 in the third frame of the video stream image can be jointly determined based on the first appearance vector B1 of the detection box B in the first frame of the video stream image of the track ID2, and the first appearance vector C2 of the detection box C in the second frame of the video stream image. Of course, the embodiment of the present application does not limit the specific process of jointly determining the second appearance feature vector B3 of the track ID2 in the third frame of the video stream image.

[0124] Since the texture features contained in the first appearance feature vector or the second appearance feature vector of the embodiment of the present application can better reflect the surface structure and details of the construction machinery, on this basis, the embodiment of the present application determines the first matching result according to the cosine distance between the first appearance feature vector of the detection frame and the second appearance feature vector corresponding to the track ID, which can effectively reduce the errors that occur in the first matching process between the track ID and the detection frame by the track tracking means. In other words, the first matching of the embodiment of the present application can correctly associate the detection frame of each construction machinery with its corresponding track ID based on accurate appearance feature matching, which can effectively reduce matching errors caused by similar appearance, and thus can improve the accuracy of construction machinery counting analysis in hydropower engineering scenarios.

[0125] In one implementation of the present application, step A41 is a process of first matching the track ID in the track set with the detection frame in the detection frame set, specifically including: Step A411, determining a cosine distance between a first appearance feature vector of the detection frame and a second appearance feature vector corresponding to the track ID whose state is the confirmation state; Step A412, determining the Mahalanobis distance between the detection frame and the track ID whose state is the confirmation state; Step A413, constructing a cost matrix; the elements in the cost matrix represent the matching cost between a detection box and a trajectory; the values ​​of the elements are determined according to the cosine distance and the Mahalanobis distance; Step A414: solving the cost matrix with the goal of minimizing the total matching cost to obtain a first matching result.

[0126] In step A411, cosine distance is a commonly used similarity measurement method, which measures the similarity between two vectors by calculating the cosine value of the angle between them. The smaller the cosine distance, the more similar the two vectors are, indicating that the detection box may correspond to the same construction machine as the track ID. The cosine distance reflects the similarity between the detection box and the track ID in terms of appearance features.

[0127] In step A412, the Mahalanobis distance is a statistic used to measure the similarity between two random vectors. The Mahalanobis distance takes into account the covariance structure in the data, so it can better handle the correlation and scale differences in the data. Generally speaking, the smaller the Mahalanobis distance, the higher the similarity between the two vectors. The Mahalanobis distance characterizes the similarity between the detection box and the trajectory ID in terms of position features.

[0128] In the process of determining the Mahalanobis distance between the detection box and the trajectory ID in the confirmed state, a first position vector corresponding to the detection box can be determined, a second position vector corresponding to the trajectory ID can be determined, and the Mahalanobis distance between the first position vector and the second position vector can be calculated.

[0129] The second position vector may be determined based on the aforementioned position information. The position information corresponding to the trajectory ID may be determined based on a Kalman filter method.

[0130] In step A413, the cost matrix is ​​a two-dimensional matrix, each element of which represents the matching cost between a detection box and a track ID. This matching cost is usually determined based on a combination of cosine distance and Mahalanobis distance. For example, the cosine distance and the Mahalanobis distance can be weighted summed, or other functions can be used to combine the cosine distance and the Mahalanobis distance to form the final matching cost.

[0131] It should be noted that before constructing the cost matrix, the cosine distance and the Mahalanobis distance may be filtered to filter out the trajectory IDs with larger cosine distance or Mahalanobis distance.

[0132] In step A414, an optimization method such as the Hungarian algorithm and the linear assignment problem solver can be used to find a matching solution that minimizes the total matching cost. The Hungarian algorithm is a classic combinatorial optimization algorithm that is suitable for solving the minimum weight matching problem of a bipartite graph. By solving the cost matrix, it can be determined which detection boxes should be matched with which trajectory IDs.

[0133] The first matching result is obtained by solving the cost matrix. Each element in the cost matrix represents the matching cost between a detection box and a track ID. This cost is obtained by combining the appearance feature (cosine distance) and the position feature (Mahalanobis distance). The purpose of solving the cost matrix is ​​to find a matching solution that minimizes the total matching cost.

[0134] For example, in the process of solving the cost matrix, the Hungarian algorithm will look for the optimal matching combination. If the matching cost between the detection box A and the trajectory ID1 is the smallest among all possible combinations, then the detection box A and the trajectory ID1 will be matched together in the solution result, and this matching relationship constitutes part of the first matching result. Therefore, the successfully matched detection box and trajectory ID in the first matching result are determined by the minimization of the cost matrix.

[0135] The first matching result of the embodiment of the present application specifically includes: a trajectory ID where the first matching is successful, a trajectory ID where the first matching fails, and a detection box where the first matching fails.

[0136] For the first successfully matched trajectory ID, the Kalman filter method can be used to update the fusion of historical trajectory information. Specifically, according to the position information of the successfully matched detection frame, the position information corresponding to the first successfully matched trajectory ID is updated, and according to the first appearance feature vector of the successfully matched detection frame, the second appearance feature vector of the first successfully matched trajectory ID is updated.

[0137] In the embodiment of the present application, step A42 may be performed for the first track ID that failed to match and the first detection frame that failed to match.

[0138] Step A42 can perform a second match on the trajectory ID that failed the first match or the trajectory ID that is in a non-confirmed state with the detection frame that failed the first match to obtain a corresponding second matching result. The second match, as a supplement to the first match, can handle those detection frames and trajectory IDs that failed to match successfully in the appearance feature match, and is suitable for handling situations where the appearance feature match fails due to occlusion, temporary disappearance, etc. Therefore, the second match of the embodiment of the present application can further improve the accuracy of the construction machinery counting analysis in the hydropower engineering scenario.

[0139] The second match can be an intersection-and-union match. The intersection-and-union match process specifically includes: calculating the area intersection-and-union ratio between the prediction box and the detection box corresponding to the trajectory ID. If the area intersection-and-union ratio is greater than a set threshold, the match is considered successful, otherwise the match fails. The prediction box corresponding to the trajectory ID can also be a rectangular box, which can be determined by a Kalman filter.

[0140] The second matching result specifically includes: a trajectory ID where the second matching is successful, a trajectory ID where the second matching is unsuccessful, and a detection frame where the second matching is unsuccessful.

[0141] For the second successfully matched trajectory ID, the Kalman filter method can be used to update the fusion of historical trajectory information. Specifically, according to the position information of the successfully matched detection frame, the position information corresponding to the second successfully matched trajectory ID is updated, and according to the first appearance feature vector of the successfully matched detection frame, the second appearance feature vector of the second successfully matched trajectory ID is updated.

[0142] The embodiment of the present application can execute step A43 for the detection frame that fails the second matching. In step A43, the detection frame that fails the second matching is determined to be a new construction machinery target object, and a track ID with a non-confirmed state is created in the track set.

[0143] The reason for the existence of the detection frame where the second match fails may be: new construction machinery appears in the excavation block, or it may be: there is an error in the target detection result, and an object that is not a construction machinery is mistakenly identified as a construction machinery. The embodiment of the present application creates a track ID with a non-confirmed state for the detection frame where the second match fails, and verifies the authenticity of the construction machinery corresponding to the track ID in the non-confirmed state based on the subsequent second matching result. The above means can identify and eliminate errors in the target detection results, thereby reducing errors in the construction machinery counting results caused by errors in the target detection results.

[0144] When the second matching failure number of the trajectory ID in the confirmation state exceeds the first threshold, the trajectory ID is deleted. The first threshold can be determined by those skilled in the art according to actual application requirements, for example, the first threshold can be 60 or other values.

[0145] For the trajectory ID that failed the second match, it can be determined whether it is in a non-confirmed state. If so, the trajectory ID can be directly deleted; if not, it can be determined whether the loss age of the trajectory ID is greater than a second threshold value, which can be a value such as 60. If it is greater, the trajectory ID is directly deleted, and if it is less, the trajectory ID is retained. Therefore, the method of the embodiment of the present application can also include: deleting the trajectory ID that is in a non-confirmed state and whose second matching result is a second match failure in the trajectory set.

[0146] In step A5, a machine identifier is assigned to the construction machine corresponding to the confirmed track ID. The machine identifier uniquely identifies the construction machine in the excavation block, and can be a string of 0001, 0002, etc. A construction machine corresponds to the same machine identifier in different frame images.

[0147] In the embodiment of the present application, for earthwork open excavation, there is a functional complementary relationship between the excavation equipment and the self-unloading equipment. The excavation equipment is responsible for excavating the earth, and the self-unloading equipment is responsible for transporting the earth. The two work closely together in the construction process. Therefore, the embodiment of the present application can use the spatial proximity relationship between the excavation equipment and the self-unloading equipment, and determine the equipment matching pairs in the above multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the above frame images.

[0148] In a specific implementation, the process of determining the device matching pairs in the multiple frame images arranged in time according to the device category, device identification and detection frame corresponding to the construction machinery in the frame images specifically includes: Step B1: determining a set of self-unloading equipment for the excavating equipment in the frame image; the distance between the self-unloading equipment and the excavating equipment in the set of self-unloading equipment does not exceed a distance threshold; Step B2: taking the maximum distance intersection-to-joint ratio index value as a criterion, determining a target self-unloading device that matches the excavating device from the set of self-unloading devices; the excavating device and the target self-unloading device form a device matching pair.

[0149] In step B1, for each excavating device in the k-th frame image, in order to screen out the dump equipment that may match it from the numerous construction machines in the k-th frame image, it is necessary to construct a dump equipment set.

[0150] The embodiment of the present application can calculate the distance between an excavating device and multiple dumping devices, and record the dumping devices whose distance does not exceed the distance threshold into the dumping device set. The distance threshold can be determined by those skilled in the art according to actual application requirements, and the embodiment of the present application does not limit the specific distance threshold.

[0151] In step B2, the DIOU (Distance - Intersection over Union) metric not only considers the Intersection over Union itself, but also incorporates the distance factor between the excavator and the dumper.

[0152] Assuming that the frame image contains N detection frames of mining equipment and M detection frames of dump trucks, the calculation of the DIOU index is shown in formula (1).

[0153] (1) in, Indicates indivual( and is a positive integer) detection frame of the self-unloading equipment; Indicates indivual( and is a positive integer) to mine the detection box of the device. , Represents the detection box , The center point of Center point , The Euclidean distance between is the diagonal distance of the minimum enclosed area that contains the detection box of the mth dump equipment and the detection box of the nth excavator. is the DIOU index between the detection box of the m-th dump equipment and the detection box of the n-th mining equipment.

[0154] After calculating the distance intersection and union ratio index values ​​for all the dump unloading equipment in the dump unloading equipment set, compare the values ​​of these distance intersection and union ratio indexes, and find the dump unloading equipment with the largest distance intersection and union ratio index value as the target dump unloading equipment that best matches the excavation equipment.

[0155] After determining the device matching pair in the jth frame image, it can be determined whether the subsequent frame image reuses the device matching pair in the jth frame image according to the position change of the dump equipment in the device matching pair in the jth frame image.

[0156] Accordingly, the process of determining the device matching pairs in the multiple frame images arranged in time according to the device category, device identification and detection frame corresponding to the construction machinery in the frame image specifically includes: Determine the first device matching pair in the j-th frame image according to the device category, device identification and detection frame corresponding to the construction machinery in the j-th frame image; j and m are both positive integers; If the position change of the dump device in the first device matching pair in the j+mth frame image is within the set range, the first device matching pair in the jth frame image is used for the j+mth frame image; If the position change of the dump equipment in the first equipment matching pair in the j+mth frame image is outside the set range, the second equipment matching pair in the j+mth frame image is determined based on the equipment category, equipment identification and detection box corresponding to the construction machinery in the j+mth frame image.

[0157] Because in normal earthwork excavation operations, the position of the dump equipment will not change much during the loading process. If the position change of the dump equipment in the first equipment matching pair in the j+mth frame image is within the set range, it means that the coordination relationship between the dump equipment and the excavation equipment has not changed, so the first equipment matching pair in the jth frame image can be reused. This is based on the working characteristics of the dump equipment in the construction scene. Its main task is to receive the materials loaded by the excavation equipment at a relatively fixed position, and its position is relatively stable.

[0158] If the position change of the self-unloading equipment in the j+mth frame image exceeds the set range, it means that the state of the self-unloading equipment has changed significantly, and it may have left the loading position or moved to other places for other operations. At this time, the previous equipment matching pair may no longer be applicable, so it is necessary to re-determine the second equipment matching pair based on the equipment category, equipment identification and detection frame of the construction machinery in the j+mth frame image to accurately reflect the actual coordination relationship between the construction machinery in the current frame image.

[0159] The release of the above-mentioned multiplexed device matching pairs can save computing resources and improve the efficiency of the entire construction machinery motion recognition and device matching process, especially when processing a large number of frame images, the effect is more obvious.

[0160] The position change of the self-unloading equipment in the first device matching pair in the j+mth frame image may specifically refer to the position change of the self-unloading equipment in the first device matching pair in the j+mth frame image relative to the position change of the self-unloading equipment in the first device matching pair in the jth frame image, that is, the position change of the self-unloading equipment in the same first device matching pair in the j+mth frame image and the jth frame image.

[0161] In practical applications, the position change can be judged whether it is within the set range based on the detection frame of a dump truck in different frame images. Specifically, the center coordinates of the detection frame of the dump truck in the jth and j+mth frame images are compared to determine the offset of the center coordinates of the detection frame. If the center coordinate offset is within the allowable coordinate difference range, the position change can be considered to be within the set range. Conversely, if the center coordinate offset is not within the allowable coordinate difference range, the position change can be considered to be not within the set range.

[0162] In a specific implementation, the format of the output result of step 201 may be: {(matchID, excavatorID, truckID)}. Wherein, excavatorID and truckID represent the excavator ID and dump device ID, respectively, which have a matching relationship. It should be noted that the excavator ID and dump device ID have corresponding mechanical identifiers, respectively. In other words, the mechanical identifier of the excavator in the equipment matching pair may be used as the excavator ID, and the mechanical identifier of the dump device in the equipment matching pair may be used as the dump device ID.

[0163] In step 202, motion recognition technology can be used to analyze the motion categories of the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images to obtain the motion category corresponding to the construction machinery.

[0164] Among them, the equipment category can pre-define the possible range of motion of the construction machinery according to the mechanical function. The above range of motion can include: the common excavation and rotation actions of excavation equipment, the general loading and unloading actions of dump equipment, etc. The equipment identification can accurately lock each construction machinery, improve the consistency and accuracy of motion recognition, and will not confuse the movement trajectories of different construction machinery. The detection frame can accurately extract the position, shape, posture and other changing features of the construction machinery in each frame image.

[0165] In a specific implementation, step 202 is a process of performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images, which specifically includes: Step C1, performing a first sampling on the multiple frame images arranged in time according to a first sampling rate, and using P first convolution modules to perform a first convolution processing on the detection frame area in the first sampling result, and the Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1; Step C2: performing a second sampling on the multiple frame images arranged in time according to a second sampling rate, and performing a second convolution processing on the detection frame area in the second sampling result using P second convolution modules, and the Pth second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module adopts a first convolution kernel, the second convolution module adopts a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel; Step C3, performing average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature; Step C4: Determine the action category corresponding to the construction machinery according to the average pooling feature.

[0166] Reference Figure 3 , shows a schematic diagram of the structure of an action recognition model of an embodiment of the present application, the input of the action recognition model specifically includes: a plurality of frame images arranged in time, and the result of target tracking.

[0167] The action recognition model is used to perform action recognition on the construction machinery in the multiple frame images arranged in time, and the output result can be in the format of: {(machine class, machineID, machine actionID,xmin, xmax, ymin, ymax)}. Among them, machine actionID represents the action category corresponding to a specific detection box (xmin, xmax, ymin, ymax), a specific machine class (equipment category) and a specific machineID (equipment identifier) ​​in a frame image.

[0168] Figure 3 In the example, the action recognition model specifically includes: a first processing branch 301, a second processing branch 302, an average pooling module 303 and a fully connected module 304.

[0169] The first processing branch 301 corresponds to step C1, and is used to perform a first sampling on the multiple frame images arranged in time according to a first sampling rate, and use P first convolution modules to perform a first convolution processing on the detection frame area in the first sampling result, and the Pth first convolution module outputs a first convolution feature. P is a positive integer greater than 1, Figure 3In the embodiment, the value of P is 3. It can be understood that the embodiment of the present application does not limit the specific value of P.

[0170] The second processing branch 302 corresponds to step C2, and is used to perform a second sampling on the multiple frame images arranged in time according to a second sampling rate, and use P second convolution modules to perform a second convolution processing on the detection frame area in the second sampling result, and the Pth second convolution module outputs a second convolution feature.

[0171] The second sampling rate is greater than the first sampling rate, so that it can pay more attention to information with obvious differences in the time dimension. For example, the sampling rate of the second sampling module can be set to 8 times the sampling rate of the first sampling module, that is, under the same video stream data input conditions, the sampling frame number of the second sampling module is 8 times that of the first sampling module.

[0172] From the perspective of capturing action details, the high sampling rate of the second sampling module can obtain more frame images, which is critical for capturing rapid changes and subtle movements in construction machinery. For example, the instantaneous action of the excavation teeth cutting into the material when the excavator is digging or the brief action of the compartment door opening when the self-unloading equipment is unloading, these details that are easily overlooked at a lower sampling rate can be well captured through a high sampling rate. These detailed action features are important for accurately identifying action categories, which can improve the accuracy of action recognition and avoid misjudging similar actions.

[0173] From the perspective of utilizing information in the time dimension, a higher sampling rate can better focus on information with obvious differences in the time dimension. The movements of construction machinery are fast or slow in time series. A high sampling rate can obtain enough frames in a short period of time to analyze the changing trend of the movement. This helps to obtain more accurate movement features in the stage where the movement changes rapidly (such as fast unloading of self-unloading equipment, fast rotation of excavation equipment, etc.), so as to more accurately judge the movement category of construction machinery as a whole, especially for those with short movement cycles and large movement changes, which can effectively improve the accuracy and completeness of recognition.

[0174] The first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel. In a convolutional neural network, the number of output channels of a convolution kernel can be simply understood as the number of feature maps generated after the convolution operation.

[0175] The first convolution kernel with a large number of channels can capture the more macroscopic and overall features of the construction machinery movements as comprehensively as possible at a relatively low sampling rate. Because the frame interval is relatively large at a low sampling rate, it is necessary to comprehensively consider the movement from multiple dimensions. More channels can map the movement from different perspectives, such as the overall movement direction of the excavator arm of the excavator, the approximate angle change range, the change in relative position in the picture, etc. These are integrated to form coarse-grained features, which facilitates a rough judgment framework for the movement.

[0176] The second convolution kernel with a small number of channels focuses on extracting key details of construction machinery movements when processing a large number of dense frame images at a high sampling rate, such as the slight position change when the excavation teeth of the excavator cut into the material, and the precise change of a local angle when the dump truck unloads the material. It extracts fine-grained features in a concise and targeted manner, which can avoid interference from redundant information and meet the needs of quickly capturing details at high sampling rates.

[0177] The average pooling module 303 is used to perform average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature.

[0178] Average pooling is a downsampling operation. The first convolution feature and the second convolution feature may contain information from many different locations. The average pooling module calculates the average value of these features in a certain area (such as a small pixel area). This can reduce the amount of data while retaining the main feature information. For example, the convolution features corresponding to multiple pixels in a local area are averaged to obtain a more representative value, avoiding overfitting, and reducing the dimension of the data, making subsequent processing more efficient.

[0179] The fully connected module 304 is used to determine the action category corresponding to the construction machine according to the average pooling feature.

[0180] Each neuron in the fully connected module is connected to all neurons in the previous layer (average pooling layer). These connections have corresponding weights. By multiplying the average pooling features with these weights and summing them, and then passing through the activation function, the action category corresponding to the construction machinery can be determined based on the learned pattern. For example, if the value output by the average pooling feature after these operations meets the numerical range corresponding to the pattern of the excavation action, it is determined to be an excavation action.

[0181] In an optional embodiment of the present application, the process of performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images may also include: fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the i+1-th first convolution module.

[0182] The above fusion can better combine the detail features at a high sampling rate contained in the second convolution result and the macro features at a low sampling rate contained in the first convolution result, so that each stage of convolution processing can make full use of the two convolution results. In addition, the above fusion can make the features subsequently output by the first convolution module integrate key information at different sampling rates, thereby improving the accuracy of construction machinery action recognition.

[0183] In a specific implementation, the process of fusing the second convolution result output by the i-th second convolution module with the first convolution result output by the i-th first convolution module may specifically include: Step D1, sampling the second convolution result according to the time step of the first convolution result to obtain a third sampling result; Step D2, performing matrix reconstruction and transposition on the third sampling result according to the matrix structure of the first convolution result, so that the matrix structure of the obtained transposition result matches the matrix structure of the first convolution; Step D3, performing time stride convolution on the transposed result and the first convolution result to obtain a stride convolution result; Step D4: Fuse the stride convolution result with the first convolution result.

[0184] In step D1, the strides of the first convolution result and the second convolution result in the time dimension (which can be understood as the interval between data points) are different due to different sampling rates. Step D1 samples the time stride of the second convolution result based on the time stride of the first convolution result, in order to make the two preliminarily aligned in the time dimension.

[0185] Although step D1 adjusts the time step, the matrix structure (such as dimension, shape, etc.) of the second convolution result may still not match the first convolution result. Step D2 reconstructs and transposes the third sampling result according to the matrix structure of the first convolution result. The reconstruction and transposition operation of step D2 is used to adjust the third sampling result to the same matrix form as the first convolution result, so that the two can match in matrix structure, providing a data basis with consistent structure for subsequent fusion operations.

[0186] After matching the matrix structures of the two results in step D2, step D3 performs time-stride convolution on the transposed result (i.e., the third sampling result after processing in step D2) and the first convolution result. The purpose of this operation is to further fuse the transposed result and the first convolution result in the time dimension and the space dimension (matrix structure). Time-stride convolution can extract the joint features of the transposed result and the first convolution result in time and space, and find a more complex correlation between them. For example, the correlation between the action details and the overall action trend of the mining equipment at different time and space positions is very important for accurately identifying the action category of construction machinery.

[0187] Step D4 fuses the stride convolution result with the first convolution result. The fusion result will include: feature information obtained from different sampling rates (the first sampling rate corresponding to the first convolution module and the second sampling rate corresponding to the second convolution module). These feature information include both macroscopic overall action features and microscopic detailed action features. Through such fusion, it is possible to provide more comprehensive and richer feature inputs for subsequent action recognition, thereby improving the accuracy of construction machinery action category judgment and avoiding misjudgment caused by relying only on features at a single sampling rate.

[0188] In step 203, each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can accurately determine the actual occurrence of each effective excavation by combining the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair, and thus determine the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0189] In one implementation, the action category of the excavating equipment in the equipment matching pair undergoes a preset change, specifically including: the action category of the excavating equipment in the equipment matching pair changes in the order of excavating, lifting and rotating, unloading, empty bucket return, and excavating.

[0190] In the construction scenario, the sequence of the action categories of the excavation equipment is a complete work cycle. The first is excavation, where the excavation equipment controls the bucket to insert into the material pile and fill it with materials. Next is lifting and rotation, where the mechanical arm raises the bucket filled with materials and rotates the body so that the bucket is aligned with the position of the dump equipment. Then comes unloading, where the materials in the bucket are poured into the dump equipment. After that, the empty bucket returns, and the empty bucket is put back to its original position or the position ready for the next excavation. Finally, excavation begins again, and the cycle continues. This sequence fully demonstrates the process of each action category working in sequence when the excavation equipment cooperates with the dump equipment.

[0191] When the excavation equipment leaves the factory, there will be a parameter setting of the rated excavation capacity, which represents the maximum volume of material that the excavation equipment can theoretically excavate during each normal excavation. In each actual effective excavation process, the excavation volume of the excavation equipment is in a fixed corresponding relationship with the rated excavation capacity. Therefore, the embodiment of the present application can determine the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0192] In practical applications, the process of determining the rated excavation capacity includes: Step E1, using the mining equipment model classification model, determining the target model corresponding to the mining equipment whose action category undergoes a preset change; Step E2: According to the target model, search in the mapping relationship between the search model and the rated mining capacity to determine the rated mining capacity corresponding to the mining equipment whose action category has undergone a preset change.

[0193] The mining device model classification model of the embodiment of the present application can have the ability to classify mining device models. In other words, it can determine the target model corresponding to the mining device in the local image based on the local image corresponding to the mining device with a preset change in action category.

[0194] In a specific implementation, the embodiment of the present application can label the model category for the mining device image data set, and train the mining device model classification model based on the mining device image data set.

[0195] Here, a process for acquiring an excavation equipment image dataset is provided. Specifically, based on a video surveillance system, excavation equipment images covering different construction backgrounds, different lighting conditions, different working postures, etc. can be collected. In addition, it is also necessary to collect a combined image of an excavation equipment and a dump truck in cooperation with the excavation operation to reflect the actual situation of the two types of construction machinery blocking each other. Therefore, the excavation equipment image dataset may include: excavation equipment images and combined images.

[0196] The embodiment of the present application does not limit the specific structure of the mining equipment model classification model. For example, the mining equipment model classification model may include: a convolutional neural network structure such as VGG (Visual GeoPetry Group).

[0197] The embodiment of the present application can save the mapping relationship between the model and the rated excavation capacity in advance, so that the target model corresponding to the excavation equipment with the preset change in action category can be searched in the above mapping relationship to obtain the rated excavation capacity corresponding to the excavation equipment with the preset change in action category. The above mapping relationship can be determined based on information sources such as the technical parameter manual of the excavation equipment or the equipment nameplate.

[0198] In the embodiment of the present application, when the excavation operation state of the current target area is monitored to be updated from the loading state to the unexcavated state or the excavation state, the current target area may be considered to have completed the current loading, and the excavation volume corresponding to the current loading of the current target area may be determined according to the rated excavation capacity corresponding to the self-unloading equipment in the current target area. The excavation volume corresponding to the current loading may be equal to the rated excavation capacity corresponding to the self-unloading equipment in the current target area.

[0199] When the action category of the excavating device in the device matching pair changes by a preset value, it is considered that the excavating device in the device matching pair completes one excavation, and the process of determining the real-time cumulative excavation amount corresponding to the excavation block according to the rated excavation capacity corresponding to the excavation device specifically includes: Step F1: Determine the number of effective excavation times according to the number of preset changes in the action category of the excavation equipment in the equipment matching. For example, when the action of the excavation equipment is completely cycled in the order of excavation, lifting rotation, unloading, empty bucket return, and excavation, it is counted as an effective excavation. The number of such complete cycles is counted by real-time monitoring of video frame images or construction processes.

[0200] Step F2: obtaining the rated excavation capacity corresponding to the excavation equipment whose action category undergoes a preset change.

[0201] The embodiment of the present application can first use the mining equipment model classification model to determine the target model of the mining equipment; then, according to the target model, search in the mapping relationship between the model and the rated mining capacity to obtain the rated mining capacity corresponding to the mining equipment with a preset change in action category.

[0202] Step F3: Determine the real-time cumulative excavation volume corresponding to the excavation block according to the effective excavation times and the rated excavation capacity.

[0203] Previously, the target tracking technology has been used to assign device identification to the excavation equipment in the frame image of the excavation block. Steps F1 and F2 can determine the target device identification corresponding to the excavation equipment whose action category has undergone a preset change, as well as the effective excavation times and rated excavation capacity corresponding to the target device identification. On this basis, the real-time cumulative excavation volume corresponding to the target device identification can be determined according to the effective excavation times and rated excavation capacity corresponding to the target device identification. In addition, the real-time cumulative excavation volumes corresponding to all target device identifications can be accumulated to obtain the real-time cumulative excavation volume corresponding to the entire excavation block.

[0204] In one example of the present application, the real-time cumulative excavation volume corresponding to the excavation block can be set to 0 at a set start time. Subsequently, the real-time cumulative excavation volume can be updated according to the update of the valid excavation times identified by any target device to obtain a continuously updated real-time cumulative excavation volume. Therefore, the real-time monitoring and analysis process of the excavation volume during the construction period can be a process of continuously updating the real-time cumulative excavation volume according to the update of the valid excavation times identified by any target device. The set start time can be determined by those skilled in the art according to actual application requirements. For example, the set start time can be the start time of the excavation operation of the excavation block, or the set start time can be the start time of a natural day.

[0205] In practical applications, if there are multiple excavation blocks in a hydropower project, the real-time cumulative excavation volume corresponding to each excavation block can be monitored separately according to natural days. After a natural day, the real-time cumulative excavation volumes corresponding to multiple excavation blocks can be merged. It can be understood that the embodiments of the present application do not limit the specific fusion method.

[0206] As for the process of determining the second excavation volume of the excavation site within the preset time period based on the video stream data of the excavation site within the preset time period, since it is similar to the process of determining the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block, it will not be described here and can be referred to each other.

[0207] In one example, the start time and the end time of a preset time period can be determined, and the data segments within the preset time period can be screened out from the recorded data of the real-time cumulative excavation volume of the excavation site. The real-time cumulative excavation volume of all excavation blocks in the excavation site can be summarized to obtain the recorded data of the real-time cumulative excavation volume of the excavation site.

[0208] If the real-time cumulative excavation volume is recorded on a natural day basis (the real-time cumulative excavation volume of each natural day is recorded from zero), the data fragments within this time period are integrated and calculated. For example, if the preset time period is from the 3rd to the 5th day, the real-time cumulative excavation volume of the 3rd natural day, the 4th natural day and the 5th natural day are obtained respectively, and the real-time cumulative excavation volume of the 3rd natural day, the 4th natural day and the 5th natural day are accumulated, and the accumulated result is the second excavation volume within the preset time period. If the real-time cumulative excavation volume is recorded in the form of cumulative time intervals (at the beginning of the excavation work of the excavation block, the real-time cumulative excavation volume is set to zero), it can be obtained by calculating the difference in the cumulative values ​​of the corresponding time intervals in the time period. For example, it is known that the real-time cumulative excavation volume is cumulatively recorded from the beginning of the project. If the preset time period is from the 3rd day to the 5th day, first obtain the real-time cumulative excavation volume at the end of the 5th day, and then subtract the real-time cumulative excavation volume at the end of the 2nd day. The difference is the second excavation volume in the preset time period. In summary, the embodiment of the present application provides a real-time monitoring and analysis process of excavation volume during the construction period executed by a computer, the process first acquires video stream data of the excavation block, the video stream data specifically includes: a plurality of frame images arranged in time; then, target detection is performed on the frame images to obtain the equipment category (dump equipment and excavation equipment) and the detection frame corresponding to the construction machinery; then, according to the equipment category and the detection frame, the construction machinery in the plurality of frame images is tracked to obtain the equipment identification corresponding to the construction machinery; thereafter, an equipment matching pair is determined according to the equipment category, the equipment identification and the detection frame, the equipment matching pair specifically represents an excavation equipment and a target dump equipment matched thereto; then, according to the equipment category, the equipment identification and the detection frame, the action of the construction machinery is identified to obtain the corresponding action category; further, in the event that the action category of the excavation equipment in the equipment matching pair undergoes a preset change, the real-time cumulative excavation volume corresponding to the excavation block is determined according to the rated excavation capacity of the excavation equipment. The above process of the embodiment of the present application performs real-time monitoring of the excavation volume during the construction period according to the objective law of the cooperation between the excavation equipment and the dump equipment during the construction process. The above objective law specifically includes: each time the excavation equipment in the equipment matching pair completes a specific sequence of action categories, it is considered that an effective excavation is completed, and there is a fixed corresponding relationship between the excavation volume each time and the rated excavation capacity of the excavation equipment. Therefore, the embodiment of the present application can combine the monitoring results of the preset changes in the action categories of the excavation equipment in the equipment matching pair to accurately determine the actual occurrence of each effective excavation, thereby determining the real-time cumulative excavation volume corresponding to the excavation block by accumulating the rated excavation capacity.

[0209] Since the above-mentioned objective laws of the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period based on the intrinsic logical connection between the actual process of the excavation and transportation operations at the construction site reflected by the video stream data and the equipment performance parameters. Therefore, the embodiments of the present application can realize real-time monitoring of the excavation volume during the construction period at a lower cost and faster speed based on the analysis of the video stream data. In other words, the embodiments of the present application can save the manpower cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation volume.

[0210] Furthermore, since the above-mentioned objective laws of the embodiments of the present application do not involve subjective assumptions and complex manual measurement and estimation, the embodiments of the present application can effectively improve the accuracy of real-time measurement of excavation volume during the construction period.

[0211] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0212] Based on the above embodiment, this embodiment also provides a device for calculating the whole process of excavation volume during construction period based on multi-source data fusion. Figure 4 The device may specifically include: an excavated model determination module 401, a first excavation amount determination module 402, a second excavation amount determination module 403, a correction coefficient determination module 404, a real-time excavation amount determination module 405 and a correction module 406.

[0213] The excavated model determination module 401 is used to determine the three-dimensional model of the excavated part of the excavation site according to the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation; the excavation site includes: at least one excavation block; The first excavation amount determination module 402 is used to determine the first excavation amount of the excavation site within a preset time period according to the three-dimensional model of the excavated part of the excavation site; the first moment is the end moment of the preset time period; The second excavation amount determination module 403 is used to determine the second excavation amount of the excavation part within the preset time period according to the video stream data of the excavation part within the preset time period; A correction coefficient determination module 404 is used to determine a correction coefficient according to the first excavation amount and the second excavation amount; A real-time excavation volume determination module 405 is used to determine the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block; A correction module 406, configured to correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the excavation volume measurement result of the excavation block; Wherein, the video stream data includes: a plurality of frame images arranged in time; The real-time excavation amount determination module includes: The equipment matching module is used to determine the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: an excavating device and a matching target dump device; an action recognition module, configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, so as to obtain the action category corresponding to the construction machinery; The excavation volume calculation module is used to determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavation equipment when the action category of the excavation equipment in the equipment matching pair undergoes a preset change.

[0214] Optionally, the excavated model determination module includes: A grid interpolation module, used for performing grid interpolation on the point cloud data of the excavation site at the first moment, so as to obtain the actual excavation contour surface of the excavation site at the first moment; A first envelope determining module, used to determine a first envelope of the excavation site at the first moment according to an actual excavation contour surface of the excavation site at the first moment; A first Boolean operation module is used to perform a Boolean difference operation on a first envelope of the excavation site at a first moment and a terrain surface model of the excavation site before excavation to obtain an operation result; The model determination module is used to obtain a three-dimensional model of the excavated portion of the excavation site from the calculation results according to the spatial scope of the excavation site.

[0215] Optionally, the first excavation amount determination module includes: A meshing module, used to divide the three-dimensional model of the excavated part of the excavation site into mesh units; A unit volume determination module, used to determine the unit volume of the grid unit in the three-dimensional model of the excavated part of the excavation site; The unit volume summary module is used to summarize the unit volumes of all grid units in the three-dimensional model of the excavated part of the excavation site to obtain the first excavation volume of the excavation site within a preset time period.

[0216] Optionally, the second excavation amount determination module includes: An image analysis module is used to analyze the first frame of the video stream data within a preset time period in a time-ordered order to obtain a first device matching pair in the first frame of the image and an action category corresponding to the construction machinery in the first frame of the image; The second excavation volume calculation module is used to determine the second excavation volume of the excavation part within a preset time period based on the rated excavation capacity corresponding to the excavation equipment, when the action category of the excavation equipment in the first equipment matching pair undergoes a preset change.

[0217] Optionally, the device further comprises: The terrain surface model building module is used to build a terrain surface model of the excavation site before excavation based on the design information of the engineering object corresponding to the excavation site and using the contour lines and elevation points in the original terrain.

[0218] Optionally, the device further comprises: An over-excavation and under-excavation analysis module is used to determine the over-excavation and under-excavation of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the design volume of the first excavation block; An updating module is used to adjust the remaining excavation amount of the second excavation block in the (s+1)th excavation area according to the over-excavation or under-excavation situation of the first excavation block; s is a positive integer; the sth excavation area and the (s+1)th excavation area are two excavation areas that are geographically adjacent.

[0219] Optionally, the process of determining the design volume of the excavation block includes: According to the excavation contour line of the excavation part, an excavation surface model of the excavation part after excavation is established; Performing Boolean operation on the second envelope corresponding to the terrain surface model and the third envelope corresponding to the excavation surface model to obtain a three-dimensional model of the excavation site; According to the spatial range of the excavation block, a three-dimensional model of the excavation block is obtained from the three-dimensional model of the excavation site; The design volume of the excavation block is determined based on the three-dimensional model of the excavation block.

[0220] Optionally, the preset change in the action category of the excavating equipment in the equipment matching pair includes: the action category of the excavating equipment in the equipment matching pair changes in the order of excavating, lifting and rotating, unloading, empty bucket return, and excavating.

[0221] Optionally, the performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images includes: According to the first sampling rate, the plurality of frame images arranged in time are first sampled, and the detection frame area in the first sampling result is first convolutionally processed by using P first convolution modules, and the Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1; According to a second sampling rate, a second sampling is performed on the multiple frame images arranged in time, and a second convolution processing is performed on the detection frame area in the second sampling result using P second convolution modules, and the P second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel; Performing average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature; The action category corresponding to the construction machinery is determined according to the average pooling feature.

[0222] Optionally, the performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images further includes: The second convolution result output by the i-th second convolution module is fused with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the i+1-th first convolution module.

[0223] The embodiment of the present application further provides a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0224] The present application embodiment provides one or more machine-readable media, on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices, servers (clusters), etc.

[0225] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method for calculating the entire process of excavation volume during construction based on multi-source data fusion as described in any of the above embodiments.

[0226] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Figure 5 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.

[0227] For one embodiment, Figure 5An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the processor(s) 1302, a memory 1306 coupled to the control module 1304, an NVP (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0228] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0229] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0230] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0231] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0232] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include a double data rate type four synchronous dynamic random access memory.

[0233] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the input / output device(s) 1310 .

[0234] For example, non-volatile memory / storage 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0235] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.

[0236] The input / output device(s) 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0237] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip.

[0238] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0239] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0240] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0241] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0242] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0243] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0244] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0245] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0246] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0247] The above is a detailed introduction to a method and device for measuring the entire process of excavation volume during the construction period based on multi-source data fusion, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for calculating the whole process of excavation volume during construction period based on multi-source data fusion, characterized in that: The method comprises: Determine a three-dimensional model of the excavated portion of the excavation site based on point cloud data of the excavation site at the first moment and a terrain surface model of the excavation site before excavation; the excavation site includes: at least one excavation block; Determining a first excavation amount of the excavation site within a preset time period according to the three-dimensional model of the excavated portion of the excavation site; the first moment is the end moment of the preset time period; Determining a second excavation amount of the excavation portion within the preset time period according to the video stream data of the excavation portion within the preset time period; determining a correction coefficient according to the first excavation amount and the second excavation amount; Determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block; According to the correction coefficient, the real-time cumulative excavation volume of the excavation block is corrected to obtain the excavation volume measurement result of the excavation block; Wherein, the video stream data includes: a plurality of frame images arranged in time; Determining the real-time cumulative excavation volume of the excavation block according to the video stream data of the excavation block includes: According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, determine the equipment matching pair in the multiple frame images arranged in time; the equipment matching pair includes: an excavating equipment and a matching target dump equipment; According to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, the action recognition of the construction machinery in the multiple frame images arranged in time is performed to obtain the action category corresponding to the construction machinery; When a preset change occurs to the action category of the excavating equipment in the equipment matching pair, it is considered that the excavating equipment in the equipment matching pair completes one excavation, and the real-time cumulative excavation volume of the excavation block is determined according to the rated excavation capacity corresponding to the excavating equipment.

2. The method according to claim 1, characterized in that: The step of determining the three-dimensional model of the excavated portion of the excavation site based on the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation includes: Performing grid interpolation on the point cloud data of the excavation site at the first moment to obtain the actual excavation contour surface of the excavation site at the first moment; Determine a first envelope of the excavation site at the first moment according to an actual excavation contour surface of the excavation site at the first moment; Perform a Boolean difference operation on the first envelope of the excavation site at the first moment and the terrain surface model of the excavation site before excavation to obtain an operation result; According to the spatial range of the excavation site, a three-dimensional model of the excavated portion of the excavation site is obtained from the calculation results.

3. The method according to claim 1, characterized in that: The step of determining a first excavation amount of the excavation site within a preset time period based on the three-dimensional model of the excavated portion of the excavation site includes: Divide the three-dimensional model of the excavated portion of the excavation site into grid units; Determine the cell volume of the grid cells in the three-dimensional model of the excavated portion of the excavation site; The unit volumes of all grid units in the three-dimensional model of the excavated portion of the excavation site are summarized to obtain the first excavation volume of the excavation block within a preset time period.

4. The method according to claim 1, characterized in that: The determining, based on the video stream data of the excavation location within the preset time period, a second excavation amount of the excavation location within the preset time period comprises: Analyze the first frame of the video stream data within the preset time period in a chronological order, and obtain a first device matching pair in the first frame of the image and an action category corresponding to the construction machinery in the first frame of the image; When a preset change occurs to the action category of the excavating equipment in the first equipment matching pair, it is considered that the excavating equipment in the first equipment matching pair completes one excavation, and the second excavation amount of the excavation part within the preset time period is determined according to the rated excavation capacity corresponding to the excavating equipment.

5. The method according to claim 1, characterized in that The method further comprises: According to the design information of the engineering object corresponding to the excavation site, the terrain surface model of the excavation site before excavation is established using the contour lines and elevation points in the original terrain.

6. The method according to claim 1, characterized in that The method further comprises: Determine the over-excavation and under-excavation situation of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the design volume of the first excavation block; According to the over-excavation or under-excavation situation of the first excavation block, the remaining excavation amount of the second excavation block in the (s+1)th excavation area is adjusted; s is a positive integer; the sth excavation area and the (s+1)th excavation area are two excavation areas that are geographically adjacent.

7. The method according to any one of claims 1 to 6, characterized in that: The process of determining the design volume of the excavation block includes: According to the excavation contour line of the excavation part, an excavation surface model of the excavation part after excavation is established; Performing Boolean operation on the second envelope corresponding to the terrain surface model and the third envelope corresponding to the excavation surface model to obtain a three-dimensional model of the excavation site; According to the spatial range of the excavation block, a three-dimensional model of the excavation block is obtained from the three-dimensional model of the excavation site; The design volume of the excavation block is determined based on the three-dimensional model of the excavation block.

8. The method according to any one of claims 1 to 6, characterized in that: The preset change in the action category of the excavating equipment in the equipment matching pair includes: the action category of the excavating equipment in the equipment matching pair changes in the order of excavating, lifting and rotating, unloading, empty bucket returning, and excavating.

9. The method according to any one of claims 1 to 6, characterized in that: The step of performing action recognition on the construction machinery in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images comprises: According to the first sampling rate, the plurality of frame images arranged in time are first sampled, and the detection frame area in the first sampling result is first convolutionally processed by using P first convolution modules, and the Pth first convolution module outputs a first convolution feature; P is a positive integer greater than 1; According to a second sampling rate, a second sampling is performed on the multiple frame images arranged in time, and a second convolution processing is performed on the detection frame area in the second sampling result using P second convolution modules, and the Pth second convolution module outputs a second convolution feature; the second sampling rate is greater than the first sampling rate; the first convolution module uses a first convolution kernel, the second convolution module uses a second convolution kernel, and the number of output channels of the first convolution kernel is greater than the number of output channels of the second convolution kernel; Performing average pooling processing on the first convolution feature and the second convolution feature to obtain an average pooling feature; The action category corresponding to the construction machinery is determined according to the average pooling feature.

10. The method according to claim 9, characterized in that The step of performing action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images further includes: The second convolution result output by the i-th second convolution module is fused with the first convolution result output by the i-th first convolution module to obtain a fusion result i, and the fusion result i is used as the input of the i+1-th first convolution module.

11. A device for calculating the whole process of excavation volume during construction period based on multi-source data fusion, characterized in that: The device comprises: The excavated model determination module is used to determine the three-dimensional model of the excavated part of the excavation site based on the point cloud data of the excavation site at the first moment and the terrain surface model of the excavation site before excavation; the excavation site includes: at least one excavation block; A first excavation amount determination module is used to determine a first excavation amount of the excavation site within a preset time period according to a three-dimensional model of the excavated portion of the excavation site; the first moment is the end moment of the preset time period; A second excavation amount determination module, used to determine a second excavation amount of the excavation part within a preset time period according to the video stream data of the excavation part within a preset time period; A correction coefficient determination module, used to determine the correction coefficient according to the first excavation amount and the second excavation amount; A real-time excavation volume determination module is used to determine the real-time cumulative excavation volume of the excavation block based on the video stream data of the excavation block; A correction module, used to correct the real-time cumulative excavation volume of the excavation block according to the correction coefficient to obtain the excavation volume measurement result of the excavation block; Wherein, the video stream data includes: a plurality of frame images arranged in time; The real-time excavation amount determination module includes: The equipment matching module is used to determine the equipment matching pairs in the multiple frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame images; the equipment matching pairs include: an excavating device and a matching target dump device; an action recognition module, configured to perform action recognition on the construction machinery in the plurality of frame images arranged in time according to the equipment category, equipment identification and detection frame corresponding to the construction machinery in the frame image, so as to obtain the action category corresponding to the construction machinery; The excavation volume calculation module is used to determine the real-time cumulative excavation volume of the excavation block according to the rated excavation capacity corresponding to the excavation equipment when the action category of the excavation equipment in the equipment matching pair undergoes a preset change.

12. The device according to claim 11, characterized in that The excavated model determination module includes: A grid interpolation module, used for performing grid interpolation on the point cloud data of the excavation site at the first moment, so as to obtain the actual excavation contour surface of the excavation site at the first moment; A first envelope determining module, used to determine a first envelope of the excavation site at the first moment according to an actual excavation contour surface of the excavation site at the first moment; A first Boolean operation module is used to perform a Boolean difference operation on a first envelope of the excavation site at a first moment and a terrain surface model of the excavation site before excavation to obtain an operation result; The model determination module is used to obtain a three-dimensional model of the excavated portion of the excavation site from the calculation results according to the spatial scope of the excavation site.

13. The device according to claim 11, characterized in that The first excavation amount determination module includes: A meshing module, used to divide the three-dimensional model of the excavated part of the excavation site into mesh units; A unit volume determination module, used to determine the unit volume of the grid unit in the three-dimensional model of the excavated part of the excavation site; The unit volume summary module is used to summarize the unit volumes of all grid units in the three-dimensional model of the excavated part of the excavation site to obtain the first excavation volume of the excavation site within a preset time period.

14. The device according to claim 11, characterized in that The second excavation amount determination module includes: An image analysis module, used to analyze the first frame of the video stream data within a preset time period in a time-ordered order, to obtain a first device matching pair in the first frame of the image and an action category corresponding to the construction machinery in the first frame of the image; The second excavation volume calculation module is used to determine the second excavation volume of the excavation part within a preset time period based on the rated excavation capacity corresponding to the excavation equipment, when the action category of the excavation equipment in the first equipment matching pair undergoes a preset change.

15. The device according to claim 11, characterized in that The device also includes: An over-excavation and under-excavation analysis module is used to determine the over-excavation and under-excavation of the first excavation block according to the real-time cumulative excavation volume of the first excavation block in the s-th excavation area and the design volume of the first excavation block; An updating module is used to adjust the remaining excavation amount of the second excavation block in the (s+1)th excavation area according to the over-excavation or under-excavation situation of the first excavation block; s is a positive integer; the sth excavation area and the (s+1)th excavation area are two excavation areas that are geographically adjacent.

16. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 10.

17. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 10.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images