A Construction Progress Prediction Method and Device Based on Long-Term Video Analysis

Abstract

An embodiment of the present application provides a construction progress prediction method and device based on long-term video analysis. The method specifically includes: analyzing relevant operation videos of construction operations to obtain the equipment category, equipment identifier, and detection frame of the target construction machinery; cropping the relevant operation videos from the spatial dimension and the time dimension to obtain the construction operation videos of each target construction machinery; segmenting the construction operation videos of each target construction machinery; determining the factual duration samples of the action categories according to multiple construction operation segments arranged in chronological order; generating the random duration samples corresponding to the action categories according to the factual duration samples corresponding to the action categories; assigning values to the construction process parameters of the target construction machinery in the construction simulation model according to the duration values corresponding to the random duration samples, and performing the simulation of the construction simulation model. The embodiment of the present application can effectively improve the accuracy of construction progress prediction.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of construction progress prediction, and particularly to a construction progress prediction method and device based on long-term video analysis. Background Art

[0002] In engineering construction, accurate and efficient construction progress prediction is crucial. With accurate progress prediction, the engineering team can reasonably allocate resources such as manpower, material resources, and financial resources according to the actual needs of each construction stage, improve the resource utilization efficiency, and reduce costs. For example, in the earthwork excavation stage, an appropriate amount of construction machinery and construction personnel can be arranged according to the progress prediction to avoid resource idleness or shortage.

[0003] Currently, construction progress prediction mainly relies on construction simulation technology. Construction simulation technology can formulate construction progress, reasonably allocate construction resources, and discover potential construction problems in advance by simulating the engineering construction process in a computer, providing strong guidance for engineering construction decisions. In actual operation, the construction of the construction simulation model is the core link, and the value of the construction simulation parameters is the key factor determining the accuracy of the model. In related technologies, the value of the construction simulation parameters is usually assigned based on the construction experience of similar projects. Taking the construction process parameters of construction machinery as an example, in related technologies, the unit excavation duration of excavation equipment is usually set to a fixed value that conforms to construction experience.

[0004] However, in actual construction, the excavation equipment is affected by geological conditions (such as different textures of soft soil, hard rock, etc.), site space (width, openness, etc.), and the working conditions of the equipment itself (new or old degree, wear condition, etc.), so its construction process parameters are usually in a changing state. If these construction process parameters are set to fixed values, the obtained construction process parameters are likely to deviate from the actual situation, thereby affecting the accuracy of construction progress prediction. Summary of the Invention

[0005] The embodiments of the present application provide a construction progress prediction method based on long-term video analysis, which can effectively improve the accuracy of construction progress prediction.

[0006] Correspondingly, the embodiments of the present application also provide a construction progress prediction device based on long-term video analysis, an electronic device, and a machine-readable medium to ensure the implementation and application of the above method.

[0007] To solve the above problems, the embodiments of the present application disclose a construction progress prediction method based on long-term video analysis, including:

[0008] Analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is the historical operation video of the construction operation, and / or, the relevant operation video is the historical operation video of the same category of operations of the construction operation;

[0009] According to the equipment identification and detection frame of the target construction machinery, crop the relevant operation video from the spatial dimension and the time dimension to obtain the construction operation video of each target construction machinery;

[0010] Use an action segmentation model to segment the construction operation video of each target construction machinery to obtain multiple construction operation segments arranged in chronological order; among them, two adjacent construction operation segments in time correspond to different action categories;

[0011] Determine the duration data of the action category according to the multiple construction operation segments arranged in chronological order;

[0012] According to the duration data, determine the factual duration sample corresponding to the action category; the factual duration sample includes: a data set composed of multiple duration values generated in the frame images at different moments of the action category;

[0013] Generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category;

[0014] Assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration values corresponding to the random duration sample; the construction process parameters include: the unit operation duration corresponding to the target construction machinery;

[0015] Perform simulation of the construction simulation model according to the construction process parameters of the target construction machinery to obtain the construction progress prediction result.

[0016] The embodiments of the present application also disclose a construction progress prediction device based on long-term video analysis, and the device includes:

[0017] A video analysis module, configured to analyze the relevant operation video of the construction operation to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is the historical operation video of the construction operation, and / or, the relevant operation video is the historical operation video of the same category of operations of the construction operation;

[0018] A video cropping module, configured to crop the relevant operation video from the spatial dimension and the time dimension according to the equipment identifier and the detection frame of the target construction machinery, so as to obtain the construction operation video of each target construction machinery;

[0019] An action segmentation module, configured to use an action segmentation model to segment the construction operation video of each target construction machinery, so as to obtain a plurality of construction operation segments arranged in chronological order; wherein, two adjacent construction operation segments in time correspond to different action categories;

[0020] A duration data determination module, configured to determine the duration data of the action category according to a plurality of construction operation segments arranged in chronological order;

[0021] A factual sample determination module, configured to determine a factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set composed of multiple duration values generated in the frame images of the action category at different moments;

[0022] A randomness sample generation module, configured to generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category;

[0023] An assignment module, configured to assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration values corresponding to the random duration sample; the construction process parameters include: the unit operation duration corresponding to the target construction machinery;

[0024] A construction simulation module, configured to perform a simulation of the construction simulation model according to the construction process parameters of the target construction machinery, so as to obtain a construction progress prediction result.

[0025] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor is caused to execute the method as described in the embodiment of the present application.

[0026] An embodiment of the present application further discloses a machine-readable medium, on which executable code is stored, and when the executable code is executed, a processor is caused to execute the method as described in the embodiment of the present application.

[0027] An embodiment of the present application further discloses a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the foregoing method is implemented.

[0028] The embodiments of the present application include the following advantages:

[0029] The technical solution of the embodiment of the present application starts from the construction operation video and obtains the construction progress prediction result through multi-step processing. First, analyze the relevant operation videos of the construction operation (including the historical operation videos of the construction operation and / or the historical operation videos of the same type of operation, and the duration is greater than the duration threshold) to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; then crop the construction operation videos of each target construction machinery from the spatio-temporal dimension; next, use the action segmentation model to segment the construction operation videos of each target construction machinery to obtain a plurality of construction operation segments arranged in chronological order, determine the duration data according to the construction operation segments, and further obtain factual and random duration samples; use the duration values corresponding to the random duration samples to assign values to the construction process parameters (such as the unit operation duration) of the target construction machinery in the construction simulation model; finally, perform construction simulation model simulation to obtain the construction progress prediction result.

[0030] The embodiment of the present application determines the parameter values of the construction process parameters based on the construction operation-related videos. Since the construction operation-related videos of the embodiment of the present application record various actions of the target construction machinery during the construction process, based on the analysis of the construction operation-related videos, real data such as action categories and factual duration samples can be obtained from the construction operation-related videos, and random duration samples can be generated by processing these real data. Since the random duration samples contain various uncertainties and change factors during the construction process, using them to assign values to the construction process parameters can make the parameter values of the construction process parameters closer to the real state of the construction operation site. In this way, the construction simulation model including the construction process parameters can more accurately simulate the construction progress based on the parameters closer to the real state. Therefore, the embodiment of the present application can effectively improve the accuracy of construction progress prediction. Brief Description of the Drawings

[0031] Figure 1 is a schematic flowchart of the steps of a construction progress prediction method based on long-term video analysis according to an embodiment of the present application;

[0032] Figure 2 is a schematic structural diagram of a denoising network model according to an embodiment of the present application;

[0033] Figure 3 is a schematic flowchart of the steps of a method for automatically constructing a construction simulation model based on video images according to an embodiment of the present application;

[0034] FIG. 4(a) and FIG. 4(b) are respectively schematic diagrams of the modeling of the construction process according to an embodiment of the present application;

[0035] Figure 5 is a schematic diagram of a construction process simulation model of an excavation block according to an embodiment of the present application;

[0036] Figure 6 It is a schematic diagram of a construction simulation model according to an embodiment of the present application;

[0037] Figure 7 It is a schematic structural diagram of a construction progress prediction device based on long-term video analysis according to an embodiment of the present application;

[0038] Figure 8 It is a schematic structural diagram of a device provided by an embodiment of the present application. Detailed implementation manners

[0039] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0040] The embodiments of the present application can be applied to the hydropower engineering industry and are used for predicting the construction progress of construction operations based on computer simulation. Construction progress prediction refers to the activity of using scientific methods and tools, comprehensively considering various factors such as resource input, process technology, environmental conditions, and historical data during the construction process, estimating the time progress and completion status of each stage and the overall situation of construction operations, and giving the time nodes of each construction process, progress deviation analysis, risk warning, etc., providing decision-making basis for construction resource allocation, construction period planning, and cost control, and ensuring the timely and efficient completion of the project.

[0041] Currently, usually according to the construction simulation parameters of the construction simulation model, the simulation of the construction simulation model is carried out, and then the construction progress prediction is realized. In the related art, the values of the construction simulation parameters are usually assigned based on the construction experience of similar projects. Taking the construction process parameters of construction machinery as an example, in the related art, the unit excavation duration of the excavation equipment is usually set to a fixed value that conforms to the construction experience.

[0042] However, in actual construction, the excavation equipment will be affected by geological conditions (such as different textures of soft soil, hard rock, etc.), site space (width, openness, etc.), and the working conditions of the equipment itself (new and old degree, wear condition, etc.). Therefore, its construction process parameters are usually in a changing state. If these construction process parameters are set as fixed values, the obtained construction process parameters are likely to deviate from the actual situation, thereby affecting the accuracy of the construction progress prediction.

[0043] In view of the technical problem of low accuracy of construction progress prediction in the related art, the embodiments of the present application provide a construction progress prediction method based on long-term video analysis. The method specifically includes the following steps:

[0044] Analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; the above target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the above relevant operation videos is greater than the duration threshold; the above relevant operation videos are the historical operation videos of the above construction operation, and / or, the above relevant operation videos are the historical operation videos of the same category of operations of the above construction operation;

[0045] According to the equipment identification and detection frame of the target construction machinery, crop the above relevant operation videos from the spatial dimension and the time dimension to obtain the construction operation videos of each target construction machinery;

[0046] Use the action segmentation model to segment the construction operation videos of each target construction machinery to obtain multiple construction operation segments arranged in chronological order; among them, two adjacent construction operation segments in time correspond to different action categories;

[0047] Determine the duration data of the action category according to the multiple construction operation segments arranged in chronological order;

[0048] According to the above duration data, determine the factual duration sample corresponding to the action category; the above factual duration sample includes: a data set composed of multiple duration values generated in the frame images at different moments of the action category;

[0049] Generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category;

[0050] Assign values to the construction process parameters of the target construction machinery in the construction simulation model according to the duration values corresponding to the above random duration sample; the above construction process parameters include: the unit operation duration corresponding to the above target construction machinery;

[0051] Perform simulation of the construction simulation model according to the construction process parameters of the above target construction machinery to obtain the construction progress prediction result.

[0052] Embodiments of this application start from construction operation videos and obtain construction progress prediction results through multi-step processing. First, analyze relevant operation videos of the construction operation (including historical operation videos of the construction operation and / or historical operation videos of the same type of operation, and the duration is greater than the duration threshold) to obtain the equipment category, equipment identifier, and detection frame of the target construction machinery; then crop the construction operation videos of each target construction machinery from the spatio-temporal dimension; next, use an action segmentation model to segment the construction operation videos of each target construction machinery to obtain multiple construction operation segments arranged in chronological order, determine the duration data based on the construction operation segments, and further obtain factual and random duration samples; use the duration values corresponding to the random duration samples to assign values to the construction process parameters (such as the unit operation duration) of the target construction machinery in the construction simulation model; finally, perform a construction simulation model simulation to obtain the construction progress prediction result.

[0053] Embodiments of this application determine the parameter values of construction process parameters based on relevant videos of construction operations. Since the relevant videos of construction operations in embodiments of this application record various actions of the target construction machinery during the construction process, based on the analysis of the relevant videos of construction operations, real data such as action categories and factual duration samples can be obtained from the relevant videos of construction operations, and random duration samples can be generated by processing these real data. Since the random duration samples contain various uncertainties and changing factors during the construction process, using them to assign values to the construction process parameters can make the parameter values of the construction process parameters closer to the real state of the construction operation site. In this way, the construction simulation model including the construction process parameters can more accurately simulate the construction progress based on parameters closer to the real state. Therefore, embodiments of this application can effectively improve the accuracy of construction progress prediction.

[0054] Method Embodiment 1

[0055] Reference Figure 1 shows a schematic flowchart of the steps of a construction progress prediction method based on long-term video analysis according to an embodiment of this application. The method specifically includes the following steps:

[0056] Step 101, analyze relevant operation videos of the construction operation to obtain the equipment category, equipment identifier, and detection frame of the target construction machinery; the above target construction machinery can be the construction machinery that restricts the progress of the construction process; the duration of the above relevant operation videos is greater than the duration threshold; the above relevant operation videos can be the historical operation videos of the above construction operation, and / or, the above relevant operation videos can be the historical operation videos of the same type of operation as the above construction operation;

[0057] Step 102: Crop the above-mentioned relevant operation videos from the spatial and temporal dimensions according to the equipment identifier and detection frame of the target construction machinery to obtain the construction operation videos of each target construction machinery;

[0058] Step 103: Use an action segmentation model to segment the construction operation videos of each target construction machinery to obtain multiple construction operation segments arranged in chronological order; among them, two adjacent construction operation segments in time correspond to different action categories;

[0059] Step 104: Determine the duration data of the action categories according to the multiple construction operation segments arranged in chronological order;

[0060] Step 105: Determine the factual duration samples corresponding to the action categories according to the above-mentioned duration data; the above-mentioned factual duration samples specifically include: a data set composed of multiple duration values generated in the frame images at different moments of the action category;

[0061] Step 106: Generate random duration samples corresponding to the action categories according to the factual duration samples corresponding to the action categories;

[0062] Step 107: Assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration values corresponding to the above-mentioned random duration samples; the above-mentioned construction process parameters include: the unit operation duration corresponding to the above-mentioned target construction machinery;

[0063] Step 108: Perform simulation of the construction simulation model according to the construction process parameters of the above-mentioned target construction machinery to obtain the construction progress prediction result.

[0064] Figure 1 The method shown above can be used for predicting the construction progress of construction operations based on computer simulation. In the engineering field, construction operations refer to a series of organized and planned production activities carried out at the construction site of a project to meet the requirements of the engineering design by arranging various construction resources (such as labor, materials, construction machinery, etc.) according to specific construction techniques, processes, and specifications. For example, in the scenario of a hydropower project, examples of construction operations can include: slope excavation operations, dam pouring operations, or tunnel excavation operations, etc. It can be understood that the embodiments of the present application do not limit the specific construction operations.

[0065] A construction operation usually includes multiple construction processes. A construction process refers to several specific construction steps or operation processes that are interrelated and restricted obtained by dividing the entire construction operation according to a certain construction sequence and construction method in order to complete a certain engineering task during the construction process.

[0066] In the embodiments of the present application, a long-duration video may refer to video content with a duration greater than a duration threshold. The duration threshold can be determined by those skilled in the art according to actual needs. For example, the duration threshold is 30 minutes, etc.

[0067] In step 101, the target construction machine may be a construction machine that restricts the progress of the construction process. The determination process of the target construction machine is as follows:

[0068] If only one construction machine is involved in the construction process, then this construction machine is the target construction machine that restricts the progress of the construction process. For example, when using only one construction machine, a rock drilling jumbo, for drilling operations, the rock drilling jumbo can be used as the target construction machine for drilling operations.

[0069] If the construction process is completed by the cooperation of multiple construction machines, then the construction machine that provides services from the perspective of queuing theory is the target construction machine that restricts the progress of the construction process. For example, when using excavation equipment and dump trucks to complete the excavation and loading operations of slope earthwork, the excavation equipment provides excavation and loading services, and the dump trucks queue up to wait for the loading service of the excavator. Accordingly, the excavation equipment is considered the target construction machine that restricts the progress of the earthwork excavation and loading operations.

[0070] The embodiments of the present application can collect relevant operation videos of the construction operation according to the target construction machine involved in the construction operation and the operation category of the construction operation. The embodiments of the present application do not limit the number of relevant operation videos collected. The relevant operation videos can be one or more.

[0071] Specifically, it is possible to first search in the video database according to the operation category of the construction operation to obtain candidate videos corresponding to the operation category. The video database includes various relevant data sets such as the internal database of the construction enterprise and the industry general database. Then, according to the keywords of the target construction video, the candidate videos are screened to obtain relevant operation videos.

[0072] The embodiments of the present application can use object detection technology and object tracking technology to analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification, and detection frame of the target construction machine.

[0073] Object detection technology and object tracking technology are designed to identify different objects from the frame images of the relevant operation videos and continuously track them to determine the position, category, and identification number of the objects in the frame images. The frame images can represent any frame images of the relevant operation videos.

[0074] Specifically for step 101 of the embodiments of this application, the targets of object detection technology and object tracking technology can specifically be: target construction machinery. By using the already trained object detection model and object tracking model (such as a convolutional neural network model based on deep learning, etc.), to process the frame images of relevant operation videos, the object detection model and object tracking model will scan and continuously locate the areas where multiple construction machinery are located in the frame images according to the learned feature patterns of various construction machinery. This area is presented in the form of a detection box. At the same time, it can judge the equipment category to which the construction machinery in each detection box belongs, such as excavation equipment or dump equipment, etc., as well as the equipment identifier corresponding to the construction machinery in each detection box. The equipment identifier can be an identification number, such as 1, 2, 3, etc. The detection box can be an external rectangular box corresponding to the construction machinery in the frame image.

[0075] In one example, the object detection model specifically includes: a feature extraction unit, a feature fusion unit, and a detection unit connected in series in sequence. The input of the object detection model is the frame image of the relevant operation video, and after calculation, it outputs the detection box and category information of the target construction machinery on each frame image. The output result of the object detection model can be {(machine class,xmin, xmax, ymin, ymax)}.

[0076] Among them, machine class represents the category information of the target construction machinery, such as "dump equipment" or "excavation equipment", etc., which is used to distinguish different categories of construction machinery.

[0077] "xmin" and "xmax" are respectively the coordinate values of the leftmost and rightmost sides of the detection box of the target construction machinery in the horizontal direction (x-axis) of the frame image; "ymin" and "ymax" are respectively the coordinate values of the uppermost and lowermost sides of the detection box of the target construction machinery in the vertical direction (y-axis) of the frame image. These four coordinate values jointly determine a rectangular detection box.

[0078] The feature extraction unit is used to extract image features from the frame image.

[0079] The feature fusion unit is used to effectively fuse image features of different levels and different types, and the obtained fused image features can make the feature information more comprehensive and accurate, so as to enhance the expression ability of the construction machinery features.

[0080] The detection unit is used to judge the category and determine the position of the construction machinery in the frame image according to the fused image features.

[0081] Object tracking technology is a technology that, in a sequence of frame images, for a specific object (the construction machinery in the embodiments of the present invention), determines the position of the object in different frame images through a series of algorithms and models, and continuously estimates and updates the state of the object (including position, speed, movement direction, etc.). Its core is to achieve accurate identification and continuous tracking of the object in a complex image environment and under continuously changing object states. The format of the object tracking result output by the object tracking model can be: {(machine class, machineID, xmin, xmax, ymin, ymax)}, where machineID represents the equipment identifier corresponding to the construction machinery.

[0082] In step 102, the equipment identifier of the target construction machinery can be used to locate the frame number of the target construction machinery in the frame images of the relevant operation video, and this frame number can represent the time dimension information of the target construction machinery in the relevant operation video. The detection frame of the target construction machinery can be used to determine the spatial range of the target construction machinery in the frame image, and this detection frame can represent the spatial dimension information of the target construction machinery in the frame image. In this way, by combining the time dimension information and the spatial dimension information, the construction operation video of each target construction machinery can be cropped.

[0083] In a specific implementation, the process of cropping the relevant operation video from the spatial dimension and the time dimension according to the equipment identifier and the detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery specifically includes:

[0084] Step 121: Determine the start frame image and the end frame image of each target construction machinery in the relevant operation video;

[0085] Step 122: Determine the maximum bounding box of each target construction machinery according to the detection frames of each target construction machinery in the start frame image, the middle frame images, and the end frame image;

[0086] Step 123: Crop the relevant operation video from the time dimension according to the start frame image and the end frame image, and crop the relevant operation video from the spatial dimension according to the maximum bounding box to obtain the construction operation video of each target construction machinery.

[0087] In step 121, for each target construction machinery with a different machineID, according to the object tracking result, analyze the frame numbers corresponding to the start frame image of the first appearance and the end frame image of the final disappearance of the target construction machinery in the relevant operation video. For example, the frame number of the start frame image is m, and the frame number of the end frame image is n.

[0088] In step 122, for each target construction machine with a different machine ID, calculate the maximum bounding box in the image frame of the relevant operation video.

[0089] Assume that the set of image frames in which the target construction machine appears in the long video is {i} (i >= m, i <= n, and i is a positive integer), and assume that the set of coordinates of the corresponding detection boxes is {(xmin i , xmax i , ymin i , ymax i ). Then the coordinate calculation method of the maximum bounding box is as follows:

[0090] Xmin = min{ xmin i}

[0091] Xmax = max{ xmax i}

[0092] Ymin = min{ ymin i}

[0093] Ymax = max{ ymax i}

[0094] Among them, Xmin and Xmax respectively represent the leftmost and rightmost coordinate values of the maximum bounding box in the horizontal direction of the frame image; Ymin and Ymax are respectively the uppermost and lowermost coordinate values of the maximum bounding box in the vertical direction of the frame image. These four coordinate values together determine the maximum bounding box of a rectangle.

[0095] In step 123, according to the m-th frame image to the n-th frame image and the coordinates of the maximum bounding box, crop the relevant operation video from the time dimension and the space dimension to obtain the construction operation video of each target construction machine.

[0096] There are differences between the relevant operation video and the construction operation video in terms of the number of frame images and the image size. In terms of the number of frame images, the construction operation video is cropped from the m-th frame to the n-th frame, and the number of frame images is n - m + 1, while the number of frame images of the relevant operation video is usually greater than that of the construction operation video. In terms of the image size, the construction operation video is cropped according to the coordinates of the maximum bounding box, usually only framing the target construction machine. Therefore, the image size of the construction operation video is usually smaller than that of the relevant operation video.

[0097] In step 103, use the action segmentation model to segment the construction operation video of each target construction machine to obtain multiple construction operation segments.

[0098] Among them, the action segmentation model is a model used to decompose the actions in a continuous video stream into different segments. Based on deep learning or other machine learning technologies, and trained with a large amount of video data with annotated actions, it can identify the changes and boundaries of actions in the video, thereby segmenting the complex action sequences in the video into relatively independent action categories.

[0099] The action segmentation model will analyze information such as the characteristics, speed, and direction of the construction machinery actions in the construction operation video according to the knowledge learned during its training, identify the start and end points of the actions, and divide the entire construction operation video into multiple shorter construction operation segments. Each construction operation segment represents a relatively complete construction operation action (abbreviated as action category) of the target construction machinery.

[0100] After action segmentation, for any two adjacent construction operation segments arranged in chronological order, the actions of the construction machinery they represent are different. For example, the previous construction operation segment may be the digging action of the excavation equipment, and the next construction operation segment adjacent to it is the action of the excavation equipment lifting the bucket and moving it above the transport vehicle. Such segmentation results help to clearly understand and analyze each specific operation step and its sequence of the construction machinery during the entire operation process.

[0101] Among them, the process of step 103 using the action segmentation model to segment the construction operation video of each target construction machinery specifically includes:

[0102] Step 131: Extract the three-dimensional temporal features of the construction operation video of each target construction machinery;

[0103] Step 132: Perform one-dimensional convolution processing, self-attention processing, and feed-forward processing on the three-dimensional temporal features in sequence to obtain the fusion region features corresponding to each target construction machinery;

[0104] Step 133: Use the denoising network model to determine the noise distribution of the fusion region features and the s-th time step, and determine the action category of the s-th time step;

[0105] Step 134: According to the action categories corresponding to each time step, segment the construction operation video of each target construction machinery into multiple construction operation segments arranged in chronological order.

[0106] In step 131, I3D (Inflated 3D ConvNets) can be used to extract the three-dimensional temporal features of the construction operation video of each target construction machinery. The three-dimensional temporal features refer to the data features that change over time in three-dimensional space. These features are usually used to describe and analyze the dynamic change process of an object in three-dimensional space.

[0107] In one implementation, the three-dimensional temporal features specifically include: RGB (Red Green Blue) features and optical flow features. Among them, RGB features can capture the color and texture information of the target construction machinery, and optical flow features can effectively characterize the motion information of the target construction machinery. The combination of the two can comprehensively and accurately describe the appearance and motion state of the target construction machinery in the video.

[0108] In practical applications, the process of extracting the 3D temporal features of the construction operation video of each target construction machine specifically includes:

[0109] Step 1311: using a sliding window with a preset window size and a preset step length, obtaining multiple video clips from the construction operation video of each target construction machine;

[0110] Step 1312: extracting RGB features and optical flow features corresponding to the multiple video clips respectively;

[0111] Step 1313: fuse the RGB features and optical flow features corresponding to the multiple video clips to obtain the three-dimensional temporal features corresponding to the multiple video clips.

[0112] In step 1312, these acquired video clips can be input into the I3D network, and each video clip can be processed separately by using the powerful convolution capability of the I3D network. RGB features are extracted from the frame images of the clips to capture spatial information such as color and texture; at the same time, optical flow features are extracted by analyzing the pixel changes between frames to record the object motion information.

[0113] In step 1313, the RGB features and optical flow features corresponding to each video clip may be fused. For example, feature splicing or the like may be used to integrate the RGB features representing the spatial dimension with the optical flow features representing the temporal dimension to form three-dimensional temporal features corresponding to the multiple video clips for subsequent analysis.

[0114] In step 132, the one-dimensional convolution processing can use a one-dimensional convolution kernel to slide on a specific dimension (such as the time dimension) of the three-dimensional time series feature, perform weighted summation of adjacent feature values on the specific dimension, and introduce nonlinearity through an activation function to mine the local feature pattern on the specific dimension to obtain a feature representation A.

[0115] Self-attention processing can be used to calculate the degree of association between each position in the feature representation A and other positions to obtain attention scores, and then these attention scores are normalized. The normalization result is used as a weight to perform weighted summation of the features at each position to obtain the feature representation B.

[0116] The feedforward processing is used to sequentially pass the feature representation B through multiple fully connected layers. Each fully connected layer is used to perform a linear transformation on the input, and then a non-linear activation function is used to increase the non-linear expression ability of the linear transformation result, obtaining the fused region features. The fused region features synthesize the information of each previous processing step and can more comprehensively and effectively characterize the characteristics of the target construction machinery.

[0117] Step 133 uses the denoising network model to determine the fused region features and the noise distribution at the s-th time step, and the process of determining the action category at the s-th time step specifically includes:

[0118] Step 1331: Extract features from the noise distribution at the s-th time step to obtain the noise distribution features at the s-th time step;

[0119] Step 1332: Fuse the noise distribution features with the time encoding vector to obtain the noise time distribution features at the s-th time step;

[0120] Step 1333: Extract features from the noise time distribution features at the s-th time step to obtain the noise representation at the s-th time step;

[0121] Step 1334: After concatenating the noise representation at the s-th time step with the fused region features as the query vector and the key vector, and using the noise representation at the s-th time step as the value vector, input them into the cross-attention layer. The cross-attention layer calculates the attention weights between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weights to obtain the cross-attention result;

[0122] Step 1335: Fuse the cross-attention result with the noise representation at the s-th time step to obtain the fused representation;

[0123] Step 1336: Use the feedforward connection layer and the convolutional layer to process the fused representation to obtain the action category at the s-th time step.

[0124] Refer to Figure 2 , which shows the structural schematic diagram of the denoising network model of an embodiment of the present application. The denoising network model specifically includes: a one-dimensional convolutional layer A201, a first fusion module 202, a one-dimensional convolutional layer B203, a cross-attention layer 204, a second fusion module 205, a feedforward connection layer 206, and a one-dimensional convolutional layer 207.

[0125] Among them, in step 1331, the one-dimensional convolutional layer A201 can be used to extract features from the noise distribution at the s-th time step to obtain the noise distribution features at the s-th time step. The one-dimensional convolutional layer A201 aims to capture the key feature information in the noise at the s-th time step to obtain the noise distribution features.

[0126] During the action recognition process, the noise distribution at the s-th time step can increase feature diversity, improve the ability of the denoising network model to distinguish different actions, enhance the robustness of the denoising network model for action recognition, and assist in analyzing the correlation between features.

[0127] The noise distribution at the s-th time step can be obtained through sensor measurement, interference in the data acquisition environment, adding simulated noise during data preprocessing, or generated by the model. For example, the types of the noise distribution at the s-th time step can be any one or a combination of the following noise types: Gaussian distribution, Poisson distribution, uniform distribution, etc.

[0128] In step 1332, the first fusion module 202 can be used to fuse the noise distribution feature and the time encoding vector to obtain the noise time distribution feature at the s-th time step. Combining the time information with the noise distribution feature can form a more time-aware noise time distribution feature and enhance the consideration of time factors in the feature.

[0129] In step 1333, the one-dimensional convolutional layer B203 can be used to extract features from the noise time distribution feature at the s-th time step to obtain the noise representation at the s-th time step. Using one-dimensional convolution again can deepen the information mining of the noise time distribution feature, strengthen important features, remove redundant information, and extract a more discriminative noise representation.

[0130] In step 1334, the noise representation at the s-th time step is concatenated with the fusion region feature and used as the query vector and the key vector, and the noise representation at the s-th time step is used as the value vector and input into the cross-attention layer. The cross-attention layer calculates the attention weights between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weights to obtain the cross-attention result.

[0131] Among them, concatenating the noise representation at the s-th time step with the fusion region feature forms a vector with both noise characteristics and mechanical overall feature information, which is used as the query vector and the key vector.

[0132] The attention weights calculated by the cross-attention layer represent the degree of correlation between the features of each dimension between the query vector (formed by concatenating the noise representation and the fusion region feature) and the key vector (also formed by concatenating the noise representation and the fusion region feature). The higher the attention weight, the closer the corresponding features are related.

[0133] Performing weighted summation on the value vector (noise representation) according to these attention weights is essentially to re-“assign values” to each part in the value vector according to the degree of correlation between features. The part of the noise feature with close correlation, due to the high weight, accounts for a larger proportion in the weighted summation result, thus being highlighted.

[0134] The finally obtained cross-attention result realizes the deep integration of the noise and the feature information of the fusion region. The integrated information contains both the key parts in the noise features that are closely related to the overall features and the comprehensive information about the target construction machinery carried by the fusion region features, providing more targeted and rich data for subsequent action recognition.

[0135] In step 1335, the second fusion module 205 can be used to fuse the cross-attention result with the noise representation at the s-th time step to obtain a fused representation. The second fusion module 205 fuses the cross-attention result and the noise representation at the s-th time step, further integrating the information to form a more comprehensive fused representation, preparing for the final action recognition.

[0136] In step 1336, the fused representation can be processed using a feed-forward connection layer and a convolutional layer to obtain the action category at the s-th time step. The feed-forward connection layer and the convolutional layer respectively convert the fused representation into an output related to the action category through linear transformation and non-linear activation, completing the recognition of the action. The action recognition result specifically includes: the probability that the s-th time step belongs to a preset action category. Examples of preset action categories can include: the excavation action or slewing action of an excavation device, or the loading action or unloading action of a dump device, etc.

[0137] In the embodiments of this application, a time step is an abstract concept when processing time series data. It can be closely related to the frame number. One time step can correspond to one or more frame numbers. For example, when collecting a video at a fixed frame rate (such as 30 frames per second), if the length of a time step is set to 1 second, then one time step corresponds to 30 consecutive frame numbers. Specifically, the first time step may include frames 1 to 30, the second time step includes frames 31 to 60, and so on. A time step can process and analyze the information in the video from a more macroscopic perspective, rather than directly operating on each individual frame. It groups consecutive frames, facilitating action recognition. In different tasks, the length of the time step can be flexibly adjusted according to specific requirements. For example, in some action recognition tasks of construction machinery with relatively slow actions, the time step may be set to a longer duration (including more frame numbers) to capture the complete action information; while for tasks with faster actions, a shorter time step may be set to more precisely analyze the subtle changes in the actions.

[0138] One time step generally corresponds to a video segment. This video segment is intercepted from the construction operation video according to a sliding window. For example, for a relatively long construction operation video, according to the division of time steps, each time step corresponds to a video segment containing a certain number of frames.

[0139] In step 134, by traversing the time steps, when it is detected that the action categories corresponding to adjacent time steps change, this moment is taken as the segment division point. Then, starting from the starting frame of the video, according to these division points, video segments of different action categories are intercepted in sequence, so as to divide the construction operation video of each target construction machinery into multiple construction operation segments arranged in chronological order.

[0140] In step 104, the multiple construction operation segments arranged in chronological order can be traversed, the start time and end time of each construction operation segment are recorded, the duration of the action category corresponding to the construction operation segment is obtained by subtracting the start time from the end time, and by integrating the duration data of all construction operation segments, the duration data of each action category can be determined. The duration data of the action category specifically includes: the duration value of the action category.

[0141] In step 105, the duration values of the same action category appearing at different times can be collected to form a data set, and this data set is the factual duration sample corresponding to the action category, covering the duration information of the action category at different time periods in the construction operation video.

[0142] In step 106, based on the factual duration sample corresponding to the action category, a random duration sample corresponding to the action category is further generated. This process simulates various unpredictable actual situations in the construction process, so that the generated random duration sample covers various uncertainties and change factors in the construction process.

[0143] In a specific implementation, the process of step 106 generating a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category specifically includes:

[0144] Step 161: Determine the target distribution function corresponding to the action category according to the factual duration sample corresponding to the action category;

[0145] Step 162: Use the rejection sampling method to generate a random duration sample that conforms to the target distribution function.

[0146] In step 161, determining the target distribution function is to establish a mathematical model basis for generating random duration samples. By analyzing the factual duration samples corresponding to the action category, finding out the distribution rules presented by these samples, so as to obtain the target distribution function that can describe the distribution characteristics of the action duration. This helps to generate random data that conforms to the actual situation distribution subsequently.

[0147] In one example, a Dirichlet process mixture model (DPMM) can be used to determine the non-parametric distribution followed by factual duration samples. A non-parametric distribution refers to a probability distribution that does not depend on a pre-assumed specific parametric probability distribution form. That is, it does not make specific parametric assumptions about the distribution followed by the data. For example, unlike the normal distribution which has explicit parameters such as mean and variance to fully determine the distribution shape, it allows the data itself to "reveal" its distribution characteristics.

[0148] As a typical representative of non-parametric models, the Dirichlet process mixture model can reduce the dependence on artificially specified distribution types by defining on an infinite-dimensional parameter space, thereby improving the modeling flexibility of the sample distribution.

[0149] Suppose is the data set corresponding to the factual duration samples, N is the number of factual duration samples, and the factual duration samples follow an unknown probability distribution . Then the Dirichlet process mixture model for estimating the unknown probability distribution based on the data set can be expressed as formula (1):

[0150] (1)

[0151] In formula (1): The function is the distribution of the sample data when the given parameter ; The parameter satisfies i.i.d (independent and identically distributed) and follows the distribution , follows the Dirichlet process.

[0152] Among them, the definitions of the Dirichlet distribution and the Dirichlet process are as follows:

[0153] The Dirichlet distribution is a multivariate generalization of the Beta distribution, and its definition is: If the joint distribution density of the random vector is:

[0154] (2)

[0155] Then it is said that follows the Dirichlet distribution, denoted as . Among them is the Gamma function. When When it is, the Dirichlet distribution degenerates into the Beta distribution.

[0156] The definition of the Dirichlet process is as follows: Assume is a probability measure of a random variable on the measure space with the concentration parameter . If the probability measure of a random variable on the measure space satisfies that for any finite partition of the measure space , there exists:

[0157] (3)

[0158] Then it is said that obeys the Dirichlet process composed of the base distribution and the concentration parameter , denoted as . The concentration parameter reflects the similarity degree between and the base distribution . The larger its value, the more similar

[0159] is to the base distribution . In the process of establishing a non-parametric distribution based on the Dirichlet process mixture model, all unknown model parameters can be denoted as , where represents the parameter of the th probability density distribution, is the weight of the th probability density distribution, and . According to Bayesian theory, the posterior distribution estimate of can be calculated through its prior distribution and likelihood function, and is used as the target distribution function that the factual duration samples of each action category of the target construction machinery obey.

[0160] (4)

[0161] In step 162, the rejection sampling method is used to generate random duration samples that conform to the target distribution function, aiming to obtain real and available random data from the theoretical target distribution function, and these random data can reflect the uncertainties and variable factors in the construction process.

[0162] In specific implementation, the process of using the rejection sampling method to generate random duration samples that conform to the target distribution function specifically includes:

[0163] Step 1621: Determine the reference distribution function and the constant; under the condition of the same independent variable, the function value of the target distribution function is not greater than the product of the function value of the reference distribution function and the constant.

[0164] Step 1622: Extract a random independent variable from the reference distribution function.

[0165] Step 1623: Extract a random number from the uniform distribution function; the upper limit value of the uniform distribution function is: under the condition of the random independent variable, the product of the function value of the reference distribution function and the constant.

[0166] Step 1624: If the random number is less than the function value of the target distribution function corresponding to the random independent variable, then take the random independent variable as the sample of the random duration that conforms to the target distribution function.

[0167] In one example, the envelope function can be used to iterate the "reject / accept" step to implement the sampling of the random duration sample.

[0168] For the target distribution function to be sampled , first select a reference distribution that is easy to implement sampling , such as the normal distribution, and assume that there is a constant such that for any there is . Then the specific rejection sampling calculation process is: based on the reference distribution generate a random independent variable , and then generate a random variable from the uniform distribution . If there is , then reject the random independent variable , otherwise accept the random independent variable , that is, take the random independent variable as the sample of the random duration that conforms to the target distribution function.

[0169] In step 107, according to the duration value of the random duration sample, assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation, so that the construction process parameters in the construction simulation model are closer to the uncertainty and changes in the actual construction, and provide a reasonable data basis for the subsequent simulation of the construction simulation model and the prediction of the construction progress.

[0170] For example, select any one from the duration values corresponding to these random duration samples to determine the unit operation duration corresponding to the target construction machinery, and assign the determined unit operation duration to the corresponding construction process parameters in the construction simulation model.

[0171] In a construction scenario, the unit operation duration refers to the time taken for the target construction machinery to complete a standard operation action.

[0172] It is an important part of the construction process parameters and can be used to measure construction efficiency. Its value varies due to factors such as construction machinery type, action category, and construction environment. By using random duration samples for assignment, it can more realistically reflect the uncertainty of this time in actual construction, provide accurate parameters for the construction simulation model, and assist in construction progress prediction.

[0173] Taking excavation equipment as an example, the unit operation duration refers to the time taken for the excavation equipment to complete a standard operation action.

[0174] For example, in earthwork excavation operations, the standard operation action of the excavation equipment is "lower the shovel - excavate - lift - rotate - unload - rotate back to position - lower the shovel in preparation for the next excavation". The time taken to complete such a cycle of standard operation actions is a type of unit operation duration.

[0175] In step 108, based on the construction process parameters of the target construction machinery after assignment, the construction simulation model is simulated to obtain the construction progress prediction result, which helps construction managers understand the construction progress in advance and reasonably arrange resources and construction periods, etc.

[0176] The construction simulation model is a mathematical or logical abstraction of the actual construction process. The construction process parameters in it determine key elements such as the time taken for construction processes in the construction simulation model. By inputting the assigned construction process parameters, the construction simulation model performs operations and simulations according to the set rules and logic, thereby predicting the construction progress.

[0177] Among them, during the simulation process of the construction simulation model, information such as the start time and end time of each construction process is recorded and calculated, and finally the construction progress prediction result is output.

[0178] In one example, the construction progress prediction result specifically includes the following information:

[0179] Time node information: The start time and end time of each construction process, as well as the estimated completion time of the entire construction operation, clarifying the time flow of the construction.

[0180] Progress deviation situation: By comparing with the planned progress, it shows whether the predicted progress is ahead or behind, and the specific duration or proportion of the deviation, which is convenient for timely adjustment. Among them, the planned progress is the percentage of the construction process that should be completed at each time point based on goals, contracts, resources, etc. before the start of construction operations, and it is the reference standard for the construction progress. The predicted progress is to use the construction process parameters for simulation, dynamically calculate the actual completion of the construction process at each time point, and obtain the percentage of the project completed at each time point, reflecting the estimated progress considering uncertain factors.

[0181] Resource demand data: It covers the demand quantity and time of resources such as manpower, materials, and equipment in each construction stage, and helps with resource preparation and allocation.

[0182] Risk warning prompt: It points out the risk factors that may affect the progress during construction, such as technical problems, bad weather, policy changes, etc.

[0183] In summary, the construction progress prediction method based on long-term video analysis in the embodiments of the present application starts from the construction operation video and obtains the construction progress prediction result through multi-step processing. First, analyze the relevant operation videos of the construction operation (including the historical operation videos of the construction operation and / or the historical operation videos of the same type of operation, and the duration is greater than the duration threshold) to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; then crop the construction operation videos of each target construction machinery from the spatio-temporal dimension; then extract the three-dimensional time-series features of the video, perform action recognition to determine the action category, determine the duration data according to the action category, and thus obtain factual and random duration samples; use the duration values corresponding to the random duration samples to assign values to the construction process parameters (such as the unit operation duration) of the target construction machinery in the construction simulation model; finally, perform construction simulation model simulation to obtain the construction progress prediction result.

[0184] In the embodiments of the present application, the parameter values of the construction process parameters are determined based on the construction operation-related videos. Since the construction operation-related videos in the embodiments of the present application record various actions of the target construction machinery during the construction process, based on the analysis of the construction operation-related videos, real data such as action categories and factual duration samples can be obtained from the construction operation-related videos, and random duration samples can be generated through the processing of these real data. Since the random duration samples contain various uncertainties and change factors in the construction process, using them to assign values to the construction process parameters can make the parameter values of the construction process parameters closer to the real state of the construction operation site. In this way, the construction simulation model containing the construction process parameters can more accurately simulate the construction progress based on the parameters closer to the real state. Therefore, the embodiments of the present application can effectively improve the accuracy of construction progress prediction.

[0185] Method Embodiment Two

[0186] This embodiment describes the construction process of the construction simulation model.

[0187] Refer to Figure 3 , which shows the schematic flow chart of the steps of the method for automatically constructing a construction simulation model based on video images according to an embodiment of the present application. The method specifically includes the following steps:

[0188] Step 301: Establish a construction simulation model for the construction operation. The above construction simulation model specifically includes: construction unit intelligent agent objects, construction resource scheduling intelligent agent objects, construction resource intelligent agent objects, and construction process simulation models corresponding to multiple construction unit intelligent agent objects respectively. The above construction process simulation model specifically includes: construction process simulation models. The above construction process simulation model includes: queue elements and delay elements. The above queue elements are used to simulate the queuing and waiting process of construction resource intelligent agent objects in the construction process. The above delay elements are used to simulate the duration of the construction process.

[0189] The simulation process of the above construction simulation model specifically includes: the construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task assignment message to at least one selected target construction resource intelligent agent object according to the above resource request message; after receiving the task assignment message, at least one target construction resource intelligent agent object moves to the target construction operation surface where the construction unit intelligent agent object is located; the construction resource intelligent agent objects that meet the set conditions at the target construction operation surface are put into the queue represented by the queue elements for queuing and waiting; the construction simulation of the construction process is performed by using the construction resource intelligent agent objects that do not need to queue and wait or are taken out from the queue elements at the target construction operation surface; after the simulation of the construction process of the construction unit intelligent agent object is completed, the corresponding delay element ends.

[0190] Step 302: According to the video images of the construction operation site, use target detection technology, target tracking technology, and clustering technology to determine the construction resource distribution information corresponding to the construction operation surface of the construction operation site.

[0191] Step 303: According to the construction resource distribution information, instantiate the queue elements and delay elements in the corresponding construction process simulation model.

[0192] Figure 3 The method shown can be used to construct a construction simulation model based on video images for the construction operation.

[0193] Taking the slope excavation operation as an example, according to the geometric structure subordination relationship of the excavation part, the construction objects of the slope excavation operation can be divided into 3 levels, namely excavation areas, excavation benches, and excavation layers.

[0194] Among them, an excavation area may include: several excavation benches. Specifically, the entire construction object can be divided into different excavation areas such as the front block excavation area and the top surface excavation area of the protective layer according to the vertical zoning line. Subsequently, each excavation area is further divided into multiple excavation benches according to the designed bench height. For example, the elevation range of 50m - 55m is designated as the first excavation bench, and 55m - 60m is designated as the second excavation bench. Due to different heights, the construction conditions of different excavation benches vary.

[0195] A specific excavation sub-bench includes: several excavation sub-blocks. Specifically, each excavation bench can be divided into multiple excavation sub-blocks along the horizontal direction. Each excavation sub-block is a basic construction unit, and subsequent specific construction processes are carried out in these excavation sub-blocks.

[0196] In the field of construction simulation technology, an agent refers to an entity that can perceive the environment, make decisions, and take actions. It has autonomy and can decide which actions to execute according to its own goals and perception of the environment to achieve the predetermined goals.

[0197] An agent object is the instantiation of an agent in a specific application scenario (such as a construction simulation model). It endows the abstract agent concept with specific attributes, states, and behavior rules, enabling it to represent an entity with autonomous behavior capabilities in a specific construction simulation model. Behavior rules refer to a series of guidelines that specify how an agent should act in a specific environment (such as a construction scenario). It clarifies the response mode of the agent to internal state changes and external stimuli (such as instructions, events, etc.), and determines the behavior logic and execution order of the agent.

[0198] For example, in a construction simulation model, construction unit agent objects, construction resource scheduling agent objects, and construction resource agent objects all have their own specific attributes (such as the geometric dimensions of construction units, the types and quantities of construction resources, etc.), states (such as idle, busy, etc.), and behavior rules (such as resource scheduling algorithms, state transition methods of construction resources).

[0199] The construction unit agent object, construction resource scheduling agent object, and construction resource agent object will be described separately below.

[0200] The construction unit agent object is used to describe construction units such as excavation sub-blocks. The collection of multiple construction unit agent objects can be used to describe the construction object corresponding to the entire construction operation. The construction unit agent object usually contains attribute parameters related to the construction unit, and these attribute parameters can be information such as construction unit size, location, and geological conditions. Each construction unit's characteristics are characterized by these attribute parameters.

[0201] The embodiments of the present application can instantiate a construction unit to obtain a construction unit intelligent body object. For example, the instantiation process of the construction unit specifically includes: First, obtain the partition number of the construction unit from the project documents; then, for the partition number of the construction unit, obtain attribute information such as the geometric shape, size data, location data, and first behavior rules of the construction unit from the project documents; bind the partition number of the construction unit with the attribute information to obtain the construction unit intelligent body object.

[0202] Taking the slope excavation operation as an example, the construction unit intelligent body object specifically includes: an excavation partition intelligent body object, an excavation bench intelligent body object, and an excavation block intelligent body object.

[0203] Among them, the attribute information of the excavation partition intelligent body object includes the partition number.

[0204] The attribute information of the excavation bench intelligent body object includes: the partition number of the affiliated excavation partition, the bench number, the bench height, and the start and end elevations.

[0205] The attribute information of the excavation block intelligent body object includes: the affiliated gradient number, the block number, and the engineering quantity information. The engineering quantity information can include the excavation block area, the length along the reference line, the width along the reference line, the excavation block height, the number of deep / shallow hole drillings, etc. The slope excavation operation can be described as a set composed of all excavation block intelligent body objects. The construction simulation of the slope excavation operation can be understood as: performing simulation calculations on the set of excavation block intelligent body objects. Optionally, the set of excavation partition intelligent body objects and the set of excavation bench intelligent body objects are only used for imposing construction constraint conditions, statistical calculation of engineering quantity information, etc., and are not used as specific construction simulation objects.

[0206] The construction resource intelligent body object represents various construction resources, such as human resources (construction personnel) and material resources (construction equipment, materials, etc.). In the construction scenario, each construction resource can be regarded as a construction resource intelligent body object. The construction resource intelligent body object can receive task instructions from the construction resource scheduling intelligent body object and play a role in the construction process corresponding to the construction unit intelligent body object. For example, the excavating equipment performs excavation operations in the designated excavation block according to the instructions, and the transport vehicle transports the excavated soil away, etc. The behavior and state of the construction resource intelligent body object will be affected by the construction process, and at the same time, it will also feedback information to the construction resource scheduling intelligent body object for better resource allocation.

[0207] Embodiments of this application can instantiate construction resources to obtain construction resource intelligent agent objects. Specifically, analyze the types of construction resources and the requirements for construction resource scheduling involved in construction operations. By defining the possible states of construction resource intelligent agents, as well as the state transition methods, triggering conditions, and second behavior rules between different states, the instantiation of construction resources is completed. The information of the construction resource scheduling intelligent agent object specifically includes: resource category, resource number, performance parameters, status, state transition method, triggering condition, etc.

[0208] The states of the construction resource intelligent agent object can include but are not limited to: initial state, idle state, working state, maintenance state, etc. The state transition methods can include: time-triggered, condition-triggered, and message-triggered. Among them, time-triggered means that after a specific time interval is met, it switches from one set state to another set state; condition-triggered means that it switches from one set state to another set state after a specific condition is met; message-triggered means that when the construction resource intelligent agent object receives a specific message, it switches from one set state to another set state.

[0209] Taking the slope excavation operation as an example, the specific resource categories of the construction resources that may be involved in the excavation construction process include: drilling equipment, bulldozing equipment, excavation and loading equipment, transportation equipment. Then, the intelligent agent objects of drilling equipment, bulldozing equipment, excavation and loading equipment, and transportation equipment can be defined accordingly.

[0210] Taking the intelligent agent object of transportation equipment as an example, its state and state transition can be defined as Figure 3 shown. At the beginning of the simulation, the state of the intelligent agent object of transportation equipment changes from the initial state to the idle state. When receiving the task assignment message, the state of the intelligent agent object of transportation equipment changes to move to the excavation working face. After arriving at the excavation working face, the state of the intelligent agent object of transportation equipment changes to waiting for loading. After completing construction processes such as loading, heavy transportation, and unloading, the state of the intelligent agent object of transportation equipment changes to idle. When receiving the start of daily maintenance message, the state of the intelligent agent object of transportation equipment changes to daily maintenance.

[0211] The construction resource scheduling intelligent agent object is used to schedule the construction resource intelligent agent object. It can perceive the resource requirements during the construction process of the construction unit intelligent agent object. For example, in the slope excavation operation, when a certain excavation block (construction unit intelligent agent object) needs construction resources such as excavation equipment and transportation vehicles, it can send a resource request message to the construction resource scheduling intelligent agent object. The construction resource scheduling intelligent agent object will make a scheduling decision based on factors such as the availability of the construction resource intelligent agent object and the priority of the construction unit, and allocate the appropriate construction resource intelligent agent object to the required construction unit intelligent agent object.

[0212] The construction resource scheduling agent object in the embodiment of this application may include the following information: scheduling scope (responsible construction operation surface), scheduling strategy (such as allocating resources according to the priority of construction units or resource utilization efficiency, etc.), status, status transition method, trigger condition, third behavior rule, etc.

[0213] The status of the construction resource scheduling agent object may include but is not limited to: initial status, construction task existence judgment status, waiting for construction task status, service construction simulation object selection status, idle construction resource judgment status, waiting for construction resource to be idle status, construction resource scheduling status, task distribution, etc.

[0214] After the construction simulation model starts simulation, the status of the construction resource scheduling agent changes from the initial status to the construction task existence judgment status. If a resource request message sent by any construction unit agent object is received, it is considered that there is a construction task, and the status changes to the service construction simulation object selection status; if no resource request message is received, the status changes to the waiting for construction task status, and the loop judgment continues until a resource request message is received;

[0215] When the status of the construction resource scheduling agent changes to the service construction simulation object selection status, according to the first preset rule, a target construction unit agent object is selected from multiple construction unit agent objects that send requests, and the status changes to the idle construction resource judgment status; if there is an idle construction resource agent object, the status changes to the construction resource scheduling status; if not, the status changes to the waiting for construction resource to be idle status, and the loop judgment continues until the status of a construction resource agent becomes idle;

[0216] When the status of the construction resource scheduling agent changes to the construction resource scheduling status, according to the second preset rule, one or more target construction resource agent objects are selected from multiple idle construction resource agent objects, the status changes to the task distribution status, and a task assignment message is sent to the target construction resource agent object; the task assignment message is used to prompt the target construction resource agent object to go to the target construction operation surface corresponding to the target construction unit agent object to perform the operation of the corresponding construction process.

[0217] The first preset rule is used to optimize the resource allocation order, and reasonably allocate the construction resource agent objects to specific construction unit agent objects in multiple resource request messages to speed up the construction progress and balance the resource supply and demand.

[0218] The first preset rule specifically covers: based on the priority of construction units, giving priority to construction units on the critical path or with earlier construction processes to obtain construction resources; or, based on the urgency of resource requirements, giving priority to meeting construction units that are time - urgent and have a great impact on resource shortage; or, based on resource allocation efficiency, preferentially allocating construction resources to construction units that are closer to the construction resource intelligent agent object. It can be understood that the embodiments of the present application do not limit the specific first preset rule.

[0219] The second preset rule aims to accurately and efficiently select construction resource intelligent agent objects to meet the resource requirements of construction unit intelligent agent objects.

[0220] The second preset rule specifically covers:

[0221] Based on resource adaptability: preferentially selecting construction resource intelligent agent objects that are highly adaptable to the construction processes of the target construction unit intelligent agent object. For example, if the target construction unit is performing fine - welding operations, preferentially selecting welding equipment intelligent agents with high welding precision and meeting the process requirements of the operation.

[0222] Based on performance parameters: preferentially selecting those with better performance parameters from the construction resource intelligent agent objects in the idle state.

[0223] Based on resource scheduling cost: comprehensively considering the resource scheduling cost, preferentially selecting construction resource intelligent agent objects with short scheduling paths and low energy consumption. For example, if there are multiple idle construction resource intelligent agent objects, preferentially selecting those that are close to the target construction operation surface and have good traffic conditions on the route to the operation surface to reduce transportation costs and time costs.

[0224] The following explains the construction process simulation model corresponding to the construction unit intelligent agent object.

[0225] In specific implementation, for the construction unit intelligent agent object, it is possible to analyze the construction processes involved in its construction process and the dependencies between construction processes, and based on the analysis results, perform simulation modeling on the construction process of the construction unit to obtain the construction process simulation model corresponding to the construction unit intelligent agent object.

[0226] The dependency relationship between construction processes refers to the mutual connection and influence between different construction processes. The above - mentioned dependency relationships specifically include: sequential dependency, parallel dependency, and end - dependency. Among them, sequential dependency means that the next construction process can start only after the previous construction process is completed. Parallel dependency means that two construction processes can start simultaneously, but one construction process needs to wait for the construction situation of the other construction process to meet the set conditions. End - dependency means that when one construction process is completed, the other construction process must also be completed.

[0227] The above construction process simulation model specifically includes: a construction process simulation model corresponding to a construction process; the above construction process simulation model includes: a queue element and a delay element; the above queue element is used to simulate the queuing and waiting process of construction resource agent objects in a construction process; the above delay element is used to simulate the duration of a construction process. When the queue element and the delay element are used simultaneously, the queue element and the delay element can be connected in series.

[0228] In an embodiment of the present application, a construction resource agent object that meets the set conditions at a target construction operation surface can be placed in the queue represented by the queue element for queuing and waiting.

[0229] The above set conditions specifically include:

[0230] Set condition 1: For the first construction resource agent object at the target construction operation surface, the first construction resource agent object corresponds to a first resource category, and the number of construction resource agent objects of the first type at the target construction operation surface exceeds the maximum number of objects that can work simultaneously; and / or

[0231] Set condition 2: For the first construction resource agent object at the target construction operation surface, the first construction resource agent object corresponds to a first resource category, and the construction resource agent objects of the first resource category cooperate with the construction resource agent objects of a second resource category to jointly simulate a construction process. The target construction operation surface may refer to the construction operation surface where the construction unit agent object that sends a resource request message is located.

[0232] For set condition 1, when a construction resource agent object arrives at the target construction operation surface, if there are construction resource agent objects of the same resource category and the number of construction resource agent objects of this resource category exceeds the maximum number of objects that can work simultaneously, the arriving construction resource agent object is placed in the queue represented by the queue element for waiting until a construction resource agent object of the same resource category leaves the current construction operation surface.

[0233] For set condition 2, when a construction resource agent object arrives at the target construction operation surface, if it is necessary to cooperate with construction resource agent objects of other resource categories to jointly perform the current construction process, the arriving construction resource agent object is placed in the queue represented by the queue element for waiting until the construction resource agent objects of other resource categories required for cooperation arrive at the target construction operation surface.

[0234] Based on the simulation modeling of a single construction process, by analyzing the dependency relationships of the construction processes involved in the construction process, the construction process simulation models can be connected in series or in parallel, thereby completing the simulation modeling of the construction process.

[0235] Referring to FIG. 4(a), a modeling schematic diagram of the construction process of an embodiment of the present application is shown. Among them, for the sequential dependency relationship between construction processes, the construction process simulation models between different construction processes can be connected to complete the modeling of the construction process. For example, the queue element 1 and the delay element 1 included in construction process 1, and the queue element 2 and the delay element 2 included in construction process 2 are connected in series.

[0236] Referring to FIG. 4(b), a modeling schematic diagram of the construction process of an embodiment of the present application is shown. Among them, for the parallel dependency and end dependency relationships, the construction processes can be modeled independently, and functions are used to implement the modeling of the dependency relationship in the way of judging conditions. For example, the queue element 3 and the delay element 3 included in construction process 3 are modeled independently, and the queue element 4 and the delay element 4 included in construction process 4 are modeled independently. Construction process 3 and construction process 4 show a parallel relationship.

[0237] The model parameters of the construction process simulation model can include: the number of queue elements, the number of delay elements, the maximum allowable capacity of the queue, and the queue queuing rule, etc. The embodiments of the present application can perform the following parameter settings on the construction process simulation model:

[0238] 1) The maximum allowable capacity of the queue represents the maximum number of construction resource agent objects that the queue elements can accommodate. If the queue is full, the newly arrived construction resource agent objects will become idle, return to wait or be rescheduled to a new construction operation surface.

[0239] 2) The queue queuing rule: represents the queuing rule of construction resource agent objects of the same resource category, such as first in first out, last in first out, etc.

[0240] Taking the slope excavation operation constructed by the drill and blast method as an example, for the excavation block, that is, the construction unit of the foundation, its construction process involves the following construction processes:

[0241] 1) Measuring and setting out (construction process A)

[0242] 2) Drilling (construction process B)

[0243] 3) Charging and connecting wires (construction process C)

[0244] 4) Blasting (construction process D)

[0245] 5) Mucking and loading (construction process E)

[0246] 6) Slag transportation (construction process F)

[0247] The dependency relationships between the construction processes are:

[0248] 1) Sequential dependency: A → B (Drilling can only be carried out after the measurement and setting out are completed)

[0249] 2) Sequential dependency: B → C (Charging and wiring can only be carried out after drilling is completed)

[0250] 3) Sequential dependency: C → D (Blasting can only be carried out after charging and wiring are completed)

[0251] 4) Sequential dependency: D → E (Mucking and loading can only be carried out after blasting is completed)

[0252] 5) Sequential dependency: E → F (Slag transportation can only be carried out after mucking and loading are completed)

[0253] Then the construction process simulation model of an excavation block is as Figure 5 shown, which specifically includes: the construction process simulation models of 6 construction processes. The construction process simulation models of 6 construction processes specifically include: the construction process simulation model of construction process A, the construction process simulation model of construction process B, the construction process simulation model of construction process C, the construction process simulation model of construction process D, the construction process simulation model of construction process E, and the construction process simulation model of construction process F. The construction process simulation model of construction process A specifically includes: queue element A and delay element A. The construction process simulation model of construction process B specifically includes: queue element B and delay element B. The construction process simulation model of construction process C specifically includes: queue element C and delay element C. The construction process simulation model of construction process D specifically includes: queue element D and delay element D. The construction process simulation model of construction process E specifically includes: queue element E and delay element E. The construction process simulation model of construction process F specifically includes: queue element E and delay element F.

[0254] In addition to realizing the modeling of the construction process simulation model, the embodiments of the present application also provide the following simulation process of the construction simulation model: The construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task assignment message to at least one selected target construction resource intelligent agent object according to the resource request message; after receiving the task assignment message, at least one target construction resource intelligent agent object moves to the target construction operation surface where the construction unit intelligent agent object is located; the construction resource intelligent agent objects that meet the set conditions at the target construction operation surface are put into the queue represented by the queue element to queue and wait; the construction resource intelligent agent objects that do not need to queue and wait or are taken out from the queue element at the target construction operation surface are used to execute the construction simulation of the construction process; after the simulation of the construction process of the construction unit intelligent agent object is completed, the corresponding delay element ends.

[0255] The construction simulation model simulates a complete construction process, involving the interaction between multiple parts and the sequential execution of construction procedures. Multiple parts work together to complete the simulation of construction procedures. The above simulation process specifically includes the following steps:

[0256] Step A1, resource request;

[0257] The construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model. Step A1 simulates the situation in actual construction where, before starting construction, the construction unit requests the required resources from the department or system responsible for resource scheduling. For example, before performing the construction procedure of mucking and loading, the construction unit intelligent agent object needs to request construction resource intelligent agent objects such as the excavation equipment intelligent agent object and the dump truck equipment intelligent agent object from the construction resource scheduling intelligent agent object.

[0258] Step A2, resource scheduling;

[0259] The construction resource scheduling intelligent agent object sends a task assignment message to at least one selected target construction resource intelligent agent object according to the received resource request message. The construction resource scheduling intelligent agent object will select appropriate construction resource intelligent agent objects based on factors such as the type, quantity, and distance of the requested resources, and send a task assignment message to notify them to prepare for work. For example, according to the location, status, etc. of the transportation equipment intelligent agent object, the nearest and idle transportation equipment intelligent agent object to the construction unit intelligent agent object is selected, and a task assignment message is sent to it.

[0260] Step A3, resource movement:

[0261] After receiving the task assignment message, at least one target construction resource intelligent agent object moves to the target construction operation surface where the construction unit intelligent agent object is located. This simulates the process of allocating construction resources to the job site in reality. For example, the selected transportation equipment intelligent agent object will drive from its current location to the target construction operation surface.

[0262] The process by which the at least one target construction resource intelligent agent object moves to the target construction operation surface where the construction unit intelligent agent object is located after receiving the task assignment message specifically includes:

[0263] Step B1, generating a road network model based on the traffic data of the construction operation area;

[0264] Taking the construction scenario of a hydropower project as an example, relying on traffic data such as the general layout plan and the CAD (Computer-Aided Design) drawing of the internal traffic layout, the road network information of the area to be modeled is obtained, and a road network model described in the form of path points is generated according to the control point coordinate information in the plane rectangular coordinate system.

[0265] A road network model is a model constructed based on traffic data in the area to be modeled, aiming to digitally describe the structure and characteristics of the road network in the area to be modeled.

[0266] The road network model specifically includes the following elements:

[0267] Path points: Represent specific positions on the road, and their exact positions are determined by coordinates in a plane rectangular coordinate system. These path points are arranged in sequence along the road direction, and when connected, they form the path of the road.

[0268] Path segments: Formed by connecting adjacent path points, representing a section of the road. Each path segment has its specific attributes, such as length, direction, road type (such as main road, secondary road, etc.).

[0269] Topological relationship: Describes the connection relationship between path points and path segments, as well as the intersection, branching, etc. relationships between roads. Through the topological relationship, the structure of the entire road network can be clearly represented, facilitating operations such as path planning and traffic flow analysis.

[0270] Step B2: Set road attribute parameters in the road network model; the road attribute parameters include: road grade, allowed maximum vehicle speed, allowed maximum traffic density, number of lanes, and road surface width;

[0271] Step B3: Set marks for the set area in the road network model; the set area is the starting point or ending point of transportation; the set area includes at least one of: construction operation surface, material storage area, equipment parking area;

[0272] Among them, the material storage area can refer to the area for material entry and management. The equipment parking area can refer to the area for storing construction resources.

[0273] Step B4: Generate a path file corresponding to the target construction resource intelligent agent object according to the road network model, and the path file includes: starting point, ending point, passing points, and path length of transportation;

[0274] Step B5: After receiving the task assignment message, the at least one target construction resource intelligent agent object moves from the starting place to the target construction operation surface where the construction unit intelligent agent object is located according to the path file.

[0275] The road network model and path file generated through steps B1 - B4 can describe the specific route of the construction resource intelligent agent object from the starting place to the target construction operation surface, including detailed information such as the starting point, ending point, and passing points, providing clear navigation for the movement of the construction resource intelligent agent object.

[0276] In step B2, road attribute parameters such as the maximum allowable vehicle speed, maximum traffic density, number of lanes, and road surface width are specified in detail. These road attribute parameters reflect the road traffic capacity and restrictive conditions.

[0277] In step B4, the travel time of each section can be calculated using the maximum allowable vehicle speed and road length. Alternatively, in step B4, the road congestion situation can be predicted based on the maximum traffic density. Or, in step B4, the road carrying capacity can be judged by combining the number of lanes and the road surface width.

[0278] By comprehensively analyzing the road attribute parameters, step B4 can accurately calculate the lengths and estimated times of different paths, so as to select the optimal travel path for the construction resource intelligent body object, enabling the target construction resource intelligent body object to reach the target construction operation surface efficiently and accurately, and ensuring the orderly progress of the construction process.

[0279] Step A4, Queue and wait;

[0280] The construction resource intelligent body objects that meet the set conditions at the target construction operation surface are placed in the queue represented by the queue element for queuing and waiting. For example, due to limited construction site, multiple construction equipment cannot construct simultaneously after arrival and can only wait in a certain order to enter the operation surface for work. The construction equipment that meets a certain set condition (such as first come, first served, sorted by equipment performance, etc.) enters the queuing sequence.

[0281] Step A5, Construction simulation;

[0282] Using the construction resource intelligent body objects that do not need to queue and wait or are taken out from the queue element at the target construction operation surface, perform the construction simulation of the construction process. For example, according to the construction technology and process standards, the excavating equipment intelligent body object simulates the excavating action, controls the excavating depth, angle, and range, and the dump truck equipment intelligent body object simulates the material loading, transportation, and unloading process. Through the collaborative simulation of these construction resource intelligent body objects, the actual operation process of the construction process is fully presented.

[0283] Step A6, Process completed.

[0284] After the simulation of the i-th construction process of the construction unit intelligent body object is completed, the corresponding delay element ends.

[0285] The delay element is used to simulate the duration of a construction process. Taking mucking and loading as an example, the delay element counts the duration of the entire mucking and loading construction process, specifically including the time for the dump truck to move to the designated position, the operation time for the excavation equipment to excavate and load, and the waiting time of the dump equipment in the queue, etc. These times together constitute the total time required to complete the entire mucking and loading process, and the delay element comprehensively counts and accumulates these times to reflect the duration of the entire construction process.

[0286] The construction process simulation model can include: the connection relationships between different construction process simulation models. In this way, the embodiments of the present application can execute all the construction processes included in the construction process simulation model according to the above connection relationships.

[0287] For example, Figure 5 The construction sequence of the shown construction process simulation model is as follows: First, construction process A (surveying and setting out) is carried out. After its completion, construction process B (drilling) is carried out. After construction process B is completed, construction process C (charging and connecting wires) is carried out. After construction process C is completed, construction process D (blasting) is started. After construction process D is completed, construction process E (mucking and loading) is started. Finally, on the basis of the completion of construction process E, construction process F (slag transportation) is implemented. Each construction process is carried out strictly in such a sequential and progressive order. The completion of the previous construction process creates conditions for the subsequent process, and each construction process is simulated and presented by the corresponding construction process simulation model (queue element and delay element) for the queuing situation and duration in the construction process.

[0288] In a specific implementation, the construction simulation model may further include: the connection relationships between different construction process simulation models. The above connection relationships may include: series or parallel, etc. The embodiments of the present application can connect the construction process simulation models of each construction unit intelligent body object in a series or parallel manner by analyzing the construction sequence between the construction unit intelligent body objects, and thus obtain the construction simulation model of the construction operation, that is, construct a static construction simulation model for the construction operation. It should be noted that in the static construction simulation model, the number of parallel construction process models will be preset in the initial stage. Subsequently, as the construction operation progresses, the number of parallel construction process simulation models will be dynamically updated according to the analysis results of the video images, so as to achieve the accuracy and real-time performance of the construction simulation model.

[0289] In a specific implementation, a process data structure can be generated according to the sequential relationship between the construction processes of different construction units; the process data structure is used to describe the sequential relationship between the construction processes of different construction units; according to the process data structure, the construction process simulation models corresponding to different construction units are connected; the connection relationships between different construction process simulation models include: parallel or serial connection.

[0290] Taking the slope excavation operation as an example, a numbered sequence of excavation blocks can be used to describe the excavation operation sequence between the excavation blocks, and the construction process models of the excavation blocks are connected in series or parallel according to the sequence to obtain a static construction simulation model of a certain excavation area.

[0291] Refer to Figure 6 , which shows a schematic diagram of the construction simulation model of an embodiment of the present application. Among them, the construction simulation model can include: parallel construction process simulation models A, B, C, and D. Assume that a certain excavation area W is composed of 4 excavation blocks A, B, C, and D, and the corresponding construction process simulation models A, B, C, and D are established respectively. If the 4 excavation blocks A, B, C, and D are excavated simultaneously, the simulation model of the excavation area W is Figure 6 as shown, which is composed of the parallel connection of the construction process simulation models A, B, C, and D, and the completion time is determined by the excavation block with the latest completion of excavation.

[0292] The embodiment of the present application can execute the construction operations corresponding to all construction unit intelligent body objects according to the connection relationship between different construction process simulation models.

[0293] The embodiment of the present application can convert the connection relationship between different construction process simulation models in the construction simulation model into data structures such as directed graphs. The nodes in the directed graph represent the construction process simulation models, and the edges in the directed graph represent the sequence relationship between the construction process simulation models.

[0294] For the serial connection relationship, it is reflected as a linear sequence in the directed graph, that is, after the construction process simulation model corresponding to the previous node is completed, the construction process simulation model corresponding to the subsequent node can start; for the parallel connection relationship, multiple nodes can point to the same subsequent node at the same time, which means that the construction process simulation models corresponding to these nodes can be carried out in parallel.

[0295] In this way, the embodiment of the present application can perform topological sorting to clarify the sequence between different construction unit intelligent body objects, traverse using depth-first or breadth-first search algorithms, use multi-threading or processes to simulate parallel execution when encountering parallel processes, and execute strictly in sequence when encountering serial processes, while taking into account resource allocation and management, constraint condition processing, etc. throughout the process.

[0296] In step 302, target detection technology, target tracking technology, and clustering technology can be used to automatically determine the construction resource distribution information corresponding to the construction operation surface.

[0297] Among them, the process of determining the construction resource distribution information corresponding to the construction operation surface of the construction operation site by using target detection technology, target tracking technology, and clustering technology based on the video image of the construction operation site specifically includes:

[0298] Step C1: Perform target detection and target tracking on the video image to obtain the detection frames, resource categories, and identification serial numbers corresponding to multiple construction resources in the video image;

[0299] Step C2: Cluster the multiple construction resources in the video image according to the center point coordinates corresponding to the detection frames to obtain several clusters; the detection frames within the same cluster belong to the same construction operation surface;

[0300] Step C3: Determine the construction resource distribution information corresponding to the construction operation surface of the construction operation site according to the resource categories corresponding to the detection frames within the same cluster.

[0301] In step C1, the target detection technology and target tracking technology aim to identify different targets from the input video image and continuously track them to determine the position, category, and identification serial number of the targets in the image. In the construction operation scenario, the construction resources are the targets to be detected and tracked. By using the trained target detection model and target tracking model (such as a convolutional neural network model based on deep learning, etc.) to process the video image, the target detection model and target tracking model will scan and continuously locate the areas where multiple construction resources are located in the video image according to the learned characteristic patterns of various construction resources. This area is presented in the form of a detection frame, and at the same time, it can judge the category to which the resources within each detection frame belong, such as a certain construction machinery, construction material, or construction personnel, etc., and the identification serial number corresponding to the resources within each detection frame, such as 1, 2, 3, etc. The construction machinery can specifically include: excavation equipment or dump equipment, etc. The detection frame can be the circumscribed rectangle frame corresponding to the construction resource in the video image.

[0302] In step C2, since the spatial distribution of construction resources at the construction site is regular, the construction resources on the same construction operation surface often have relatively close spatial position characteristics in the input image. The clustering technique operates based on the center point coordinates corresponding to the detection frames, which are attributes that can reflect the spatial scale. Grouping the detection frames with relatively close spatial distances into one category, that is, clustering them into a cluster, means that these construction resources with similar spatial scales in the input image are very likely to be on the same construction operation surface. Because in the actual construction operation process, the spatial distances presented by the construction resources cooperating to complete the same construction process in the image are close. Through this clustering method, numerous construction resources at the entire construction site can be initially divided according to the construction operation surfaces they belong to, achieving the distinction of construction resources on different construction operation surfaces.

[0303] Assume that the detection frame is represented by the center point coordinates (x, y). Combine the center point coordinates of all detection frames to form a two-dimensional data set, in the format of:

[0304] data = [[x1, y1], [x2, y2],...]

[0305] Use the Manhattan distance as the similarity metric index. According to the optimization goal of minimizing the distance within the cluster, use the clustering algorithm to divide all detection frames into several clusters. The detection frames within the same cluster are considered to belong to the same construction operation surface. Based on the number of clusters, obtain the number of construction operation surfaces.

[0306] After clustering to divide each cluster (i.e., the construction operation surface), each cluster contains multiple detection frames, and each detection frame corresponds to a specific resource category. Step C3 can determine which specific construction resources are on each construction operation surface and the distribution of construction resources by counting the resource categories corresponding to each detection frame within the same cluster and the quantity of each resource category, etc.

[0307] The construction resource distribution information specifically includes: at least one resource category and the resource quantity corresponding to each resource category.

[0308] In the specific implementation, when meeting the preset call conditions, send a call request to the video image analysis module. The video image analysis module determines the construction resource distribution information corresponding to the construction operation surface at the construction site according to the video image of the construction site.

[0309] The preset call conditions specifically include:

[0310] The start of construction operations; and / or

[0311] The update of construction procedures; and / or

[0312] The change of construction resources; and / or

[0313] The next time period arrives.

[0314] During the construction modeling process, to accurately grasp the distribution information of construction resources on each construction operation surface at the construction site, it is necessary to use the video image analysis module for analysis and processing. When the preset call condition is met, a call request is sent to the video image analysis module. The purpose of doing this is to ensure that key information can be obtained in a timely manner while avoiding unnecessary resource waste (such as computing resources, time costs, etc.), enabling the analysis work to be carried out as needed, and accurately and efficiently obtaining the distribution of construction resources.

[0315] Among them, when the construction operation starts, a call request is sent to the video image analysis module to clarify the initial resource configuration of each construction operation surface, providing basic data for the instantiation of the construction simulation model.

[0316] When the construction process is updated, it means that the construction process has switched. Sending a call request to the video image analysis module can master whether the resource configuration under the new construction process is reasonable, ensuring the smooth connection of the construction and the on-demand allocation of resources.

[0317] When the construction resources change, sending a call request to the video image analysis module can quickly know the resource change situation of the construction operation surface.

[0318] When the next time period arrives, a call request is sent to the video image analysis module to regularly understand the dynamic changes of resources.

[0319] In the specific implementation, the construction resource distribution information specifically includes: at least one resource category and the corresponding resource quantity for each resource category;

[0320] Then the process of instantiating the queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information specifically includes:

[0321] Step C1: If the resource quantity of a resource category in the construction resource distribution information exceeds the maximum number of simultaneous operations, then perform the first sorting on the construction resource agent objects that exceed the maximum number of simultaneous operations according to the arrival time or priority to obtain the first sorting result;

[0322] Step C2: Save the first sorting result to the first queue corresponding to the queue elements in the corresponding construction process simulation model.

[0323] Among them, in the construction resource distribution information, when the quantity of a resource category exceeds the maximum number of simultaneous operations, in order to reasonably arrange these resources, a sorting mechanism needs to be introduced. Sorting by arrival time enables the construction resource agent objects that arrive first to be preferentially considered for use; sorting by priority highlights the important construction resource agent objects. In this way, the construction resource agent objects that exceed the quantity limit are sorted in an orderly manner.

[0324] On the one hand, the above-mentioned first sorting can make the use of construction resources more reasonable and orderly, avoid disorderly competition and chaotic allocation among resource agent objects, and ensure the orderly progress of the construction process. On the other hand, the above-mentioned first sorting creates good conditions for accurately recording the maintenance time of delay elements, making the statistics of relevant data such as resource usage duration more logical and accurate, and thus better reflecting the time utilization of construction resources in the entire construction process.

[0325] Step C2 stores the first sorting result obtained in Step C1 into the first queue corresponding to the queue element in the corresponding construction process simulation model. It realizes the orderly management of construction resource agent objects.

[0326] Optionally, the above method may further include:

[0327] In the case where the simulation of the construction process of the construction resource agent objects of a resource category is completed and the current working quantity of a resource category is less than the maximum number of simultaneous operations, the agent construction resource agent objects of the corresponding resource category are taken out from the first queue for further simulation of the construction process.

[0328] In the construction process simulation, taking out the construction resource agent objects of the corresponding resource category from the first queue needs to meet the following two conditions:

[0329] Condition 1: The simulation of the construction process of the construction resource agent objects of a resource category is completed, that is, some agent objects in this category of resources have completed the current construction tasks assigned to them and are in an idle state.

[0330] Condition 2: The current working quantity of a resource category is less than the maximum number of simultaneous operations, which indicates that there is still remaining available space for this category of resources, and more resource agent objects can be put into work.

[0331] When these two conditions are simultaneously met, the construction resource agent objects of the corresponding resource category are taken out from the first queue and put into the further simulation of the construction process to make full use of the resources and ensure the continuity and efficiency of the construction simulation.

[0332] In a specific implementation, the process of instantiating queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information specifically includes:

[0333] Step D1: Match the first construction resource agent object and the second construction resource agent object corresponding to the construction resource distribution information according to the cooperation relationship between the first resource category and the second resource category to obtain a number of matching pairs; a matching pair includes: a first construction resource agent object and a second construction resource agent object; the number of the matching pairs does not exceed the maximum number of simultaneous operations;

[0334] Step D2: Perform a second sorting on the first construction resource agent objects not in the matching pairs according to the arrival time or priority to obtain a second sorting result;

[0335] Step D3: Perform a third sorting on the second construction resource agent objects not in the matching pairs according to the arrival time or priority to obtain a third sorting result;

[0336] Step D4: Save the second sorting result to the second queue corresponding to the queue elements in the corresponding construction process simulation model, and save the third sorting result to the third queue corresponding to the queue elements in the corresponding construction process simulation model.

[0337] Among them, step D1 considers the cooperation relationship between different resource categories, pairs the relevant construction resource agent objects, and based on the cooperation, the role of the resources can be better exerted, and the number of matching pairs is controlled not to exceed the maximum number of simultaneous operations to ensure the reasonable utilization of resources. Step D1 forms a cooperative combination through matching, improves the resource utilization efficiency, conforms to the resource cooperation situation in actual construction, and helps to make the construction process simulation more accurate.

[0338] Step D2 sorts the first construction resource agent objects that do not participate in the matching pairs according to the arrival time or priority to determine the subsequent order of use.

[0339] Step D3 sorts the second construction resource agent objects not in the matching pairs according to the arrival time or priority to standardize their order of use.

[0340] Step D4 stores the sorted results into the corresponding second queue and third queue respectively. Using the characteristics of the queue to store the resource order can realize the orderly storage and management of different resources, and facilitate the subsequent extraction of resources in order for the construction process simulation.

[0341] In summary, in steps D1 to D4, according to the collaboration relationship and quantity of different resource categories in the construction resource distribution information, the construction resource agent objects are reasonably matched and sorted, and the results are saved to the corresponding queues, providing an orderly resource allocation plan that meets the actual collaboration requirements for the construction process simulation model, enabling the construction simulation to more accurately reflect the deployment and usage of resources in the actual construction process, thereby assisting construction management and decision-making, and improving construction efficiency and quality.

[0342] Optionally, the above method may further include: when the simulation of a construction process of a matching pair is completed and the number of working matching pairs is less than the maximum simultaneous working number, the construction resource agent objects of the corresponding resource categories are taken out from the second and third queues respectively for further simulation of the construction process.

[0343] During the construction process simulation, for the resource matching pairs formed according to the collaboration relationship before, when the simulation of a corresponding construction process of a certain matching pair is completed, the number of working matching pairs will be checked. If the number of working matching pairs is less than the pre-set maximum simultaneous working number, it means that there is still room to invest new resources to continue the construction simulation. At this time, the construction resource agent objects of the corresponding resource categories will be taken out from the second queue and the third queue respectively and added to ensure that the construction process simulation can continue and proceed orderly, so that the entire construction simulation is more in line with the actual situation of resource dynamic deployment and usage in the actual construction. Taking out the construction resource agent objects of the corresponding resource categories from the second and third queues respectively can form new matching pairs.

[0344] The delay element is used to count the duration of the construction process, while the queue element stores the sorting information of the construction resource agent objects. The state changes and operation sequence of the construction resource agent objects represented by the queue element during the construction process will affect the duration of the construction process, that is, the time counted by the delay element. For example, the waiting time of the construction resource agent object in the queue is part of the time counted by the delay element.

[0345] In a specific implementation, a delay element instance can be created for each construction process. The delay element instance can be a data structure for storing and accumulating relevant time information. For example, an object containing attributes such as total duration, total waiting time, and total operation time can be created.

[0346] Moreover, the calculation of the total duration, total waiting time, and total operation time can be performed.

[0347] Calculation of the total waiting time: According to the enqueue time and dequeue time of the queue element, calculate the waiting time of each construction resource agent object in the queue and accumulate it to the total waiting time attribute of the delay element.

[0348] Comprehensive calculation of operation time: According to the start time and end time of each construction operation executed by the construction resource agent object, calculate the duration of each construction operation, and accumulate it into the operation time sum attribute of the delay element.

[0349] Total duration calculation: Add the sum of waiting time and the sum of operation time to obtain the total duration of the construction process, and store it in the total duration attribute of the delay element.

[0350] During the construction process, as the construction resource agent objects continuously enter and leave the queue and execute construction operations, continuously update the time statistical information of the delay element.

[0351] In an alternative implementation of this application, the construction simulation model may further include: the connection relationship between different construction process simulation models; the connection relationship specifically includes: parallel connection.

[0352] The method may further include: determining the number of construction work surfaces at the construction site according to the video image of the construction site; instantiating the number of parallel construction process models according to the number of construction work surfaces.

[0353] The construction simulation model covers the parallel connection relationship of different construction process simulation models. By analyzing the video image of the construction site and using technologies such as image recognition, based on visual features such as construction resource distribution and construction activity range, the number of construction work surfaces can be determined. Since each construction work surface is relatively independent and has specific construction tasks, the parallel construction process models are instantiated according to the number of construction work surfaces, making the parallel processes in the construction simulation model match the parallel construction situation of multiple work surfaces in actual construction, so that the simulation is more in line with the actual construction state.

[0354] By instantiating the parallel construction process model according to the number of construction work surfaces, the degree of fit between the construction simulation model and actual construction can be significantly improved, accurately simulating the scenario of multiple construction work surfaces constructing simultaneously, providing a reliable reference for construction decision-making. At the same time, this method also helps to optimize the allocation and scheduling of construction resources, clearly presenting the resource requirements of each work surface, avoiding resource waste, and improving utilization efficiency.

[0355] In summary, the method for automatically constructing a construction simulation model based on video images according to the embodiments of the present application analyzes the video images of the construction site using object detection technology, object tracking technology, and clustering technology to quickly and automatically determine the construction resource distribution information, and uses the automatically determined construction resource distribution information for the instantiation of queue elements and delay elements. On the one hand, since the embodiments of the present application do not require manual counting of the construction resource distribution information one by one, the collection time of the construction resource distribution information can be shortened. On the other hand, in view of the fact that the embodiments of the present application omit the link of manually inputting the construction resource distribution information, the manual input time can be saved. In summary, the embodiments of the present application can improve the construction efficiency of the construction simulation model.

[0356] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0357] On the basis of the above embodiments, the present embodiment further provides a construction progress prediction device based on long-term video analysis. Refer to Figure 7 , the device may specifically include: a video analysis module 701, a video cropping module 702, an action segmentation module 703, a duration data determination module 704, a factual sample determination module 705, a random sample generation module 706, an assignment module 707, and a construction simulation module 708.

[0358] Among them, the video analysis module 701 is used to analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is the historical operation video of the construction operation, and / or, the relevant operation video is the historical operation video of the same category of operations of the construction operation;

[0359] The video cropping module 702 is used to crop the relevant operation video from the spatial dimension and the time dimension according to the equipment identification and detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery;

[0360] The action segmentation module 703 uses an action segmentation model to segment the construction operation video of each target construction machinery to obtain a plurality of construction operation segments arranged in chronological order; among them, two adjacent construction operation segments in time correspond to different action categories;

[0361] The duration data determination module 704 is configured to determine the duration data of the action category according to a plurality of construction operation segments arranged in chronological order;

[0362] The factual sample determination module 705 is configured to determine a factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set composed of multiple duration values generated in the frame images of the action category at different moments;

[0363] The random sample generation module 706 is configured to generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category;

[0364] The assignment module 707 is configured to assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration values corresponding to the random duration sample; the construction process parameters include: the unit operation duration corresponding to the target construction machinery;

[0365] The construction simulation module 708 is configured to simulate the construction simulation model according to the construction process parameters of the target construction machinery to obtain a construction progress prediction result.

[0366] Optionally, the random sample generation module 707 includes:

[0367] The distribution determination module is configured to determine the target distribution function corresponding to the action category according to the factual duration sample corresponding to the action category;

[0368] The rejection sampling module is configured to generate a random duration sample that conforms to the target distribution function by using the rejection sampling method.

[0369] Optionally, the rejection sampling module includes:

[0370] The first determination module is configured to determine the reference distribution function and the constant; under the condition of the same independent variable, the function value of the target distribution function is not greater than the product of the function value of the reference distribution function and the constant;

[0371] The first extraction module is configured to extract a random independent variable from the reference distribution function;

[0372] The second extraction module is configured to extract a random number from the uniform distribution function; the upper limit value of the uniform distribution function is: the product of the function value of the reference distribution function and the constant under the condition of the random independent variable;

[0373] A determination module, configured to use the random variable as a randomness duration sample conforming to the target distribution function if the random number is less than the function value of the target distribution function corresponding to the random variable.

[0374] Optionally, the video cropping module includes:

[0375] A start and end image determination module, configured to determine the start frame image and the end frame image of each target construction machine in the relevant operation video;

[0376] A maximum bounding box determination module, configured to determine the maximum bounding box of each target construction machine according to the detection boxes of each target construction machine in the start frame image, the middle frame image, and the end frame image;

[0377] A multi-dimensional cropping module, configured to crop the relevant operation video from the time dimension according to the start frame image and the end frame image, and crop the relevant operation video from the space dimension according to the maximum bounding box, to obtain the construction operation video of each target construction machine.

[0378] Optionally, the action segmentation module includes:

[0379] A video segment acquisition module, configured to use a sliding window with a preset window size and a preset step length to acquire multiple video segments from the construction operation video of each target construction machine;

[0380] A multi-dimensional extraction module, configured to extract the RGB features and the optical flow features respectively corresponding to the multiple video segments;

[0381] A feature fusion module, configured to fuse the RGB features and the optical flow features respectively corresponding to the multiple video segments to obtain the three-dimensional temporal features respectively corresponding to the multiple video segments.

[0382] A first feature processing module, configured to perform one-dimensional convolution processing, self-attention processing, and feed-forward processing on the three-dimensional temporal features in sequence to obtain the fusion region features corresponding to each target construction machine;

[0383] A noise feature processing module, configured to extract features from the noise distribution at the s-th time step to obtain the noise distribution features at the s-th time step;

[0384] A noise time fusion module, configured to fuse the noise distribution features with the time encoding vector to obtain the noise time distribution features at the s-th time step;

[0385] A second feature processing module, configured to extract features from the noise time distribution features at the s-th time step to obtain the noise representation at the s-th time step;

[0386] The cross-attention processing module is used to take the concatenation of the noise representation at the s-th time step and the fused region features as the query vector and the key vector, and the noise representation at the s-th time step as the value vector, and input them into the cross-attention layer. The cross-attention layer calculates the attention weights between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weights to obtain the cross-attention result;

[0387] The fusion processing module is used to fuse the cross-attention result and the noise representation at the s-th time step to obtain the fused representation;

[0388] The result determination module is used to process the fused representation by using a feed-forward connection layer and a convolutional layer to obtain the action category at the s-th time step.

[0389] Optionally, the construction simulation model includes: a construction unit intelligent agent object, a construction resource scheduling intelligent agent object, a construction resource intelligent agent object, and construction process simulation models respectively corresponding to a plurality of construction unit intelligent agent objects; the construction process simulation model includes: a construction process simulation model; the construction process simulation model includes: a queue element and a delay element; the queue element is used to simulate the queuing and waiting process of the construction resource intelligent agent object in the construction process; the delay element is used to simulate the duration of the construction process;

[0390] The simulation process of the construction simulation model includes: the construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task assignment message to at least one selected target construction resource intelligent agent object according to the resource request message; after receiving the task assignment message, at least one target construction resource intelligent agent object moves to the target construction work surface where the construction unit intelligent agent object is located; the construction resource intelligent agent object that meets the set conditions at the target construction work surface is put into the queue represented by the queue element for queuing and waiting; the construction simulation of the construction process is performed by using the construction resource intelligent agent object that does not need to queue and wait or is taken out from the queue element at the target construction work surface; after the simulation of the construction process of the construction unit intelligent agent object is completed, the corresponding delay element ends;

[0391] The device further includes:

[0392] The resource analysis module determines the construction resource distribution information corresponding to the construction work surface at the construction operation site by using object detection technology, object tracking technology and clustering technology according to the video image of the construction operation site;

[0393] The instantiation module is used to instantiate the queue element and the delay element in the corresponding construction process simulation model according to the construction resource distribution information.

[0394] Optionally, the resource analysis module includes:

[0395] A target detection module for performing target detection on a video image to obtain detection frames and resource categories corresponding to multiple construction resources in the video image;

[0396] A clustering module for clustering multiple construction resources in the video image according to the width and height corresponding to the detection frames to obtain a number of clusters; the detection frames within the same cluster belong to the same construction operation surface;

[0397] A distribution determination module for determining the construction resource distribution information corresponding to the construction operation surface at the construction work site according to the resource categories corresponding to the detection frames within the same cluster.

[0398] Optionally, the construction simulation model further includes: the connection relationship between different construction process simulation models; the connection relationship includes: parallel connection;

[0399] The device further includes:

[0400] An operation surface quantity determination module for determining the quantity of construction operation surfaces at the construction work site according to the video image of the construction work site;

[0401] A quantity instantiation module for instantiating the quantity of parallel construction process models according to the quantity of construction operation surfaces.

[0402] Optionally, the set conditions include:

[0403] For the first construction resource agent object at the target construction operation surface, the first construction resource agent object corresponds to the first resource category, and the quantity of the first type of construction resource agent object at the target construction operation surface exceeds the maximum simultaneous working quantity; and / or

[0404] For the first construction resource agent object at the target construction operation surface, the first construction resource agent object corresponds to the first resource category, and the construction resource agent objects of the first resource category cooperate with the construction resource agent objects of the second resource category to jointly simulate the construction process.

[0405] Optionally, the construction resource distribution information includes: at least one resource category, and the resource quantity corresponding to each resource category;

[0406] The instantiation module includes:

[0407] A first sorting module for, if the resource quantity of a resource category in the construction resource distribution information exceeds the maximum simultaneous working quantity, sorting the construction resource agent objects exceeding the maximum simultaneous working quantity according to the arrival time or priority to obtain a first sorting result;

[0408] A first storage module, configured to store the first sorting result into a first queue corresponding to a queue element in a corresponding construction process simulation model.

[0409] Optionally, the apparatus further includes:

[0410] A dequeue module, configured to, when a construction resource agent object of a resource category completes the simulation of a construction process and the current working quantity of a resource category is less than the maximum simultaneous working quantity, take out the agent construction resource agent object of the corresponding resource category from the first queue for further simulation of the construction process.

[0411] Optionally, the construction resource distribution information includes: at least one resource category and the resource quantity corresponding to each resource category;

[0412] The instantiation module includes:

[0413] A matching module, configured to match a first construction resource agent object and a second construction resource agent object corresponding to the construction resource distribution information according to the cooperation relationship between the first resource category and the second resource category to obtain a plurality of matching pairs; a matching pair includes: a first construction resource agent object and a second construction resource agent object; the number of the matching pairs does not exceed the maximum simultaneous working quantity;

[0414] A second sorting module, configured to perform a second sorting on the first construction resource agent objects not in the matching pairs according to the arrival time or priority to obtain a second sorting result;

[0415] A third sorting module, configured to perform a third sorting on the second construction resource agent objects not in the matching pairs according to the arrival time or priority to obtain a third sorting result;

[0416] An enqueue module, configured to store the second sorting result into a second queue corresponding to a queue element in a corresponding construction process simulation model, and store the third sorting result into a third queue corresponding to a queue element in a corresponding construction process simulation model.

[0417] An embodiment of the present application further provides a non-volatile readable storage medium, in which one or more modules (prograPs) are stored, and when the one or more modules are applied to a device, instructions (instructions) for enabling the device to execute the method steps in the embodiments of the present application can be obtained.

[0418] Embodiments of the present application provide one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In embodiments of the present application, the electronic device includes various types of devices such as a terminal device, a server (cluster), etc.

[0419] Embodiments of the present application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any one of the construction progress prediction methods based on long-term video analysis described in the above embodiments.

[0420] Embodiments of the present disclosure can be implemented as a device configured with any suitable hardware, firmware, software, or any combination thereof, and the device may include: electronic devices such as a terminal device, a server (cluster), etc. Figure 8 Exemplary device 1300 that can be used to implement the various embodiments described in the present application is schematically shown.

[0421] For one embodiment, Figure 8 Exemplary device 1300 is shown, which has one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, an NVP (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0422] Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of a general-purpose processor or a dedicated processor (such as a graphics processor, an application processor, a baseband processor, etc.). In some embodiments, device 1300 can act as the terminal device, server (cluster), etc. described in embodiments of the present application.

[0423] In some embodiments, device 1300 may include one or more computer-readable media (such as memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 combined with the one or more computer-readable media and configured to execute the instructions 1314 to implement modules to perform the actions described in the present disclosure.

[0424] For one embodiment, control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the (one or more) processors 1302 and / or any suitable device or component communicating with control module 1304.

[0425] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0426] The memory 1306 may be used to load and store data and / or instructions 1314 for the device 1300, for example. For one embodiment, the memory 1306 may include any suitable volatile memory, such as, for example, suitable DRAP (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include double data rate type four synchronous dynamic random access memory.

[0427] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the (one or more) input / output devices 1310.

[0428] For example, the non-volatile memory / storage device 1308 may be used to store data and / or instructions 1314. The non-volatile memory / storage device 1308 may include any suitable non-volatile memory (such as flash memory) and / or may include any suitable (one or more) non-volatile storage devices (such as one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile disk drives).

[0429] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the device 1300 is mounted, or it may be accessible by the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via the (one or more) input / output devices 1310 over a network.

[0430] (One or more) Input / output devices 1310 can provide an interface for device 1300 to communicate with any other suitable devices. The input / output devices 1310 can include communication components, audio components, sensor components, etc. The network interface 1312 can provide an interface for device 1300 to communicate through one or more networks. Device 1300 can wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on communication standards, such as WiFi (Wireless Fidelity), 2G (2-Generation wireless telephone technology), 3G (3-Generation wireless telephone technology), 4G (4-Generation wireless telephone technology), 5G (5-Generation wireless telephone technology), etc., or a combination thereof for wireless communication.

[0431] For one embodiment, at least one of (one or more) processors 1302 can be logically packaged together with one or more controllers (e.g., memory controller modules) of the control module 1304. For one embodiment, at least one of (one or more) processors 1302 can be logically packaged together with one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of (one or more) processors 1302 can be logically integrated with one or more controllers of the control module 1304 on the same die. For one embodiment, at least one of (one or more) processors 1302 can be logically integrated with one or more controllers of the control module 1304 on the same die to form a system-on-chip.

[0432] In various embodiments, device 1300 can be, but is not limited to: a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touchscreen device, a netbook, etc.) and other terminal devices. In various embodiments, device 1300 can have more or fewer components and / or a different architecture. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit, and speakers.

[0433] Among them, a main control chip can be used as a processor or a control module in the detection device. Sensor data, position information, etc. are stored in a memory or a non-volatile memory / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0434] For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiments.

[0435] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0436] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general computer, a special computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for realizing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0437] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device realizes the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0438] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable terminal device provide steps for realizing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0439] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0440] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the said element.

[0441] The above has introduced in detail a construction progress prediction method and device, an electronic device and a machine-readable medium based on long-term video analysis provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scenarios. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A construction progress prediction method based on long-term video analysis, characterized in that, The method includes: Analyze relevant operation videos of the construction operation to obtain the equipment category, equipment identification, and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is the historical operation video of the construction operation, and / or, the relevant operation video is the historical operation video of the same category of operations as the construction operation; According to the equipment identification and detection frame of the target construction machinery, crop the relevant operation video from the spatial dimension and the time dimension to obtain the construction operation video of each target construction machinery; Use an action segmentation model to segment the construction operation video to obtain multiple construction operation segments arranged in chronological order; among them, two adjacent construction operation segments in time correspond to different action categories; Determine the duration data of the action category according to the multiple construction operation segments arranged in chronological order; According to the duration data, determine the factual duration sample corresponding to the action category; the factual duration sample includes: a data set composed of multiple duration values generated in the frame images at different moments of the action category; Generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category; Assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration values corresponding to the random duration sample; the construction process parameters include: the unit operation duration corresponding to the target construction machinery; Perform simulation of the construction simulation model according to the construction process parameters of the target construction machinery to obtain the construction progress prediction result; Among them, the generating a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category includes: determining the target distribution function corresponding to the action category according to the factual duration sample corresponding to the action category; using the rejection sampling method to generate a random duration sample that conforms to the target distribution function.

2. The method according to claim 1, wherein The using the rejection sampling method to generate a random duration sample that conforms to the target distribution function includes: Determine the reference distribution function and the constant; under the condition of the same independent variable, the function value of the target distribution function is not greater than the product of the function value of the reference distribution function and the constant; Extract a random independent variable from the reference distribution function; Extract a random number from the uniform distribution function; the upper limit value of the uniform distribution function is: the product of the function value of the reference distribution function and the constant under the condition of the random independent variable; If the random number is less than the function value of the target distribution function corresponding to the random independent variable, then use the random independent variable as the random duration sample that conforms to the target distribution function.

3. The method according to claim 1, wherein The crop the relevant operation video from the spatial dimension and the time dimension according to the equipment identification and detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery includes: Determine the starting frame image and the ending frame image of each target construction machinery in the relevant operation video; Determine the maximum bounding box of each target construction machine according to the detection boxes of each target construction machine in the starting frame image, the intermediate frame image, and the ending frame image; Crop the relevant operation video in the time dimension according to the starting frame image and the ending frame image, and crop the relevant operation video in the space dimension according to the maximum bounding box to obtain the construction operation video of each target construction machine.

4. The method according to claim 1, wherein The using of the action segmentation model to segment the construction operation video of each target construction machine includes: Extract the three-dimensional temporal features of the construction operation video of each target construction machine; Perform one-dimensional convolution processing, self-attention processing, and feed-forward processing on the three-dimensional temporal features in sequence to obtain the fused region features corresponding to each target construction machine; Use the denoising network model to determine the noise distribution at the s-th time step of the fused region features, and determine the action category at the s-th time step; According to the action categories corresponding to each time step, segment the construction operation video of each target construction machine into multiple construction operation segments arranged in chronological order.

5. The method according to claim 4, wherein The using of the denoising network model to determine the noise distribution at the s-th time step of the fused region features and determine the action category at the s-th time step includes: Extract the features of the noise distribution at the s-th time step to obtain the noise distribution features at the s-th time step; Fuse the noise distribution features with the time encoding vector to obtain the noise time distribution features at the s-th time step; Extract the features of the noise time distribution features at the s-th time step to obtain the noise representation at the s-th time step; The noise representation at the s-th time step is concatenated with the fused region features as the query vector and the key vector, and the noise representation at the s-th time step is used as the value vector and input into the cross-attention layer. The cross-attention layer calculates the attention weights between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weights to obtain the cross-attention result; Fuse the cross-attention result with the noise representation at the s-th time step to obtain the fused representation; Use the feed-forward connection layer and the convolutional layer to process the fused representation to obtain the action category at the s-th time step.

6. The method according to any one of claims 1 to 5, characterized in that, The construction simulation model includes: a construction unit intelligent agent object, a construction resource scheduling intelligent agent object, a construction resource intelligent agent object, and construction process simulation models corresponding to multiple construction unit intelligent agent objects respectively; the construction process simulation model includes: a construction process simulation model; the construction process simulation model includes: a queue element and a delay element; the queue element is used to simulate the queuing and waiting process of the construction resource intelligent agent object in the construction process; the delay element is used to simulate the duration of the construction process; The simulation process of the construction simulation model includes: the construction unit agent object sends a resource request message to the construction resource scheduling agent object according to the corresponding construction process simulation model; the construction resource scheduling agent object sends a task assignment message to at least one selected target construction resource agent object according to the resource request message; after receiving the task assignment message, at least one target construction resource agent object moves to the target construction operation surface where the construction unit agent object is located; the construction resource agent objects that meet the set conditions at the target construction operation surface are put into the queue represented by the queue element for queuing; the construction simulation of the construction process is performed by using the construction resource agent objects that do not need to queue or are taken out from the queue element at the target construction operation surface; after the simulation of the construction process of the construction unit agent object is completed, the corresponding delay element ends; The method further includes: According to the video image of the construction site, using object detection technology, object tracking technology and clustering technology to determine the construction resource distribution information corresponding to the construction operation surface of the construction site; According to the construction resource distribution information, instantiate the queue element and the delay element in the corresponding construction process simulation model.

7. The method according to claim 6, wherein The step of using object detection technology, object tracking technology and clustering technology to determine the construction resource distribution information corresponding to the construction operation surface of the construction site according to the video image of the construction site includes: Performing object detection on the video image to obtain the detection frames and resource categories corresponding to multiple construction resources in the video image; Clustering the multiple construction resources in the video image according to the width and height corresponding to the detection frame to obtain several clusters; the detection frames within the same cluster belong to the same construction operation surface; According to the resource categories corresponding to the detection frames within the same cluster, determine the construction resource distribution information corresponding to the construction operation surface of the construction site.

8. The method according to claim 6, characterized in that The construction simulation model further includes: the connection relationship between different construction process simulation models; the connection relationship includes: parallel connection; The method further includes: According to the video image of the construction site, determine the number of construction operation surfaces at the construction site; According to the number of construction operation surfaces, instantiate the number of parallel construction process models.

9. The method according to claim 6, wherein The set conditions include: For the first construction resource agent object at the target construction operation surface, the first construction resource agent object corresponds to the first resource category, and the number of construction resource agent objects of the first type at the target construction operation surface exceeds the maximum number of simultaneous operations; and / or For the first construction resource agent object at the target construction operation surface, the first construction resource agent object corresponds to the first resource category, and the construction resource agent objects of the first resource category cooperate with the construction resource agent objects of the second resource category to jointly perform the construction simulation of the construction process.

10. The method according to claim 6, characterized in that, The construction resource distribution information includes: at least one resource category, and the resource quantity corresponding to each resource category; The step of instantiating the queue element and the delay element in the corresponding construction process simulation model according to the construction resource distribution information includes: If the number of resources of a resource category in the construction resource distribution information exceeds the maximum number of simultaneous operations, the construction resource agent objects that exceed the maximum number of simultaneous operations are sorted first according to the arrival time or priority to obtain a first sorting result; Save the first sorting result to the first queue corresponding to the queue element in the corresponding construction process simulation model.

11. The method according to claim 10, wherein The method further includes: When the simulation of the construction process of the construction resource agent object of a resource category is completed and the current number of operations of a resource category is less than the maximum number of simultaneous operations, the agent construction resource agent object of the corresponding resource category is taken out from the first queue for the simulation of the construction process.

12. The method according to claim 6, characterized in that The construction resource distribution information includes: at least one resource category and the number of resources corresponding to each resource category; The instantiation of the queue element and the delay element in the corresponding construction process simulation model according to the construction resource distribution information includes: According to the cooperation relationship between the first resource category and the second resource category, the first construction resource agent object and the second construction resource agent object corresponding to the construction resource distribution information are matched to obtain a number of matching pairs; a matching pair includes: a first construction resource agent object and a second construction resource agent object; the number of the matching pairs does not exceed the maximum number of simultaneous operations; Sort the first construction resource agent objects not in the matching pairs second according to the arrival time or priority to obtain a second sorting result; Sort the second construction resource agent objects not in the matching pairs third according to the arrival time or priority to obtain a third sorting result; Save the second sorting result to the second queue corresponding to the queue element in the corresponding construction process simulation model, and save the third sorting result to the third queue corresponding to the queue element in the corresponding construction process simulation model.

13. A construction progress prediction device based on long-term video analysis, characterized in that, The device includes: A video analysis module for analyzing the relevant operation video of the construction operation to obtain the equipment category, equipment identifier and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is the historical operation video of the construction operation, and / or, the relevant operation video is the historical operation video of the same category of operations of the construction operation; A video cropping module for cropping the relevant operation video in the spatial dimension and time dimension according to the equipment identifier and detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery; An action segmentation module for using an action segmentation model to segment the construction operation video of each target construction machinery to obtain a plurality of construction operation segments arranged in chronological order; wherein, two adjacent construction operation segments in time correspond to different action categories; A duration data determination module for determining the duration data of the action category according to the plurality of construction operation segments arranged in chronological order; A factual sample determination module, configured to determine a factual duration sample corresponding to an action category according to the duration data; the factual duration sample includes: a data set composed of multiple duration values generated by the action category in frame images at different moments; A random sample generation module, configured to generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category; An assignment module, configured to assign values to the construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration values corresponding to the random duration samples; the construction process parameters include: the unit operation duration corresponding to the target construction machinery; A construction simulation module, configured to simulate the construction simulation model according to the construction process parameters of the target construction machinery to obtain a construction progress prediction result; The random sample generation module includes: A distribution determination module, configured to determine a target distribution function corresponding to the action category according to the factual duration sample corresponding to the action category; A rejection sampling module, configured to generate a random duration sample that conforms to the target distribution function by using the rejection sampling method.

14. An electronic device, characterized in that, Including: A processor; And A memory, on which executable code is stored, and when the executable code is executed, the processor is caused to execute the method according to any one of claims 1-12.

15. A machine-readable medium, on which executable code is stored, and when the executable code is executed, a processor is caused to execute the method according to any one of claims 1-12.

16. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the method according to any one of claims 1-12 is implemented.

Images