Construction progress prediction method and device based on long-time video analysis

Abstract

The embodiment of the invention provides a construction progress prediction method and device based on long-time video analysis, and the method specifically comprises the steps: carrying out the analysis of a related operation video of construction operation, and obtaining an equipment type, an equipment identifier and a detection frame of a target construction machine; cutting the related operation video from a space dimension and a time dimension to obtain a construction operation video of each target construction machine; segmenting the construction operation video of each target construction machine; determining a factual duration sample of the action category according to the plurality of construction operation segments arranged according to the time sequence; generating a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category; and according to the duration value corresponding to the random duration sample, assignment is performed on construction process parameters of the target construction machinery in the construction simulation model, and simulation of the construction simulation model is performed. According to the embodiment of the invention, the accuracy of construction progress prediction can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of construction progress prediction, and in particular, to a construction progress prediction method and device based on long-term video analysis. Background Art

[0002] In engineering construction, accurate and efficient construction progress forecasting is crucial. With accurate progress forecasting, the engineering team can reasonably allocate human, material, financial and other resources according to the actual needs of each construction stage, improve resource utilization efficiency and reduce costs. For example, in the earth excavation stage, the appropriate amount of construction machinery and construction personnel can be arranged according to the progress forecast to avoid idle or shortage of resources.

[0003] At present, construction progress prediction mainly relies on construction simulation technology. By simulating the construction process of the project on a computer, construction simulation technology can formulate the construction schedule, reasonably allocate construction resources, discover potential construction problems in advance, and provide strong guidance for engineering construction decisions. In actual operation, the construction of the construction simulation model is the core link, and the value of the construction simulation parameter is the key factor in determining the accuracy of the model. In related technologies, the value of the construction simulation parameter is usually assigned based on the construction experience of similar projects. Taking the construction process parameters of construction machinery as an example, related technologies usually set the unit excavation time of the excavation equipment to a fixed value that conforms to the construction experience.

[0004] However, in actual construction, excavation equipment is affected by geological conditions (such as soft soil, hard rock and other different textures), site space (width, openness, etc.) and the equipment's own working conditions (newness, wear and tear, etc.), so its construction process parameters are usually in a state of change. If these construction process parameters are set to fixed values, the resulting construction process parameters are likely to deviate from the actual situation, which will affect the accuracy of the construction progress forecast. Summary of the invention

[0005] The embodiment of the present application provides a construction progress prediction method based on long-term video analysis, which can effectively improve the accuracy of construction progress prediction.

[0006] Correspondingly, the embodiments of the present application also provide a construction progress prediction device based on long-term video analysis, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application embodiment discloses a construction progress prediction method based on long-term video analysis, including: Analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is a historical operation video of the construction operation, and / or the relevant operation video is a historical operation video of the same type of operation as the construction operation; According to the equipment identification and detection frame of the target construction machinery, the related operation video is cropped from the spatial dimension and the temporal dimension to obtain the construction operation video of each target construction machinery; Using the action segmentation model, the construction operation video of each target construction machine is segmented to obtain multiple construction operation clips arranged in chronological order; wherein two construction operation clips adjacent in time correspond to different action categories; Determine duration data of action categories based on a plurality of construction work segments arranged in chronological order; Determine the factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set consisting of multiple duration values ​​generated by the action category in the frame images at different times; Generate random duration samples corresponding to the action categories according to the factual duration samples corresponding to the action categories; According to the duration value corresponding to the random duration sample, the construction process parameters of the target construction machinery in the construction simulation model of the construction operation are assigned; the construction process parameters include: the unit operation time corresponding to the target construction machinery; According to the construction process parameters of the target construction machinery, a construction simulation model is simulated to obtain a construction progress prediction result.

[0008] The embodiment of the present application also discloses a construction progress prediction device based on long-term video analysis, the device comprising: The video analysis module is used to analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is a historical operation video of the construction operation, and / or the relevant operation video is a historical operation video of the same type of operation as the construction operation; A video cropping module is used to crop the relevant operation video from the spatial dimension and the temporal dimension according to the equipment identification and the detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery; The action segmentation module is used to segment the construction operation video of each target construction machine using the action segmentation model to obtain multiple construction operation segments arranged in chronological order; wherein two construction operation segments adjacent in time correspond to different action categories; A duration data determination module, used to determine duration data of an action category according to a plurality of construction operation segments arranged in chronological order; A factual sample determination module is used to determine the factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set consisting of multiple duration values ​​generated by the action category in the frame images at different moments; A random sample generation module, used for generating random duration samples corresponding to action categories according to factual duration samples corresponding to action categories; An assignment module is used to assign a construction process parameter of a target construction machinery in a construction simulation model of a construction operation according to a duration value corresponding to the random duration sample; the construction process parameter includes: a unit operation time corresponding to the target construction machinery; The construction simulation module is used to simulate the construction simulation model according to the construction process parameters of the target construction machinery to obtain the construction progress prediction result.

[0009] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0010] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0011] The embodiment of the present application also discloses a computer program product, including a computer program / instruction, which implements the aforementioned method when executed by a processor.

[0012] The embodiments of the present application include the following advantages: The technical solution of the embodiment of the present application starts from the construction operation video and obtains the construction progress prediction result through multi-step processing. First, the relevant operation videos of the construction operation (including the historical operation videos of the construction operation and / or the historical operation videos of the same type of operation, and the duration is greater than the duration threshold) are analyzed to obtain the equipment category, equipment identification and detection frame of the target construction machinery; then the construction operation video of each target construction machinery is cut from the time and space dimensions; then the action segmentation model is used to segment the construction operation video of each target construction machinery to obtain multiple construction operation segments arranged in chronological order, and the duration data is determined according to the construction operation segments, and then the factual and random duration samples are obtained; the duration value corresponding to the random duration sample is used to assign the construction process parameters (such as unit operation duration) of the target construction machinery in the construction simulation model; finally, the construction simulation model is simulated to obtain the construction progress prediction result.

[0013] The embodiment of the present application is based on the construction operation related video, and determines the parameter value of the construction process parameter. Since the construction operation related video of the embodiment of the present application records various actions of the target construction machinery in the construction process, based on the analysis of the construction operation related video, it is possible to obtain real data such as action category and factual duration sample from the construction operation related video, and it is possible to generate a random duration sample by processing these real data. Since the random duration sample includes various uncertainties and changing factors in the construction process, it is used for the assignment of the construction process parameter, and the parameter value of the construction process parameter can be made closer to the real state of the construction site. In this way, the construction simulation model including the construction process parameter can simulate the construction progress more accurately based on the parameters close to the real state. Therefore, the embodiment of the present application can effectively improve the accuracy of the construction progress prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of the steps of a construction progress prediction method based on long-term video analysis according to an embodiment of the present application; Figure 2 is a structural diagram of a denoising network model according to an embodiment of the present application; Figure 3 It is a schematic diagram of the steps of a method for automatically building a construction simulation model based on video images according to an embodiment of the present application; FIG4 (a) and FIG4 (b) are schematic diagrams of modeling of a construction process of an embodiment of the present application; Figure 5 It is a schematic diagram of a construction process simulation model of excavation blocks in an embodiment of the present application; Figure 6 is a schematic diagram of a construction simulation model of an embodiment of the present application; Figure 7 It is a structural schematic diagram of a construction progress prediction device based on long-term video analysis according to an embodiment of the present application; Figure 8 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0016] The embodiments of the present application can be applied to the hydropower engineering industry for predicting the construction progress of construction operations based on computer simulation. Construction progress prediction refers to the use of scientific methods and tools to comprehensively consider multiple factors such as resource input, process technology, environmental conditions, historical data, etc. in the construction process, and to estimate the time progress and completion status of each stage and the whole of the construction operation, and to provide the time nodes of each construction process, progress deviation analysis, risk warning and other results, so as to provide decision-making basis for construction resource allocation, construction period planning, and cost control, and to ensure the timely and efficient completion of the project.

[0017] At present, the construction simulation model is usually simulated according to the construction simulation parameters of the construction simulation model, thereby realizing the construction progress prediction. In the related art, the values ​​of the construction simulation parameters are usually assigned based on the construction experience of similar projects. Taking the construction process parameters of construction machinery as an example, the related art usually sets the unit excavation time of the excavation equipment to a fixed value that conforms to the construction experience.

[0018] However, in actual construction, excavation equipment is affected by geological conditions (such as soft soil, hard rock and other different textures), site space (width, openness, etc.) and the equipment's own working conditions (newness, wear and tear, etc.), so its construction process parameters are usually in a state of change. If these construction process parameters are set to fixed values, the resulting construction process parameters are likely to deviate from the actual situation, which will affect the accuracy of the construction progress forecast.

[0019] In view of the technical problem of low accuracy of construction progress prediction in the related art, the embodiment of the present application provides a construction progress prediction method based on long-term video analysis, which specifically includes the following steps: Analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery is a construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is a historical operation video of the construction operation, and / or the relevant operation video is a historical operation video of the same category of the construction operation; According to the equipment identification and detection frame of the target construction machinery, the above-mentioned related operation videos are cropped from the spatial dimension and the temporal dimension to obtain the construction operation video of each target construction machinery; Using the action segmentation model, the construction operation video of each target construction machine is segmented to obtain multiple construction operation clips arranged in chronological order; wherein two construction operation clips adjacent in time correspond to different action categories; Determine duration data of action categories based on a plurality of construction work segments arranged in chronological order; According to the duration data, a factual duration sample corresponding to the action category is determined; the factual duration sample includes: a data set consisting of a plurality of duration values ​​generated in frame images of the action category at different times; Generate random duration samples corresponding to the action categories according to the factual duration samples corresponding to the action categories; According to the duration value corresponding to the random duration sample, the construction process parameters of the target construction machinery in the construction simulation model of the construction operation are assigned; the construction process parameters include: the unit operation time corresponding to the target construction machinery; According to the construction process parameters of the above-mentioned target construction machinery, the construction simulation model is simulated to obtain the construction progress prediction result.

[0020] The embodiment of the present application starts from the construction operation video and obtains the construction progress prediction result through multi-step processing. First, the relevant operation videos of the construction operation (including the historical operation videos of the construction operation and / or the historical operation videos of the same type of operation, and the duration is greater than the duration threshold) are analyzed to obtain the equipment category, equipment identification and detection frame of the target construction machinery; then the construction operation video of each target construction machinery is cut from the time and space dimensions; then the action segmentation model is used to segment the construction operation video of each target construction machinery to obtain multiple construction operation segments arranged in chronological order, and the duration data is determined according to the construction operation segments, and then the factual and random duration samples are obtained; the duration value corresponding to the random duration sample is used to assign the construction process parameters (such as unit operation duration) of the target construction machinery in the construction simulation model; finally, the construction simulation model is simulated to obtain the construction progress prediction result.

[0021] The embodiment of the present application is based on the construction operation related video, and determines the parameter value of the construction process parameter. Since the construction operation related video of the embodiment of the present application records various actions of the target construction machinery in the construction process, based on the analysis of the construction operation related video, it is possible to obtain real data such as action category and factual duration sample from the construction operation related video, and it is possible to generate a random duration sample by processing these real data. Since the random duration sample includes various uncertainties and changing factors in the construction process, it is used for the assignment of the construction process parameter, and the parameter value of the construction process parameter can be made closer to the real state of the construction site. In this way, the construction simulation model including the construction process parameter can simulate the construction progress more accurately based on the parameters close to the real state. Therefore, the embodiment of the present application can effectively improve the accuracy of the construction progress prediction.

[0022] Method Example 1 refer to Figure 1 , shows a schematic diagram of the steps of a construction progress prediction method based on long-term video analysis according to an embodiment of the present application, the method specifically comprises the following steps: Step 101: Analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery may be a construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video may be a historical operation video of the construction operation, and / or the relevant operation video may be a historical operation video of the same type of operation as the construction operation; Step 102: According to the equipment identification and detection frame of the target construction machinery, the above-mentioned related operation videos are cut from the spatial dimension and the temporal dimension to obtain the construction operation video of each target construction machinery; Step 103: using the action segmentation model, segment the construction operation video of each target construction machine to obtain a plurality of construction operation segments arranged in chronological order; wherein two construction operation segments adjacent in time correspond to different action categories; Step 104: determining duration data of the action category according to the multiple construction operation segments arranged in chronological order; Step 105: determining a factual duration sample corresponding to the action category according to the duration data; the factual duration sample specifically includes: a data set consisting of a plurality of duration values ​​generated in frame images of the action category at different moments; Step 106: Generate a random duration sample corresponding to the action category according to the factual duration sample corresponding to the action category; Step 107: assigning values ​​to construction process parameters of the target construction machinery in the construction simulation model of the construction operation according to the duration value corresponding to the random duration sample; the construction process parameters include: the unit operation time corresponding to the target construction machinery; Step 108: simulate the construction simulation model according to the construction process parameters of the target construction machinery to obtain a construction progress prediction result.

[0023] Figure 1 The method shown can be used to predict the construction progress of construction operations based on computer simulation. In the field of engineering, construction operations refer to a series of production activities organized and planned at the construction site in order to achieve the requirements of the engineering design, using various construction resources (such as manpower, materials, mechanical equipment, etc.) in accordance with specific construction processes, processes and specifications. For example, in a hydropower engineering scenario, examples of construction operations may include: slope excavation operations, dam pouring operations, or tunnel excavation operations, etc. It can be understood that the embodiments of the present application do not limit specific construction operations.

[0024] A construction operation usually includes multiple construction processes. A construction process refers to the process of dividing the entire construction operation into several interrelated and mutually restrictive specific construction steps or operation processes in accordance with a certain construction sequence and construction method in order to complete a certain engineering task during the construction process.

[0025] In the embodiment of the present application, a long video may refer to a video content whose duration is greater than a duration threshold. The duration threshold may be determined by a person skilled in the art according to actual needs, for example, the duration threshold is 30 points.

[0026] In step 101, the target construction machine may be a construction machine that restricts the progress of the construction process. The process of determining the target construction machine is as follows: If only one type of construction machinery is involved in the construction process, then this construction machinery is the target construction machinery that restricts the progress of the construction process. For example, if a drilling rig is used for drilling operations, then the drilling rig can be used as the target construction machinery for the drilling operations.

[0027] If the construction process is completed by multiple construction machines, the construction machine that provides services from the perspective of queuing theory is the target construction machine that restricts the progress of the construction process. For example, when using excavators and dump trucks to complete the excavation and loading of slope earth and stone, the excavator provides excavation and loading services, and the dump trucks queue up to receive the loading services of the excavator. Accordingly, the excavator is considered to be the target construction machine that restricts the progress of earth and stone excavation and loading operations.

[0028] The embodiment of the present application can collect relevant operation videos of the construction operation according to the target construction machinery involved in the construction operation and the operation category of the construction operation. The embodiment of the present application does not limit the number of relevant operation videos collected. The relevant operation video can be one or more.

[0029] Specifically, we can first search in the video database according to the job category of the construction work to obtain candidate videos corresponding to the job category. The video database contains various related data sets such as the internal database of the construction enterprise and the general industry database. Then, we filter the candidate videos according to the keywords of the target construction video to obtain relevant job videos.

[0030] The embodiment of the present application can use target detection technology and target tracking technology to analyze relevant operation videos of construction operations to obtain the equipment category, equipment identification and detection frame of the target construction machinery.

[0031] The target detection technology and target tracking technology are designed to identify different targets from the frame images of the relevant operation video and continuously track them to determine the position, category and identification number of the target in the frame image. The frame image can represent any frame image of the relevant operation video.

[0032] Specifically, in step 101 of the embodiment of the present application, the target of the target detection technology and the target tracking technology can be specifically: target construction machinery. By using the trained target detection model and target tracking model (such as a convolutional neural network model based on deep learning, etc.), the frame image of the relevant operation video is processed. The target detection model and the target tracking model will scan and continuously locate the area where multiple construction machinery are located in the frame image based on the learned characteristic patterns of various construction machinery. This area is presented in the form of a detection frame, and at the same time, it can determine the equipment category to which the construction machinery in each detection frame belongs, such as excavation equipment or self-unloading equipment, etc., as well as the equipment identification corresponding to the construction machinery in each detection frame. The equipment identification can be an identification serial number, such as 1, 2, 3, etc. The detection frame can be the corresponding circumscribed rectangular frame of the construction machinery in the frame image.

[0033] In one example, the target detection model specifically includes: a feature extraction unit, a feature fusion unit, and a detection unit connected in series. The input of the target detection model is a frame image of the relevant operation video, and after calculation, the detection frame and category information of the target construction machine on each frame image are output. The output result of the target detection model can be {(machine class,xmin, xmax, ymin, ymax)}.

[0034] Among them, machine class represents the category information of the target construction machinery, such as "dump equipment" or "excavation equipment", etc., which is used to distinguish different categories of construction machinery.

[0035] "xmin" and "xmax" are the leftmost and rightmost coordinate values ​​of the detection frame of the target construction machinery in the horizontal direction (x-axis) of the frame image; "ymin" and "ymax" are the topmost and bottommost coordinate values ​​of the detection frame of the target construction machinery in the vertical direction (y-axis) of the frame image. These four coordinate values ​​together determine a rectangular detection frame.

[0036] The feature extraction unit is used to extract image features from the frame image.

[0037] The feature fusion unit is used to effectively fuse image features of different levels and types. The obtained fused image features can make the feature information more comprehensive and accurate, so as to enhance the ability to express the characteristics of construction machinery.

[0038] The detection unit is used to determine the category and position of the construction machinery in the frame image according to the fused image features. Target tracking technology is a technology that uses a series of algorithms and models to determine the position of a specific target (the construction machinery in the embodiment of the present invention) in different frame images in a frame image sequence, and continuously estimates and updates the target state (including position, speed, direction of movement, etc.). Its core is to achieve accurate recognition and continuous tracking of the target in a complex image environment and constantly changing target state. The target tracking result output by the target tracking model can be in the format of: {(machine class, machineID, xmin, xmax, ymin, ymax)}, where machineID represents the equipment identifier corresponding to the construction machinery.

[0039] In step 102, the device identification of the target construction machinery can be used to locate the frame number of the target construction machinery in the frame image of the relevant operation video, and the frame number can represent the time dimension information of the target construction machinery in the relevant operation video. The detection frame of the target construction machinery can be used to determine the spatial range of the target construction machinery in the frame image, and the detection frame can represent the spatial dimension information of the target construction machinery in the frame image. In this way, by combining the time dimension information and the space dimension information, the construction operation video of each target construction machinery can be cut out.

[0040] In a specific implementation, according to the equipment identification and detection frame of the target construction machinery, the relevant operation video is cropped from the spatial dimension and the temporal dimension to obtain the construction operation video of each target construction machinery, which specifically includes: Step 121, determining a start frame image and an end frame image of each target construction machine in a related operation video; Step 122, determining the maximum bounding box of each target construction machine according to the detection box of each target construction machine in the starting frame image, the intermediate frame image and the ending frame image; Step 123: crop the related operation video from the time dimension according to the starting frame image and the ending frame image, and crop the related operation video from the space dimension according to the maximum bounding box to obtain the construction operation video of each target construction machine.

[0041] In step 121, for each target construction machine with different machineID, the frame numbers corresponding to the start frame image of the target construction machine that first appears in the relevant operation video and the end frame image that finally disappears are analyzed according to the target tracking result. For example, the frame number of the start frame image is m, and the frame number of the end frame image is n.

[0042] In step 122, for each target construction machine with a different machineID, its maximum bounding box in the image screen of the relevant work video is calculated.

[0043] Assume that the image frame number set of the target construction machinery in the long-term video is {i} (i>=m, i<=n, and i is a positive integer), and the coordinate set of the corresponding detection box is {(xmin i , xmax i , ymin i , ymax i )}, the coordinates of the maximum bounding box are calculated as follows: Xmin = min{ i} Xmax = max{ i} Ymin = min{ i} Ymax= max{ ymax i} Among them, Xmin and Xmax represent the leftmost and rightmost coordinate values ​​of the maximum bounding box in the horizontal direction of the frame image respectively; Ymin and Ymax represent the topmost and bottommost coordinate values ​​of the maximum bounding box in the vertical direction of the frame image respectively. These four coordinate values ​​together determine the maximum bounding box of a rectangle.

[0044] In step 123, the relevant operation video is cropped from the time dimension and the space dimension according to the m-th frame image to the n-th frame image and the coordinates of the maximum bounding box to obtain the construction operation video of each target construction machine.

[0045] There are differences between the related operation videos and the construction operation videos in terms of the number of frame images and image size. In terms of the number of frame images, the construction operation video is cropped from the mth frame to the nth frame, and the number of frame images is n - m + 1, while the number of frame images of the related operation video is usually larger than that of the construction operation video. In terms of image size, the construction operation video is cropped according to the maximum bounding box coordinates, usually only framing the target construction machinery, so the image size of the construction operation video is usually smaller than that of the related operation video.

[0046] In step 103, the action segmentation model is used to segment the construction operation video of each target construction machine to obtain multiple construction operation segments.

[0047] Among them, the action segmentation model is a model used to decompose actions in a continuous video stream into different segments. It is based on deep learning or other machine learning technologies, and after being trained with a large amount of video data with annotated actions, it can identify the changes and boundaries of actions in the video, thereby segmenting the complex action sequences in the video into relatively independent action categories.

[0048] The action segmentation model will analyze the characteristics, speed, direction and other information of the construction machinery actions in the construction operation video based on the knowledge learned during its training, identify the starting and ending points of the action, and divide the entire construction operation video into multiple shorter construction operation clips. Each construction operation clip represents a relatively complete construction operation action of the target construction machinery (referred to as action category).

[0049] After action segmentation, any two adjacent construction operation clips arranged in chronological order represent different construction machine actions. For example, the previous construction operation clip may be the digging action of the excavator, and the next construction operation clip is the action of the excavator lifting the bucket and moving it above the transport vehicle. Such segmentation results help to clearly understand and analyze each specific operation step and its sequence in the entire operation process of the construction machinery.

[0050] Among them, step 103 uses the action segmentation model to segment the construction operation video of each target construction machine, which specifically includes: Step 131: extracting three-dimensional temporal features of the construction operation video of each target construction machine; Step 132, sequentially performing one-dimensional convolution processing, self-attention processing, and feedforward processing on the three-dimensional time series features to obtain fusion region features corresponding to each target construction machine; Step 133: using a denoising network model, determining the fusion region feature and the noise distribution at the s-th time step, and determining the action category at the s-th time step; Step 134 , according to the action category corresponding to each time step, the construction operation video of each target construction machine is divided into a plurality of construction operation segments arranged in chronological order.

[0051] In step 131, I3D (Inflated 3D ConvNets) can be used to extract the 3D temporal features of the construction operation video of each target construction machine. ‌3D temporal features‌ refer to data features that change over time in a 3D space. These features are usually used to describe and analyze the dynamic change process of an object in a 3D space.

[0052] In one implementation, the three-dimensional temporal features specifically include: RGB (Red Green Blue) features and optical flow features. Among them, RGB features can capture the color and texture information of the target construction machinery, and optical flow features can effectively characterize the motion information of the target construction machinery. The combination of the two can comprehensively and accurately describe the appearance and motion state of the target construction machinery in the video.

[0053] In practical applications, the process of extracting the 3D temporal features of the construction operation video of each target construction machine specifically includes: Step 1311: using a sliding window with a preset window size and a preset step length, obtaining multiple video clips from the construction operation video of each target construction machine; Step 1312: extracting RGB features and optical flow features corresponding to the multiple video clips respectively; Step 1313: fuse the RGB features and optical flow features corresponding to the multiple video clips to obtain the three-dimensional temporal features corresponding to the multiple video clips.

[0054] In step 1312, these acquired video clips can be input into the I3D network, and each video clip can be processed separately by using the powerful convolution capability of the I3D network. RGB features are extracted from the frame images of the clips to capture spatial information such as color and texture; at the same time, optical flow features are extracted by analyzing the pixel changes between frames to record the object motion information.

[0055] In step 1313, the RGB features and optical flow features corresponding to each video clip may be fused. For example, feature splicing or the like may be used to integrate the RGB features representing the spatial dimension with the optical flow features representing the temporal dimension to form three-dimensional temporal features corresponding to the multiple video clips for subsequent analysis.

[0056] In step 132, the one-dimensional convolution processing can use a one-dimensional convolution kernel to slide on a specific dimension (such as the time dimension) of the three-dimensional time series feature, perform weighted summation of adjacent feature values ​​on the specific dimension, and introduce nonlinearity through an activation function to mine the local feature pattern on the specific dimension to obtain a feature representation A.

[0057] Self-attention processing can be used to calculate the degree of association between each position in the feature representation A and other positions to obtain attention scores, and then these attention scores are normalized. The normalization result is used as a weight to perform weighted summation of the features at each position to obtain the feature representation B.

[0058] The feedforward process is used to pass the feature representation B through multiple fully connected layers in sequence. Each fully connected layer is used to perform a linear transformation on the input, and then a nonlinear activation function is used to increase the nonlinear expression ability of the linear transformation result to obtain the fused regional features. The fused regional features integrate the information of the previous processing steps and can more comprehensively and effectively characterize the characteristics of the target construction machinery.

[0059] Step 133 uses a denoising network model to determine the fusion region feature and the noise distribution at the s-th time step, and determines the action category at the s-th time step, specifically including: Step 1331, extracting features of the noise distribution at the s-th time step to obtain noise distribution features at the s-th time step; Step 1332: Fusing the noise distribution feature with the time coding vector to obtain the noise time distribution feature of the s-th time step; Step 1333: extract the noise time distribution characteristics of the s-th time step to obtain the noise representation of the s-th time step; Step 1334: The noise representation of the s-th time step is concatenated with the fused region feature as the query vector and the key vector, and the noise representation of the s-th time step is used as the value vector, which is input into the cross attention layer. The cross attention layer calculates the attention weight between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weight to obtain the cross attention result. Step 1335: Fuse the cross attention result with the noise representation of the s-th time step to obtain a fused representation; Step 1336: Use the feedforward connection layer and the convolution layer to process the fused representation to obtain the action category of the s-th time step.

[0060] Reference Figure 2, shows a schematic diagram of the structure of a denoising network model of an embodiment of the present application. The denoising network model specifically includes: a one-dimensional convolution layer A201, a first fusion module 202, a one-dimensional convolution layer B203, a cross attention layer 204, a second fusion module 205, a feedforward connection layer 206 and a one-dimensional convolution layer 207.

[0061] In step 1331, the one-dimensional convolution layer A201 can be used to extract the noise distribution at the s-th time step to obtain the noise distribution characteristics at the s-th time step. The one-dimensional convolution layer A201 is intended to capture the key feature information in the noise at the s-th time step to obtain the noise distribution characteristics.

[0062] In the process of action recognition, the noise distribution of the sth time step can increase the feature diversity, improve the ability of the denoising network model to distinguish different actions, enhance the robustness of the denoising network model for action recognition, and assist in analyzing the correlation between features.

[0063] The noise distribution of the sth time step can be obtained by sensor measurement, interference from the data acquisition environment, adding simulated noise in data preprocessing, or generated by a model. For example, the type of the noise distribution of the sth time step can be any one or a combination of the following noise types: Gaussian distribution, Poisson distribution, and uniform distribution.

[0064] In step 1332, the noise distribution feature and the time coding vector may be fused by the first fusion module 202 to obtain the noise time distribution feature of the sth time step. Combining the time information with the noise distribution feature can form a more time-aware noise time distribution feature and enhance the consideration of the time factor of the feature.

[0065] In step 1333, the one-dimensional convolution layer B203 can be used to extract the noise time distribution characteristics of the s-th time step to obtain the noise representation of the s-th time step. Using the one-dimensional convolution again can deepen the information mining of the noise time distribution characteristics, strengthen important features, remove redundant information, and extract more discriminative noise representations.

[0066] In step 1334, the noise representation of the sth time step is concatenated with the fused region features as the query vector and the key vector, and the noise representation of the sth time step is used as the value vector and input into the cross-attention layer. The cross-attention layer calculates the attention weight between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weight to obtain the cross-attention result.

[0067] Among them, the noise representation of the sth time step is concatenated with the fusion region feature, and the resulting vector has both noise characteristics and overall mechanical feature information, and is used as a query vector and key vector.

[0068] The attention weight calculated by the cross-attention layer represents the degree of correlation between the query vector (which is composed of the concatenation of the noise representation and the fusion region features) and the key vector (which is also composed of the concatenation of the noise representation and the fusion region features). The higher the attention weight, the closer the corresponding features are.

[0069] The value vector (noise representation) is weighted and summed according to these attention weights, which essentially re-"assigns" each part of the value vector according to the degree of correlation between features. The closely related noise feature parts have a higher weight and account for a larger proportion in the weighted summation result, thus being highlighted.

[0070] The final cross-attention result achieves a deep integration of noise and fusion region feature information. This integrated information not only includes the key parts of the noise features that are closely related to the overall features, but also incorporates the comprehensive information about the target construction machinery carried by the fusion region features, providing more targeted and rich data for subsequent action recognition.

[0071] In step 1335, the second fusion module 205 may be used to fuse the cross attention result with the noise representation of the s-th time step to obtain a fused representation. The second fusion module 205 fuses the cross attention result with the noise representation of the s-th time step to further integrate information and form a more comprehensive fused representation to prepare for the final action recognition.

[0072] In step 1336, the fused representation can be processed using a feedforward connection layer and a convolution layer to obtain the action category of the sth time step. The feedforward connection layer and the convolution layer convert the fused representation into an output related to the action category through linear transformation and nonlinear activation, respectively, to complete the recognition of the action. The action recognition result specifically includes: the probability that the sth time step belongs to a preset action category. Examples of preset action categories may include: excavation action or rotation action of an excavator, or loading action or unloading action of a dump truck, etc.

[0073] In the embodiment of the present application, the time step is an abstract concept when processing time series data. It can be closely related to the frame number. A time step can correspond to one or more frame numbers. For example, when capturing video at a fixed frame rate (such as 30 frames per second), if the length of the time step is set to 1 second, then a time step corresponds to 30 consecutive frame numbers. Specifically, the first time step may include frames 1 to 30, the second time step includes frames 31 to 60, and so on. The time step can process and analyze the information in the video from a more macro perspective, rather than directly operating each individual frame. It groups continuous frames for easy action recognition. In different tasks, the length of the time step can be flexibly adjusted according to specific needs. For example, in some construction machinery action recognition tasks with slower movements, the time step may be set to a longer duration (including more frame numbers) to capture complete action information; for tasks with faster movements, a shorter time step may be set to more accurately analyze subtle changes in the action.

[0074] A time step usually corresponds to a video clip. This video clip is captured from the construction video according to the sliding window. For example, for a long construction video, according to the division of time steps, each time step corresponds to a video clip containing a certain number of frames.

[0075] In step 134, by traversing the time steps, when it is detected that the action category corresponding to the adjacent time steps has changed, this moment is used as the segmentation point, and then starting from the starting frame of the video, based on these segmentation points, video segments of different action categories are captured in turn, thereby segmenting the construction operation video of each target construction machinery into multiple construction operation segments arranged in chronological order.

[0076] In step 104, multiple construction operation segments arranged in chronological order can be traversed, and the start time and end time of each construction operation segment can be recorded. The duration of the action category corresponding to the construction operation segment can be obtained by subtracting the start time from the end time. By integrating the duration data of all construction operation segments, the duration data of each action category can be determined. The duration data of the action category specifically includes: the duration value of the action category.

[0077] In step 105, the duration values ​​of the same action category appearing at different times may be collected to form a data set, which is the factual duration sample corresponding to the action category, covering the duration information of the action category in different time periods of the construction operation video.

[0078] In step 106, based on the factual duration samples corresponding to the action categories, random duration samples corresponding to the action categories are further generated. This process simulates various unpredictable actual situations in the construction process, so that the generated random duration samples cover various uncertainties and changing factors in the construction process.

[0079] In a specific implementation, the process of step 106 generating a random duration sample corresponding to an action category according to the factual duration sample corresponding to the action category specifically includes: Step 161, determining a target distribution function corresponding to the action category according to the factual duration samples corresponding to the action category; Step 162: Use a rejection sampling method to generate a random duration sample that conforms to the target distribution function.

[0080] In step 161, determining the target distribution function is to establish a mathematical model basis for generating random duration samples. By analyzing the factual duration samples corresponding to the action categories, the distribution law presented by these samples is found, thereby obtaining a target distribution function that can describe the distribution characteristics of the action duration. This helps to generate random data that conforms to the actual distribution in the subsequent process.

[0081] In one example, the Dirichlet process mixture model (DPMM) can be used to determine the non-parametric distribution of the factual duration sample. A non-parametric distribution is a probability distribution that does not rely on a specific pre-assumed parameterized probability distribution form. In other words, it does not make specific parameterized assumptions about the distribution that the data follows, such as unlike the normal distribution, which has clear parameters such as mean and variance to fully determine the distribution shape, but allows the data itself to "reveal" its distribution characteristics.

[0082] As a typical representative of non-parametric models, the Dirichlet process mixture model can reduce the dependence on artificially specified distribution types by defining it in an infinite-dimensional parameter space, thereby improving the modeling flexibility of sample distribution.

[0083] Assumptions is the data set corresponding to the factual duration samples, N is the number of factual duration samples, and the factual duration samples obey an unknown probability distribution . Then estimate the unknown probability distribution based on the data set The Dirichlet process mixture model can be expressed as formula (1): (1) In formula (1): function For a given parameter Sample data The distribution of Satisfies iid (independent and identically distributed) and obeys the distribution , It obeys the Dirichlet process.

[0084] Among them, the definitions of Dirichlet distribution and Dirichlet process are as follows: The Dirichlet distribution is a multivariate generalization of the Beta distribution, which is defined as: if the random vector The joint distribution density of is: (2) Then it is called It obeys Dirichlet distribution, denoted as .in is the Gamma function. When , the Dirichlet distribution degenerates into the Beta distribution.

[0085] The Dirichlet process is defined as: Assume It is a measure space Probability measures of random variables on , with concentrated parameters If the measure space Probability measures of random variables on Satisfy the measure space Any finite partition of , both exist: (3) Then it is called obey On the basis distribution and concentrated parameters The Dirichlet process composed of . Concentrated parameters Reflects With the base distribution The greater the value of similarity, With the base distribution The more similar.

[0086] In the process of establishing nonparametric distribution based on Dirichlet process mixture model, all unknown parameters of the model can be expressed as ,in, Indicates The parameters of the probability density distribution, For the The weight of the probability density distribution, and , According to Bayesian theory, The posterior distribution estimate of can be obtained through its prior distribution and likelihood function calculation, which is the target distribution function obeyed by the factual duration samples of each action category of the target construction machinery.

[0087] (4) In step 162, a rejection sampling method is used to generate random duration samples that conform to the target distribution function, with the goal of obtaining truly usable random data from the theoretical target distribution function, which can reflect the uncertainty and variation factors in the construction process.

[0088] In a specific implementation, the process of using the rejection sampling method to generate random duration samples that meet the target distribution function specifically includes: Step 1621, determining a reference distribution function and a constant; under the condition of the same independent variable, the function value of the target distribution function is not greater than the product of the function value of the reference distribution function and the constant; Step 1622, extracting a random independent variable from the reference distribution function; Step 1623: extract a random number from a uniform distribution function; the upper limit value of the uniform distribution function is: the product of the function value of the reference distribution function and a constant under the condition of the random independent variable; Step 1624: If the random number is smaller than the function value of the target distribution function corresponding to the random independent variable, the random independent variable is used as a random duration sample that conforms to the target distribution function.

[0089] In one example, an envelope function may be used to iterate the "rejection / acceptance" steps to achieve sampling of random duration samples.

[0090] For the target distribution function to be sampled , first select a reference distribution that is easy to sample , such as the normal distribution, and assumes that there is a constant So that for any Both . The specific rejection sampling calculation process is: based on the reference distribution Generate random variables , and then from the uniform distribution Generate random variables in If it exists , then reject the random variable , otherwise accept random variables , that is, the random independent variable is taken as a random duration sample that conforms to the target distribution function.

[0091] In step 107, the construction process parameters of the target construction machinery in the construction simulation model of the construction operation are assigned according to the duration value of the random duration sample, so that the construction process parameters in the construction simulation model are closer to the uncertainty and changes in the actual construction, providing a reasonable data basis for the subsequent simulation of the construction simulation model and the prediction of the construction progress.

[0092] For example, any one of the duration values ​​corresponding to these random duration samples is selected to determine the unit operation duration corresponding to the target construction machinery, and the determined unit operation duration is assigned to the corresponding construction process parameters in the construction simulation model.

[0093] In the construction scenario, unit operation time refers to the time it takes for the target construction machinery to complete a standard operation action.

[0094] It is an important part of the construction process parameters and can be used to measure construction efficiency. Its value will vary depending on factors such as the type of construction machinery, action category, and construction environment. By using random duration sample assignment, it can more truly reflect the uncertainty of this time in actual construction, provide accurate parameters for the construction simulation model, and assist in construction progress prediction.

[0095] Taking excavation equipment as an example, unit operation time refers to the time it takes for the excavation equipment to complete a standard operation.

[0096] For example, in earth excavation operations, the standard operating action of the excavation equipment is "shovel down - dig - lift - rotate - unload - rotate back to position - shovel down to prepare for the next excavation". The time spent to complete such a standard operating action cycle is a unit operating time.

[0097] In step 108, the construction simulation model is simulated according to the assigned construction process parameters of the target construction machinery to obtain a construction progress prediction result, so as to help construction managers understand the construction progress in advance and arrange resources and construction period reasonably.

[0098] The construction simulation model is a mathematical or logical abstraction of the actual construction process, in which the construction process parameters determine key factors such as the time spent on the construction process in the construction simulation model. By inputting the assigned construction process parameters, the construction simulation model performs calculations and simulations according to the set rules and logic, thereby predicting the construction progress.

[0099] During the simulation process, the construction simulation model records and calculates the start time, end time and other information of each construction process, and finally outputs the construction progress prediction results.

[0100] In one example, the construction progress prediction result specifically includes the following information: Time node information: the start time and end time of each construction process, as well as the estimated completion time of the entire construction work, to clarify the construction time flow.

[0101] Progress deviation: Compare the planned progress to show whether the predicted progress is ahead or behind, and the specific duration or proportion of the deviation, so as to facilitate timely adjustment. Among them, the planned progress is the percentage of construction processes that should be completed at each time point before the construction work starts, based on goals, contracts, resources, etc., and is the reference standard for construction progress. The predicted progress is simulated using construction process parameters to dynamically calculate the actual completion of the construction process at each time point. The completion percentage of the project at each time point obtained reflects the estimated progress after considering uncertain factors.

[0102] Resource demand data: covers the quantity and time of demand for manpower, materials, equipment and other resources at each construction stage, facilitating resource preparation and deployment.

[0103] Risk warning tips: Point out the risk factors that may affect the progress during construction, such as technical difficulties, bad weather, policy changes, etc.

[0104] In summary, the construction progress prediction method based on long-term video analysis in the embodiment of the present application starts from the construction operation video and obtains the construction progress prediction result through multi-step processing. First, the relevant operation videos of the construction operation (including historical operation videos of the construction operation and / or historical operation videos of the same type of operation, and the duration is greater than the duration threshold) are analyzed to obtain the equipment category, equipment identification and detection frame of the target construction machinery; then the construction operation video of each target construction machinery is cut from the time and space dimensions; then the three-dimensional time series features of the video are extracted, and the action category is determined by action recognition, and the duration data is determined according to the action category, and then the factual and random duration samples are obtained; the duration value corresponding to the random duration sample is used to assign the construction process parameters (such as unit operation duration) of the target construction machinery in the construction simulation model; finally, the construction simulation model is simulated to obtain the construction progress prediction result.

[0105] The embodiment of the present application is based on the construction operation related video, and determines the parameter value of the construction process parameter. Since the construction operation related video of the embodiment of the present application records various actions of the target construction machinery in the construction process, based on the analysis of the construction operation related video, it is possible to obtain real data such as action category and factual duration sample from the construction operation related video, and it is possible to generate a random duration sample by processing these real data. Since the random duration sample includes various uncertainties and changing factors in the construction process, it is used for the assignment of the construction process parameter, and the parameter value of the construction process parameter can be made closer to the real state of the construction site. In this way, the construction simulation model including the construction process parameter can simulate the construction progress more accurately based on the parameters close to the real state. Therefore, the embodiment of the present application can effectively improve the accuracy of the construction progress prediction.

[0106] Method Example 2 This embodiment illustrates the construction process of the construction simulation model.

[0107] refer to Figure 3 , shows a schematic flow chart of the steps of a method for automatically building a construction simulation model based on video images according to an embodiment of the present application, the method specifically comprising the following steps: Step 301, for the construction operation, establish a construction simulation model; the above-mentioned construction simulation model specifically includes: a construction unit intelligent agent object, a construction resource scheduling intelligent agent object, a construction resource intelligent agent object, and a construction process simulation model corresponding to a plurality of construction unit intelligent agent objects; the above-mentioned construction process simulation model specifically includes: a construction process simulation model; the above-mentioned construction process simulation model includes: a queue element and a delay element; the above-mentioned queue element is used to simulate the queuing and waiting process of the construction resource intelligent agent object in the construction process; the above-mentioned delay element is used to simulate the duration of the construction process; The simulation process of the above-mentioned construction simulation model specifically includes: the construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task issuance message to at least one selected target construction resource intelligent agent object according to the above-mentioned resource request message; after receiving the task issuance message, at least one target construction resource intelligent agent object moves to the target construction work surface where the construction unit intelligent agent object is located; the construction resource intelligent agent objects that meet the set conditions at the target construction work surface are placed in the queue represented by the queue element for queuing and waiting; the construction resource intelligent agent objects that do not need to wait in line or are taken out from the queue element at the target construction work surface are used to perform construction simulation of the construction process; after completing the simulation of the construction process of the construction unit intelligent agent object, the corresponding delay element ends; Step 302: Determine the construction resource distribution information corresponding to the construction work surface of the construction work site by using target detection technology, target tracking technology and clustering technology according to the video image of the construction work site; Step 303: Instantiate queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information.

[0108] Figure 3 The method shown can be used to build a construction simulation model based on video images for construction operations.

[0109] Taking slope excavation as an example, the construction objects of slope excavation can be divided into three levels: excavation zones, excavation steps, and excavation layers according to the geometric structure subordination of the excavation parts.

[0110] Among them, an excavation zone may include: several excavation steps. Specifically, the entire construction object can be divided into different excavation zones such as the front block excavation zone and the protective layer top surface excavation zone according to the vertical partition line. Then, each excavation zone is further divided into multiple excavation steps according to the designed step height. For example, if the elevation of 50m-55m is designated as the first excavation step, and 55m-60m is designated as the second excavation step, the construction conditions of different excavation steps are different due to their different heights.

[0111] An excavation step specifically includes: a number of excavation blocks. Specifically, each excavation step can be divided into multiple excavation blocks along the horizontal direction, and each excavation block is a basic construction unit, and subsequent specific construction processes are carried out in these excavation blocks.

[0112] In the field of construction simulation technology, an intelligent agent refers to an entity that can perceive the environment, make decisions and take actions. It has autonomy and can decide what actions to perform based on its own goals and perception of the environment to achieve the predetermined goals.

[0113] The agent object is the instantiation of the agent in a specific application scenario (such as a construction simulation model). It is an abstract agent concept that is given specific attributes, states, and behavior rules, so that it can represent an entity with autonomous behavior capabilities in a specific construction simulation model. Behavioral rules refer to a series of rules that specify how an agent should act in a specific environment (such as a construction scene). It clarifies the agent's response to internal state changes and external stimuli (such as instructions, events, etc.), and determines the agent's behavior logic and execution order.

[0114] For example, in the construction simulation model, the construction unit agent object, the construction resource scheduling agent object and the construction resource agent object all have their own clear attributes (such as the geometric dimensions of the construction unit, the type and quantity of construction resources, etc.), status (such as idle, busy, etc.) and behavior rules (such as resource scheduling algorithms, the state transition methods of construction resources).

[0115] The construction unit intelligent agent object, construction resource scheduling intelligent agent object and construction resource intelligent agent object are explained below respectively.

[0116] The construction unit agent object is used to describe a construction unit such as an excavation block. A collection of multiple construction unit agent objects can be used to describe the construction object corresponding to the entire construction operation. The construction unit agent object usually contains attribute parameters related to the construction unit. These attribute parameters can be information such as the size, location, geological conditions, etc. of the construction unit. These attribute parameters are used to characterize the characteristics of each construction unit.

[0117] The embodiment of the present application can instantiate the construction unit to obtain the intelligent object of the construction unit. For example, the instantiation process of the construction unit specifically includes: first, obtaining the partition number of the construction unit from the engineering data; then, for the partition number of the construction unit, obtaining the geometric shape, size data, position data, first behavior rule and other attribute information of the construction unit from the engineering data; binding the partition number of the construction unit with the attribute information to obtain the intelligent object of the construction unit.

[0118] Taking slope excavation as an example, the construction unit intelligent object specifically includes: excavation partition intelligent object, excavation step intelligent object and excavation block intelligent object.

[0119] Among them, the attribute information of the excavation partition intelligent agent object includes the partition number.

[0120] The attribute information of the excavation step intelligent object includes: the partition number of the excavation partition to which it belongs, the step number, the step height, and the start and end elevations.

[0121] The attribute information of the excavation block intelligent object includes: the gradient number, the block number, and the engineering quantity information. The engineering quantity information may include the excavation block area, the length along the baseline, the width along the baseline, the excavation block height, the number of deep / shallow holes drilled, etc. The slope excavation operation can be described as a set consisting of all excavation block intelligent object objects. The construction simulation of the slope excavation operation can be understood as: simulating and calculating the set of excavation block intelligent object objects. Optionally, the set of excavation partition intelligent object objects and the set of excavation step intelligent object objects are only used for the imposition of construction constraints, engineering quantity information statistics, etc., and are not used as specific construction simulation objects.

[0122] The construction resource agent object represents various construction resources, such as manpower (construction personnel) and material resources (construction equipment, materials, etc.). In the construction scene, each construction resource can be regarded as a construction resource agent object. The construction resource agent object can receive task instructions from the construction resource scheduling agent object and play a role in the construction process corresponding to the construction unit agent object. For example, the excavation equipment performs excavation operations in the specified excavation blocks according to the instructions, and the transport vehicle transports the excavated earth away. The behavior and status of the construction resource agent object will be affected by the construction process, and it will also feedback information to the construction resource scheduling agent object to better allocate resources.

[0123] The embodiments of the present application can instantiate construction resources to obtain construction resource intelligent body objects. Specifically, the types of construction resources and construction resource scheduling requirements that may be involved in the construction work are analyzed, and the construction resource intelligent body is instantiated by defining the states that the construction resource intelligent body may be in, as well as the state transition mode, trigger conditions, and second behavior rules between different states. The information of the construction resource scheduling intelligent body object specifically includes: resource category, resource number, performance parameters, state, state transition mode, trigger conditions, etc.

[0124] The states of the construction resource intelligent object may include but are not limited to: initial state, idle state, working state, maintenance state, etc. The state transition modes may include: time trigger, condition trigger and message trigger. Among them, time trigger means switching from one set state to another set state after a specific time interval is met; condition trigger means switching from one set state to another set state after a specific condition is met; message trigger means switching from one set state to another set state when the construction resource intelligent object receives a specific message.

[0125] Taking slope excavation as an example, the resource categories of construction resources that may be involved in the excavation construction process include: drilling equipment, bulldozer equipment, excavation and loading equipment, and transportation equipment. The drilling equipment intelligent object, bulldozer equipment intelligent object, excavation and loading equipment intelligent object, and transportation equipment intelligent object can be defined accordingly.

[0126] Taking the transport equipment agent object as an example, its state and state transition can be defined as follows: Figure 3 As shown in the figure. At the beginning of the simulation, the state of the transport equipment intelligent object changes from the initial state to the idle state. When receiving the task issuance message, the state of the transport equipment intelligent object changes to moving to the excavation surface. After arriving at the excavation surface, the state of the transport equipment intelligent object changes to waiting for loading. After completing the construction processes such as loading, re-hauling, and unloading, the state of the transport equipment intelligent object changes to idle. When receiving the start of daily maintenance message, the state of the transport equipment intelligent object changes to daily maintenance.

[0127] The construction resource scheduling agent object is used to schedule the construction resource agent objects. It can sense the resource demand of the construction unit agent object during the construction process. For example, in the slope excavation operation, when a certain excavation block (construction unit agent object) needs construction resources such as excavation equipment and transportation vehicles, it can send a resource request message to the construction resource scheduling agent object. The construction resource scheduling agent object will make scheduling decisions based on factors such as the availability of the construction resource agent object and the priority of the construction unit, and allocate the appropriate construction resource agent object to the required construction unit agent object.

[0128] The construction resource scheduling intelligent agent object of the embodiment of the present application may include the following information: scheduling scope (responsible construction work surface), scheduling strategy (such as allocating resources according to the priority of construction units or resource utilization efficiency, etc.), status, state transition method, trigger conditions, third behavior rules, etc.

[0129] The states of the construction resource scheduling intelligent object may include but are not limited to: initial state, construction task existence judgment state, waiting for construction task state, service construction simulation object selection state, idle construction resource judgment state, waiting for construction resource idle state, construction resource scheduling state, task issuance, etc.

[0130] After the construction simulation model starts simulating, the state of the construction resource scheduling agent changes from the initial state to the construction task existence judgment state. If a resource request message is received from any construction unit agent object, it is considered that there is a construction task, and the state changes to the service construction simulation object selection state; if no resource request message is received, the state changes to the waiting state for the construction task, and the judgment is cyclic until a resource request message is received; When the state of the construction resource scheduling agent changes to the service construction simulation object selection state, according to the first preset rule, a target construction unit agent object is selected from the multiple construction unit agent objects that send the request, and the state changes to the idle construction resource judgment state; if there is an idle construction resource agent object, the state changes to the construction resource scheduling state; if not, the state changes to the waiting construction resource idle state, and the judgment is cyclically judged until the state of a construction resource agent changes to the idle state; When the state of the construction resource scheduling intelligent body changes to the construction resource scheduling state, according to the second preset rule, one or more target construction resource intelligent body objects are selected from multiple idle construction resource intelligent body objects, the state changes to the task issuance state, and a task issuance message is sent to the target construction resource intelligent body object; the task issuance message is used to prompt the target construction resource intelligent body object to go to the target construction work surface corresponding to the target construction unit intelligent body object to perform the corresponding construction process.

[0131] The first preset rule is used to optimize the resource allocation order and reasonably allocate the construction resource intelligent agent objects to specific construction unit intelligent agent objects in multiple resource request messages to speed up the construction progress and balance the supply and demand of resources.

[0132] The first preset rule specifically includes: based on the priority of the construction unit, giving priority to the construction units with key routes or construction processes to obtain construction resources; or, based on the urgency of resource demand, giving priority to the construction units with tight time and large resource shortages; or, based on resource allocation efficiency, giving priority to allocating construction resources to the construction units that are closer to the construction resource intelligent body object. It can be understood that the embodiments of the present application do not limit the specific first preset rule.

[0133] The second preset rule aims to accurately and efficiently select construction resource intelligent object to meet the resource requirements of the construction unit intelligent object.

[0134] The second preset rule specifically covers: Based on resource adaptability: Prioritize the selection of construction resource agent objects that are highly compatible with the construction process of the target construction unit agent object. For example, if the target construction unit is to perform fine welding operations, priority will be given to welding equipment agents that have high welding accuracy and can meet the process requirements of the operation.

[0135] Based on performance parameters: From the idle construction resource intelligent objects, those with better performance parameters are selected first.

[0136] Based on resource scheduling cost: Comprehensively consider the resource scheduling cost and give priority to construction resource intelligent objects with short scheduling paths and low energy consumption. For example, if there are multiple idle construction resource intelligent objects, give priority to construction resource intelligent objects that are close to the target construction work surface and have good traffic conditions on the route to the work surface to reduce transportation costs and time costs.

[0137] The construction process simulation model corresponding to the construction unit intelligent agent object is explained below.

[0138] In the specific implementation, the construction procedures involved in the construction process and the dependencies between the construction procedures can be analyzed for the construction unit intelligent object, and the construction process of the construction unit can be simulated and modeled based on the analysis results to obtain a construction process simulation model corresponding to the construction unit intelligent object.

[0139] The dependency of construction processes refers to the mutual connection and influence between different construction processes. The above dependencies specifically include: sequential dependency, parallel dependency, and end dependency. Among them, sequential dependency means that the next construction process can only start after the previous construction process is completed. Parallel dependency means that two construction processes can start at the same time, but one construction process needs to wait for the construction status of the other construction process to meet the set conditions. End dependency means that when one construction process is completed, the other construction process must also be completed.

[0140] The construction process simulation model specifically includes: a construction process simulation model corresponding to the construction process; the construction process simulation model includes: a queue element and a delay element; the queue element is used to simulate the queuing and waiting process of the construction resource intelligent object in the construction process; the delay element is used to simulate the duration of the construction process. When the queue element and the delay element are used at the same time, the queue element and the delay element can be connected in series.

[0141] The embodiment of the present application can put the construction resource intelligent body objects that meet the set conditions at the target construction work surface into the queue represented by the queue element to queue and wait.

[0142] The above-mentioned setting conditions specifically include: Setting condition 1: for the first construction resource intelligent agent object at the target construction work surface, the first construction resource intelligent agent object corresponds to the first resource category, and the number of the first type of construction resource intelligent agent objects at the target construction work surface exceeds the maximum number of simultaneous working; and / or Setting condition 2: For the first construction resource intelligent body object at the target construction work surface, the first construction resource intelligent body object corresponds to the first resource category, and the construction resource intelligent body object of the first resource category cooperates with the construction resource intelligent body object of the second resource category to jointly simulate the construction process. The target construction work surface may refer to the construction work surface where the construction unit intelligent body object that sends the resource request message is located.

[0143] For setting condition 1, when a construction resource intelligent body object arrives at the target construction work surface, if there are construction resource intelligent body objects of the same resource category, and the number of construction resource intelligent body objects of this resource category exceeds the maximum number that can work simultaneously, the arriving construction resource intelligent body object will be placed in the queue represented by the queue element to wait until a construction resource intelligent body object of the same resource category leaves the current construction work surface.

[0144] For setting condition 2, when a construction resource intelligent body object arrives at the target construction work surface, if it needs to cooperate with construction resource intelligent body objects of other resource categories to jointly carry out the current construction process, the arrived construction resource intelligent body object will be placed in the queue represented by the queue element to wait until the construction resource intelligent body objects of other resource categories that need to be coordinated arrive at the target construction work surface.

[0145] On the basis of the simulation modeling of a single construction process, the dependencies of the construction processes involved in the construction process can be analyzed, and the construction process simulation models can be connected in series or in parallel to complete the simulation modeling of the construction process.

[0146] Referring to FIG. 4 (a), a schematic diagram of a construction process modeling of an embodiment of the present application is shown, wherein for the sequential dependency between construction processes, the construction process simulation models between different construction processes can be connected to complete the modeling of the construction process. For example, queue element 1 and delay element 1 included in construction process 1, and queue element 2 and delay element 2 included in construction process 2 are connected in series.

[0147] Referring to FIG. 4 (b), a schematic diagram of the modeling of the construction process of an embodiment of the present application is shown, wherein for parallel dependency and end dependency, the construction processes can be independently modeled, and the modeling of the dependency relationship can be achieved by using functions to impose judgment conditions. For example, queue element 3 and delay element 3 included in construction process 3 are independently modeled, and queue element 4 and delay element 4 included in construction process 4 are independently modeled. Construction process 3 and construction process 4 are in a parallel relationship.

[0148] The model parameters of the construction process simulation model may include: the number of queue elements, the number of delayed elements, the maximum allowed capacity of the queue, and the queue queuing rules, etc. In the embodiment of the present application, the following parameters may be set for the construction process simulation model: 1) The maximum allowed capacity of the queue, which indicates the maximum number of construction resource agent objects that the queue element can accommodate. If the queue is full, the newly arrived construction resource agent object will become idle and return to wait or be rescheduled to a new construction work surface.

[0149] 2) Queue rules: Queue rules for construction resource agent objects of the same resource category, such as first-in-first-out, first-in-last-out, etc.

[0150] Taking the slope excavation operation constructed by drilling and blasting as an example, for the excavation block, that is, the basic construction unit, its construction process involves the following construction procedures: 1) Measurement and layout (construction process A) 2) Drilling (Construction Process B) 3) Charging line (construction process C) 4) Blasting (Construction Process D) 5) Slag loading (construction process E) 6) Slag transportation (construction process F) The dependencies between the construction processes are as follows: 1) Sequence dependency: A→B (drilling can only be carried out after measurement and layout are completed) 2) Sequence dependency: B→C (charge loading and connection can only be carried out after drilling is completed) 3) Sequence dependency: C→D (blasting can only be carried out after the charging line is completed) 4) Sequence dependency: D→E (slag loading can only be carried out after blasting is completed) 5) Sequence dependency: E→F (slag transportation can only be carried out after slag loading is completed) Then a construction process simulation model of excavation blocks is as follows Figure 5 As shown, it specifically includes: construction process simulation models of 6 construction processes. The construction process simulation models of the 6 construction processes specifically include: a construction process simulation model of construction process A, a construction process simulation model of construction process B, a construction process simulation model of construction process C, a construction process simulation model of construction process D, a construction process simulation model of construction process E, and a construction process simulation model of construction process F. The construction process simulation model of construction process A specifically includes: queue element A and delay element A. The construction process simulation model of construction process B specifically includes: queue element B and delay element B. The construction process simulation model of construction process C specifically includes: queue element C and delay element C. The construction process simulation model of construction process D specifically includes: queue element D and delay element D. The construction process simulation model of construction process E specifically includes: queue element E and delay element E. The construction process simulation model of construction process F specifically includes: queue element E and delay element F.

[0151] In addition to realizing the modeling of the construction process simulation model, the embodiment of the present application also provides the following simulation process of the construction simulation model: the construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task issuance message to at least one selected target construction resource intelligent agent object according to the resource request message; after receiving the task issuance message, at least one target construction resource intelligent agent object moves to the target construction work surface where the construction unit intelligent agent object is located; the construction resource intelligent agent objects that meet the set conditions at the target construction work surface are placed in the queue represented by the queue element for queuing and waiting; the construction resource intelligent agent objects that do not need to wait in queue or are taken out from the queue element at the target construction work surface are used to perform construction simulation of the construction process; after completing the simulation of the construction process of the construction unit intelligent agent object, the corresponding delay element ends.

[0152] The construction simulation model simulates a complete construction process, involving the interaction between multiple parts and the sequential execution of construction procedures. Multiple parts work together to complete the simulation of the construction process. The above simulation process specifically includes the following steps: Step A1, resource request; The construction unit agent object sends a resource request message to the construction resource scheduling agent object according to the corresponding construction process simulation model. Step A1 simulates the situation in which the construction unit requests the required resources from the department or system responsible for resource scheduling before starting construction in actual construction. For example, before the construction unit agent object performs the slag loading construction process, it needs to request construction resource agent objects such as the excavation equipment agent object and the dump equipment agent object from the construction resource scheduling agent object.

[0153] Step A2: resource scheduling; The construction resource scheduling agent object sends a task issuing message to at least one selected target construction resource agent object based on the received resource request message. The construction resource scheduling agent object selects appropriate construction resource agent objects based on the requested resource category, quantity, distance and other factors, and issues a task issuing message to notify them to prepare for work. For example, based on the location, status and other information of the transportation equipment agent object, the nearest and idle transportation equipment agent object to the construction unit agent object is selected, and a task issuing message is sent to it.

[0154] Step A3: Resource movement: After receiving the task assignment message, at least one target construction resource agent object moves to the target construction work surface where the construction unit agent object is located. This simulates the deployment process of construction resources to the work site in reality, such as the selected transportation equipment agent object will drive from the current location to the target construction work surface.

[0155] The process in which the at least one target construction resource intelligent agent object moves to the target construction work surface where the construction unit intelligent agent object is located after receiving the task issuing message specifically includes: Step B1, generating a road network model according to traffic data of the construction area; Taking the construction scenario of a hydropower project as an example, we can rely on traffic data such as the general layout plan and on-site traffic layout CAD (Computer-Aided Design) drawings to obtain the road network information of the area to be modeled, and generate a road network model described in the form of path points based on the coordinate information of the control points in the plane rectangular coordinate system.

[0156] The road network model is a model built based on the traffic data of the area to be modeled, which aims to describe the structure and characteristics of the road network in the area to be modeled in a digital way.

[0157] The road network model specifically includes the following elements: Waypoint: represents a specific location on the road, and its exact location is determined by coordinates in a plane rectangular coordinate system. These waypoints are arranged in sequence according to the direction of the road, and connected together to form the path of the road.

[0158] Path segment: It is composed of adjacent path points and represents a section of the road. Each path segment has its own specific attributes, such as length, direction, road type (such as main road, secondary road, etc.).

[0159] Topological relationship: describes the connection relationship between path points and path segments, as well as the intersection and branching relationships between roads. Through topological relationships, the structure of the entire road network can be clearly represented, which facilitates operations such as path planning and traffic flow analysis.

[0160] Step B2, setting road attribute parameters in the road network model; the road attribute parameters include: road grade, maximum allowed speed, maximum allowed traffic density, number of lanes and road width; Step B3, setting a mark of a set area in the road network model; the set area is the starting point or end point of the transportation; the set area includes: at least one of a construction work surface, a material storage area, and an equipment parking area; The material storage area may refer to the area where materials are brought in and managed, and the equipment parking area may refer to the area used to store construction resources.

[0161] Step B4: Generate a path file corresponding to the target construction resource intelligent object according to the road network model, wherein the path file includes: the starting point, the end point, the passing points, and the path length of the transportation; Step B5: After receiving the task issuing message, the at least one target construction resource intelligent agent object moves from the starting point to the target construction work surface where the construction unit intelligent agent object is located according to the path file.

[0162] The road network model and path file generated by steps B1-B4 can describe the specific route of the construction resource intelligent object from the starting point to the target construction work surface, including detailed information such as the starting point, end point, and passing points, providing clear navigation for the movement of the construction resource intelligent object. In step B2, road attribute parameters such as the maximum allowed vehicle speed, maximum vehicle density, number of lanes and road width are set in detail. These road attribute parameters reflect the road capacity and restriction conditions.

[0163] Step B4 can calculate the travel time of each road section by using the maximum allowed speed and the road length. Alternatively, step B4 can predict the road congestion according to the maximum traffic density. Alternatively, step B4 can determine the road carrying capacity by combining the number of lanes and the road width.

[0164] By comprehensively analyzing the road attribute parameters, step B4 can accurately calculate the length and estimated time of different paths, thereby selecting the optimal driving path for the construction resource intelligent object, so that the target construction resource intelligent object can efficiently and accurately reach the target construction work surface, ensuring the orderly progress of the construction process.

[0165] Step A4: Wait in line; The construction resource agent objects that meet the set conditions at the target construction work surface are placed in the queue represented by the queue element to wait in line. For example, due to the limited construction site, multiple construction equipment cannot work at the same time after arriving, and can only wait to enter the work surface in a certain order. Construction equipment that meets certain set conditions (such as first come first served, sorted by equipment performance, etc.) enters the queue sequence.

[0166] Step A5: construction simulation; The construction resource agent objects at the target construction work surface that do not need to wait in line or be taken out from the queue elements are used to perform construction simulation of the construction process. For example, according to the construction process and process standards, the excavation equipment agent object simulates the excavation action to control the excavation depth, angle and range, and the dump equipment agent object simulates the material loading, transportation and unloading process. Through the collaborative simulation of these construction resource agent objects, the actual operation process of the construction process is fully presented.

[0167] Step A6: The process is completed.

[0168] After completing the simulation of the i-th construction process of the construction unit agent object, the corresponding delay element ends.

[0169] Delay elements are used to simulate the duration of a construction process. Taking slag loading as an example, the delay element counts the duration of the entire construction process of slag loading, including the time it takes for the dump truck to move to the designated location, the time it takes for the excavator to dig and load, and the time it takes for the dump truck to wait in the queue. These times together constitute the total time required to complete the entire slag loading process, and the delay element comprehensively counts and accumulates these times to reflect the duration of the entire construction process.

[0170] The construction process simulation model may include: the connection relationship between different construction process simulation models. In this way, the embodiment of the present application can execute all construction processes included in the construction process simulation model according to the above connection relationship.

[0171] For example, Figure 5 The construction sequence of the construction process simulation model shown is: first carry out construction process A (measurement and layout), and after its completion, carry out construction process B (drilling), after the completion of construction process B, carry out construction process C (charging and connecting), after the completion of construction process C, start construction process D (blasting), after the completion of construction process D, start construction process E (slag loading), and finally, implement construction process F (slag transportation) based on the completion of construction process E. Each construction process is carried out strictly in this progressive order. The completion of the previous construction process creates conditions for the subsequent processes, and each construction process is simulated by the corresponding construction process simulation model (queue elements and delay elements) to simulate the queuing situation and duration in the construction process.

[0172] In a specific implementation, the construction simulation model may also include: a connection relationship between different construction process simulation models. The above connection relationship may include: series connection or parallel connection, etc. The embodiment of the present application can analyze the construction sequence between the construction unit intelligent body objects, and connect the construction process simulation models of each construction unit intelligent body object in series or parallel connection to obtain a construction simulation model of the construction operation, that is, to construct a static construction simulation model for the construction operation. It should be noted that in the static construction simulation model, the number of parallel construction process models will be pre-set in the initial stage. Subsequently, as the construction operation progresses, the number of parallel construction process simulation models will be dynamically updated based on the analysis results of the video image, so as to achieve the accuracy and real-time performance of the construction simulation model.

[0173] In a specific implementation, a process data structure can be generated according to the sequential relationship between the construction processes of different construction units; the process data structure is used to describe the sequential relationship between the construction processes of different construction units; according to the process data structure, the construction process simulation models corresponding to different construction units are connected; the connection relationship between different construction process simulation models includes: parallel or series connection.

[0174] Taking slope excavation as an example, the numbering sequence of the excavation blocks can be used to describe the excavation operation sequence between the excavation blocks, and the construction process models of the excavation blocks can be connected in series or in parallel in sequence to obtain a static construction simulation model of a certain excavation partition.

[0175] Reference Figure 6 , shows a schematic diagram of a construction simulation model of an embodiment of the present application, wherein the construction simulation model may include: parallel construction process simulation model A, construction process simulation model B, construction process simulation model C and construction process simulation model D. Assume that a certain excavation partition W is composed of four excavation blocks A, B, C, and D, and corresponding construction process simulation models A, B, C, and D are established respectively. If the four excavation blocks A, B, C, and D are excavated at the same time, the simulation model of the excavation partition W is Figure 6 As shown, it is composed of construction process simulation models A, B, C, and D in parallel, and the completion time is determined by the excavation block that ends excavation the latest.

[0176] The embodiment of the present application can execute the construction operations corresponding to all construction unit intelligent body objects according to the connection relationship between different construction process simulation models.

[0177] The embodiment of the present application can convert the connection relationship between different construction process simulation models in the construction simulation model into a data structure such as a directed graph, where the nodes in the directed graph represent the construction process simulation models, and the edges in the directed graph represent the chronological relationship between the construction process simulation models.

[0178] For the series relationship, it is reflected as a linear sequence in the directed graph, that is, the construction process simulation model corresponding to the next node can only start after the construction process simulation model corresponding to the previous node is completed; for the parallel relationship, multiple nodes can point to the same subsequent node at the same time, which means that the construction process simulation models corresponding to these nodes can be carried out in parallel.

[0179] In this way, the embodiment of the present application can perform topological sorting to clarify the order between the intelligent body objects of different construction units, use depth or breadth-first search algorithm to traverse, use multi-threading or process simulation to execute in parallel when encountering parallel processes, and strictly execute in sequence when encountering serial processes, while taking into account resource allocation and management, constraint processing, etc. in the entire process.

[0180] In step 302, target detection technology, target tracking technology and clustering technology may be used to automatically determine the construction resource distribution information corresponding to the construction work surface.

[0181] The process of determining the construction resource distribution information corresponding to the construction work surface of the construction work site by using target detection technology, target tracking technology and clustering technology according to the video image of the construction work site specifically includes: Step C1, performing target detection and target tracking on the video image, and obtaining detection frames, resource categories and identification numbers corresponding to a plurality of construction resources in the video image; Step C2: clustering the multiple construction resources in the video image according to the center point coordinates corresponding to the detection frames to obtain a plurality of clusters; the detection frames in the same cluster belong to the same construction work surface; Step C3: Determine the construction resource distribution information corresponding to the construction work surface of the construction work site according to the resource category corresponding to the detection frame in the same cluster.

[0182] In step C1, the target detection technology and target tracking technology are intended to identify different targets from the input video image and continuously track them, and determine the position, category and identification number of the target in the image. In the construction operation scenario, the construction resources are the targets to be detected and tracked. By using the trained target detection model and target tracking model (such as a convolutional neural network model based on deep learning, etc.), the video image is processed. The target detection model and the target tracking model will scan and continuously locate the area where multiple construction resources are located in the video image based on the learned feature patterns of various construction resources. This area is presented in the form of a detection frame, and at the same time, it can determine the category of the resource in each detection frame, such as a certain construction machinery, construction materials or construction personnel, etc., and the identification number corresponding to the resource in each detection frame, such as 1, 2, 3, etc. The construction machinery can specifically include: excavation equipment or self-unloading equipment, etc. The detection frame can be the corresponding circumscribed rectangular frame of the construction resource in the video image.

[0183] In step C2, since the spatial distribution of construction resources at the construction site is regular, the construction resources on the same construction surface often have relatively close spatial position characteristics in the input image. Clustering technology operates based on the attributes that can reflect the spatial scale, such as the coordinates of the center point corresponding to the detection frame. Classifying the detection frames with relatively close spatial distances into one category, that is, clustering them into a cluster, means that these construction resources with similar spatial scales in the input image are likely to be on the same construction surface. Because in the actual construction process, the spatial distances presented by the construction resources that cooperate to complete the same construction process in the image are similar, through this clustering method, the numerous construction resources at the entire construction site can be preliminarily divided according to the construction surfaces to which they belong, so as to distinguish the construction resources of different construction surfaces.

[0184] Assuming that the detection frame is represented by the center point coordinates (x, y), the center point coordinates of all detection frames are combined into a two-dimensional data set in the format of: data=[[x1,y1],[x2,y2],...] Manhattan distance is used as the similarity metric. According to the optimization goal of minimizing the distance within a cluster, a clustering algorithm is used to divide all detection frames into several clusters. The detection frames in the same cluster are considered to belong to the same construction work surface. According to the number of clusters, the number of construction work surfaces is obtained.

[0185] After clustering, each cluster (i.e., construction work surface) contains multiple detection frames, and each detection frame corresponds to a specific resource category. Step C3 can determine the specific construction resources and their distribution on each construction work surface by counting the resource categories corresponding to each detection frame in the same cluster and the number of each resource category.

[0186] The construction resource distribution information specifically includes: at least one resource category, and the quantity of resources corresponding to each resource category.

[0187] In a specific implementation, when the preset calling conditions are met, a calling request is sent to the video image analysis module, and the video image analysis module determines the construction resource distribution information corresponding to the construction working surface of the construction working site based on the video image of the construction working site; The preset calling conditions specifically include: Construction work commences; and / or There is an update in the construction process; and / or Changes in construction resources; and / or The next time period arrives.

[0188] In the process of construction modeling, if you want to accurately grasp the construction resource distribution information of each construction work surface at the construction site, you need to use the video image analysis module for analysis and processing. When the preset call conditions are met, a call request is sent to the video image analysis module. The purpose of this is to ensure that key information can be obtained in a timely manner while avoiding unnecessary waste of resources (such as computing resources, time costs, etc.), so that the analysis work can be carried out on demand and the distribution of construction resources can be accurately and efficiently obtained.

[0189] Among them, when the construction work starts, a call request is sent to the video image analysis module to clarify the initial resource configuration of each construction work surface and provide basic data for the instantiation of the construction simulation model.

[0190] When the construction process is updated, it means that the construction process has been switched. Sending a call request to the video image analysis module can determine whether the resource allocation under the new construction process is reasonable, ensuring smooth construction connection and on-demand resource allocation.

[0191] When construction resources change, a call request is sent to the video image analysis module to quickly understand the resource changes on the construction site.

[0192] When the next time period arrives, a call request is sent to the video image analysis module to regularly understand the dynamic changes of resources.

[0193] In a specific implementation, the construction resource distribution information specifically includes: at least one resource category, and the number of resources corresponding to each resource category; The above process of instantiating the queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information specifically includes: Step C1: if the number of resources of a resource category in the construction resource distribution information exceeds the maximum number of simultaneous workers, the construction resource agent objects exceeding the maximum number of simultaneous workers are first sorted according to arrival time or priority to obtain a first sorting result; Step C2: Save the first sorting result to the first queue corresponding to the queue element in the corresponding construction process simulation model.

[0194] Among them, in the construction resource distribution information, when the number of a resource category exceeds the maximum number of simultaneous workers, a sorting mechanism needs to be introduced to arrange these resources reasonably. Sorting by arrival time can give priority to the construction resource intelligent objects that arrive first; sorting by priority can highlight important construction resource intelligent objects. In this way, the construction resource intelligent objects that exceed the quantity limit are arranged in order.

[0195] On the one hand, the first sorting can make the use of construction resources more reasonable and orderly, avoid disordered competition and chaotic allocation between resource intelligent objects, and ensure the orderly development of the construction process. On the other hand, the first sorting creates good conditions for the accurate recording of the maintenance time of the delay element, making the statistics of relevant data such as resource usage time more logical and accurate, and thus better reflecting the time utilization of construction resources in the entire construction process.

[0196] Step C2 stores the first sorting result obtained in step C1 into the first queue corresponding to the queue element in the corresponding construction process simulation model, thereby achieving orderly management of the construction resource intelligent body objects.

[0197] Optionally, the above method may further include: When a construction resource intelligent agent object of a resource category completes the simulation of the construction process and the current number of tasks of a resource category is less than the maximum number of simultaneous tasks, the intelligent construction resource intelligent agent object of the corresponding resource category is taken out from the first queue for further simulation of the construction process.

[0198] In the construction process simulation, the construction resource agent object of the corresponding resource category taken out from the first queue must meet the following two conditions: Condition 1: The intelligent agent objects of a resource category complete the simulation of the construction process, that is, some intelligent agent objects in this resource category have completed the construction tasks currently assigned to them and are in an idle state.

[0199] Condition 2: The current number of tasks for a resource category is less than the maximum number of tasks that can be performed simultaneously, which indicates that there is still available space for this type of resource, allowing more resource agent objects to be put into work.

[0200] When these two conditions are met at the same time, the construction resource agent object of the corresponding resource category is taken out from the first queue and put into further simulation of the construction process to make full use of resources and ensure the continuity and efficiency of the construction simulation.

[0201] In a specific implementation, the process of instantiating the queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information specifically includes: Step D1: According to the collaborative relationship between the first resource category and the second resource category, the first construction resource intelligent agent object and the second construction resource intelligent agent object corresponding to the construction resource distribution information are matched to obtain a plurality of matching pairs; a matching pair includes: a first construction resource intelligent agent object and a second construction resource intelligent agent object; the number of the matching pairs does not exceed the maximum number of simultaneous operations; Step D2: performing a second sorting on the first construction resource agent objects that are not in the matching pair according to the arrival time or priority to obtain a second sorting result; Step D3, performing a third sorting on the second construction resource agent objects that are not in the matching pair according to the arrival time or priority, to obtain a third sorting result; Step D4: save the second sorting result to the second queue corresponding to the queue element in the corresponding construction process simulation model, and save the third sorting result to the third queue corresponding to the queue element in the corresponding construction process simulation model.

[0202] Among them, step D1 considers the collaborative relationship between different resource categories, pairs the related construction resource intelligent objects, and better utilizes the resources according to the collaboration, and controls the matching number not to exceed the maximum number of simultaneous working to ensure the rational use of resources. Step D1 forms a collaborative combination through matching, improves the efficiency of resource utilization, fits the resource coordination situation in actual construction, and helps to simulate the construction process more accurately.

[0203] Step D2 sorts the first construction resource agent objects that are not involved in the matching pair according to arrival time or priority to determine the order of their subsequent use.

[0204] In step D3, the second construction resource agent objects that are not in the matching pair are also sorted according to the arrival time or priority to standardize their usage order.

[0205] Step D4 stores the sorted results into the corresponding second queue and third queue respectively. By using the queue characteristics to store the resource sequence, it is possible to achieve orderly storage and management of different resources, making it convenient to subsequently retrieve resources in sequence for construction process simulation.

[0206] In summary, steps D1 to D4 reasonably match and sort the construction resource intelligent body objects according to the collaborative relationship and quantity of different resource categories in the construction resource distribution information, and save the results to the corresponding queue, so as to provide the construction process simulation model with an orderly resource allocation plan that meets the actual collaborative needs, so that the construction simulation can more accurately reflect the allocation and use of resources in the actual construction process, thereby assisting construction management and decision-making, and improving construction efficiency and quality.

[0207] Optionally, the above method may also include: when a matching pair completes the simulation of the construction process and the number of matching pairs working is less than the maximum number of matching pairs working simultaneously, taking out construction resource intelligent body objects of corresponding resource categories from the second and third queues respectively for further simulation of the construction process.

[0208] During the construction process simulation, for the resource matching pairs previously formed according to the collaborative relationship, when one of the matching pairs completes the simulation of the corresponding construction process, the number of matching pairs that are working will be checked. If the number of matching pairs that are working is less than the preset maximum number of simultaneous working pairs, it means that there is still room for new resources to continue the construction simulation. At this time, the construction resource intelligent body objects of the corresponding resource categories will be taken out from the second and third queues respectively, and they will be added to ensure that the construction process simulation can be carried out continuously and orderly, so that the entire construction simulation is more in line with the dynamic allocation and use of resources in actual construction. New matching pairs can be formed by taking out the construction resource intelligent body objects of the corresponding resource categories from the second and third queues respectively.

[0209] The delay element is used to count the duration of the construction process, while the queue element stores the sorting information of the construction resource agent objects. The state changes and operation sequence of the construction resource agent objects represented by the queue element during the construction process will affect the duration of the construction process, which is the time counted by the delay element. For example, the waiting time of the construction resource agent objects in the queue is part of the delay element statistics.

[0210] In a specific implementation, a delay element instance may be created for each construction process, and the delay element instance may be a data structure for storing and accumulating relevant time information. For example, an object containing attributes such as total duration, total waiting time, and total operation time may be created.

[0211] In addition, the total duration, the total waiting time, and the total working time can be calculated.

[0212] Calculation of the total waiting time: Based on the queue element's entry and exit times, calculate the waiting time of each construction resource agent object in the queue and add it to the total waiting time attribute of the delay element.

[0213] Calculation of operation time synthesis: According to the start time and end time of each construction operation performed by the construction resource agent object, the duration of each construction operation is calculated and added to the operation time sum attribute of the delay element.

[0214] Total duration calculation: Add the total waiting time to the total working time to get the total duration of the construction process, which is stored in the total duration attribute of the delay element.

[0215] During the construction process, as construction resource agent objects continue to enter and leave the queue and perform construction operations, the time statistics of delayed elements are continuously updated.

[0216] In an optional implementation of the present application, the construction simulation model may further include: a connection relationship between different construction process simulation models; the connection relationship specifically includes: parallel connection; The method may further include: determining the number of construction work surfaces at the construction work site according to the video image of the construction work site; and instantiating the number of parallel construction process models according to the number of construction work surfaces.

[0217] The construction simulation model covers the parallel connection relationship of different construction process simulation models. By analyzing the video images of the construction site and using image recognition and other technologies, the number of construction work surfaces can be determined based on visual features such as construction resource distribution and construction activity scope. Since each construction work surface is relatively independent and has specific construction tasks, the parallel construction process model is instantiated according to the number of construction work surfaces, so that the parallel process in the construction simulation model matches the parallel construction of multiple work surfaces in actual construction, so that the simulation is more in line with the actual construction status.

[0218] By instantiating the parallel construction process model according to the number of construction work surfaces, the fit between the construction simulation model and the actual construction can be significantly improved, and the scene of simultaneous construction of multiple construction work surfaces can be accurately simulated, providing a reliable reference for construction decision-making. At the same time, this method also helps to optimize the allocation and scheduling of construction resources, clearly present the resource requirements of each work surface, avoid resource waste, and improve utilization efficiency.

[0219] In summary, the automatic construction method of the construction simulation model based on video images of the embodiment of the present application adopts target detection technology, target tracking technology and clustering technology to analyze the video images of the construction site, so as to quickly and automatically determine the construction resource distribution information, and use the automatically determined construction resource distribution information for the instantiation of queue elements and delay elements. On the one hand, because the embodiment of the present application does not need to manually count the construction resource distribution information one by one, the collection time of the construction resource distribution information can be shortened. On the other hand, in view of the fact that the embodiment of the present application eliminates the link of manually inputting the construction resource distribution information, it is possible to save manual input time. In summary, the embodiment of the present application can improve the construction efficiency of the construction simulation model.

[0220] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0221] Based on the above embodiment, this embodiment also provides a construction progress prediction device based on long-term video analysis. Figure 7 The device may specifically include: a video analysis module 701, a video cropping module 702, an action segmentation module 703, a duration data determination module 704, a factual sample determination module 705, a random sample generation module 706, an assignment module 707 and a construction simulation module 708.

[0222] The video analysis module 701 is used to analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is a historical operation video of the construction operation, and / or the relevant operation video is a historical operation video of the same type of operation as the construction operation; The video cropping module 702 is used to crop the relevant operation video from the spatial dimension and the temporal dimension according to the equipment identification and the detection frame of the target construction machine to obtain the construction operation video of each target construction machine; The action segmentation module 703 uses the action segmentation model to segment the construction operation video of each target construction machine to obtain a plurality of construction operation segments arranged in chronological order; wherein two construction operation segments adjacent in time correspond to different action categories; The duration data determination module 704 is used to determine the duration data of the action category according to the multiple construction operation segments arranged in chronological order; The factual sample determination module 705 is used to determine the factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set consisting of multiple duration values ​​generated by the action category in the frame images at different times; A random sample generation module 706, configured to generate a random duration sample corresponding to an action category according to the factual duration sample corresponding to the action category; The assignment module 707 is used to assign a construction process parameter of a target construction machine in a construction simulation model of the construction operation according to the duration value corresponding to the random duration sample; the construction process parameter includes: a unit operation time corresponding to the target construction machine; The construction simulation module 708 is used to simulate the construction simulation model according to the construction process parameters of the target construction machinery to obtain the construction progress prediction result.

[0223] Optionally, the random sample generation module 707 includes: A distribution determination module, used for determining a target distribution function corresponding to an action category according to factual duration samples corresponding to the action category; The rejection sampling module is used to generate random duration samples that meet the target distribution function by adopting the rejection sampling method.

[0224] Optionally, the rejection sampling module includes: The first determination module is used to determine a reference distribution function and a constant; under the condition of the same independent variable, the function value of the target distribution function is not greater than the product of the function value of the reference distribution function and the constant; A first extraction module, used for extracting random independent variables from a reference distribution function; A second extraction module is used to extract random numbers from a uniform distribution function; the upper limit value of the uniform distribution function is: under the condition of the random independent variable, the product of the function value of the reference distribution function and a constant; The determination module is used for treating the random independent variable as a random duration sample that meets the target distribution function if the random number is smaller than the function value of the target distribution function corresponding to the random independent variable.

[0225] Optionally, the video cropping module includes: A start and end image determination module is used to determine the start frame image and the end frame image of each target construction machine in the relevant operation video; A maximum bounding box determination module is used to determine the maximum bounding box of each target construction machine according to the detection box of each target construction machine in the starting frame image, the intermediate frame image and the ending frame image; The multi-dimensional cropping module is used to crop the relevant operation video from the time dimension according to the starting frame image and the ending frame image, and to crop the relevant operation video from the space dimension according to the maximum bounding box to obtain the construction operation video of each target construction machinery.

[0226] Optionally, the action segmentation module includes: A video clip acquisition module, used for acquiring a plurality of video clips from a construction operation video of each target construction machine by using a sliding window with a preset window size and a preset step length; A multi-dimensional extraction module is used to extract RGB features and optical flow features corresponding to multiple video clips; The feature fusion module is used to fuse the RGB features and optical flow features corresponding to the multiple video clips to obtain the three-dimensional temporal features corresponding to the multiple video clips.

[0227] A first feature processing module is used to sequentially perform one-dimensional convolution processing, self-attention processing and feedforward processing on the three-dimensional time series features to obtain a fusion region feature corresponding to each target construction machine; A noise feature processing module is used to extract features of the noise distribution at the s-th time step to obtain the noise distribution features at the s-th time step; The noise time fusion module is used to fuse the noise distribution characteristics with the time coding vector to obtain the noise time distribution characteristics of the sth time step; The second feature processing module is used to extract the noise time distribution feature of the s-th time step to obtain the noise representation of the s-th time step; The cross attention processing module is used to concatenate the noise representation of the s-th time step with the fusion region feature as the query vector and the key vector. The noise representation of the s-th time step is used as the value vector and input into the cross attention layer. The cross attention layer calculates the attention weight between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weight to obtain the cross attention result. A fusion processing module is used to fuse the cross attention result with the noise representation of the s-th time step to obtain a fused representation; The result determination module is used to process the fused representation using a feed-forward connection layer and a convolutional layer to obtain the action category at the s-th time step.

[0228] Optionally, the construction simulation model includes: a construction unit intelligent agent object, a construction resource scheduling intelligent agent object, a construction resource intelligent agent object, and a construction process simulation model corresponding to a plurality of construction unit intelligent agent objects; the construction process simulation model includes: a construction process simulation model; the construction process simulation model includes: a queue element and a delay element; the queue element is used to simulate the queuing and waiting process of the construction resource intelligent agent objects in the construction process; the delay element is used to simulate the duration of the construction process; The simulation process of the construction simulation model includes: the construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task issuing message to at least one selected target construction resource intelligent agent object according to the resource request message; after receiving the task issuing message, at least one target construction resource intelligent agent object moves to the target construction work surface where the construction unit intelligent agent object is located; the construction resource intelligent agent objects that meet the set conditions at the target construction work surface are placed in a queue represented by a queue element for queuing and waiting; the construction resource intelligent agent objects that do not need to wait in queue or are taken out of the queue element at the target construction work surface are used to perform construction simulation of the construction process; after completing the simulation of the construction process of the construction unit intelligent agent object, the corresponding delay element ends; The device also includes: The resource analysis module determines the construction resource distribution information corresponding to the construction work surface of the construction work site by using target detection technology, target tracking technology and clustering technology based on the video image of the construction work site; The instantiation module is used to instantiate the queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information.

[0229] Optionally, the resource analysis module includes: The target detection module is used to perform target detection on the video image to obtain detection frames and resource categories corresponding to multiple construction resources in the video image; A clustering module, used for clustering the multiple construction resources in the video image according to the width and height corresponding to the detection frame to obtain a plurality of clusters; the detection frames in the same cluster belong to the same construction work surface; The distribution determination module is used to determine the construction resource distribution information corresponding to the construction work surface of the construction site according to the resource category corresponding to the detection frame in the same cluster.

[0230] Optionally, the construction simulation model further includes: a connection relationship between different construction process simulation models; the connection relationship includes: parallel connection; The device also includes: A working surface quantity determination module, used to determine the number of construction working surfaces at the construction working site based on the video image of the construction working site; The quantity instantiation module is used to instantiate the number of parallel construction process models according to the number of construction work surfaces.

[0231] Optionally, the setting conditions include: For a first construction resource intelligent agent object at a target construction work surface, the first construction resource intelligent agent object corresponds to a first resource category, and the number of first type of construction resource intelligent agent objects at the target construction work surface exceeds the maximum number of simultaneous operations; and / or For the first construction resource intelligent body object at the target construction work surface, the first construction resource intelligent body object corresponds to the first resource category, and the construction resource intelligent body object of the first resource category cooperates with the construction resource intelligent body object of the second resource category to jointly simulate the construction process.

[0232] Optionally, the construction resource distribution information includes: at least one resource category, and the quantity of resources corresponding to each resource category; The instantiation module includes: A first sorting module is used for, if the number of resources of a resource category in the construction resource distribution information exceeds the maximum number of simultaneous workers, to perform a first sorting on the construction resource agent objects exceeding the maximum number of simultaneous workers according to the arrival time or priority, to obtain a first sorting result; The first saving module is used to save the first sorting result to the first queue corresponding to the queue element in the corresponding construction process simulation model.

[0233] Optionally, the device further comprises: The dequeue module is used to take out the intelligent construction resource agent object of the corresponding resource category from the first queue for further simulation of the construction process when the construction resource agent object of a resource category completes the simulation of the construction process and the current number of jobs of a resource category is less than the maximum number of simultaneous jobs.

[0234] Optionally, the construction resource distribution information includes: at least one resource category, and the quantity of resources corresponding to each resource category; The instantiation module includes: A matching module is used to match the first construction resource intelligent agent object and the second construction resource intelligent agent object corresponding to the construction resource distribution information according to the collaborative relationship between the first resource category and the second resource category, so as to obtain a plurality of matching pairs; a matching pair includes: a first construction resource intelligent agent object and a second construction resource intelligent agent object; the number of the matching pairs does not exceed the maximum number of simultaneous operations; A second sorting module is used to perform a second sorting on the first construction resource agent objects that are not in the matching pair according to the arrival time or priority to obtain a second sorting result; A third sorting module, used to perform a third sorting on the second construction resource agent objects that are not in the matching pair according to the arrival time or priority, to obtain a third sorting result; The queue entry module is used to save the second sorting result to the second queue corresponding to the queue element in the corresponding construction process simulation model, and to save the third sorting result to the third queue corresponding to the queue element in the corresponding construction process simulation model.

[0235] The embodiment of the present application further provides a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0236] The present application embodiment provides one or more machine-readable media, on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices, servers (clusters), etc.

[0237] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the construction progress prediction methods based on long-term video analysis described in the above embodiments.

[0238] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Figure 8 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.

[0239] For one embodiment, Figure 8 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the processor(s) 1302, a memory 1306 coupled to the control module 1304, an NVP (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0240] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0241] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0242] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0243] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0244] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, for example, a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include a double data rate type four synchronous dynamic random access memory.

[0245] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the input / output device(s) 1310 .

[0246] For example, non-volatile memory / storage 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0247] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.

[0248] The input / output device(s) 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0249] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip.

[0250] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0251] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0252] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0253] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0254] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0255] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0256] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0257] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0258] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0259] The above is a detailed introduction to a construction progress prediction method and device based on long-term video analysis, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A construction progress prediction method based on long-term video analysis, characterized in that: The method comprises: Analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is a historical operation video of the construction operation, and / or the relevant operation video is a historical operation video of the same type of operation as the construction operation; According to the equipment identification and detection frame of the target construction machinery, the related operation video is cropped from the spatial dimension and the temporal dimension to obtain the construction operation video of each target construction machinery; Using the action segmentation model, the construction operation video is segmented to obtain multiple construction operation clips arranged in chronological order; wherein two construction operation clips adjacent in time correspond to different action categories; Determine duration data of action categories based on a plurality of construction work segments arranged in chronological order; Determine the factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set consisting of multiple duration values ​​generated by the action category in the frame images at different times; Generate random duration samples corresponding to the action categories according to the factual duration samples corresponding to the action categories; According to the duration value corresponding to the random duration sample, the construction process parameters of the target construction machinery in the construction simulation model of the construction operation are assigned; the construction process parameters include: the unit operation time corresponding to the target construction machinery; According to the construction process parameters of the target construction machinery, a construction simulation model is simulated to obtain a construction progress prediction result.

2. The method according to claim 1, characterized in that The step of generating a random duration sample corresponding to an action category according to the factual duration sample corresponding to the action category includes: Determine the target distribution function corresponding to the action category according to the factual duration samples corresponding to the action category; The rejection sampling method is used to generate random duration samples that conform to the target distribution function.

3. The method according to claim 2, characterized in that The method of adopting a rejection sampling method to generate a random duration sample that conforms to the target distribution function includes: Determine the reference distribution function and constant; under the condition of the same independent variable, the function value of the target distribution function is not greater than the product of the function value of the reference distribution function and the constant; Draw random independent variables from the reference distribution function; Extracting a random number from a uniform distribution function; the upper limit of the uniform distribution function is: the product of the function value of the reference distribution function and a constant under the condition of the random independent variable; If the random number is smaller than the function value of the target distribution function corresponding to the random independent variable, the random independent variable is taken as a random duration sample that conforms to the target distribution function.

4. The method according to claim 1, characterized in that: The relevant operation video is cut from the spatial dimension and the temporal dimension according to the equipment identification and the detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery, including: Determine the start frame image and the end frame image of each target construction machine in the relevant operation video; Determine the maximum bounding box of each target construction machine according to the detection box of each target construction machine in the starting frame image, the intermediate frame image and the ending frame image; The related operation video is cropped from the time dimension according to the starting frame image and the ending frame image, and the related operation video is cropped from the space dimension according to the maximum bounding box to obtain the construction operation video of each target construction machinery.

5. The method according to claim 1, characterized in that The use of the action segmentation model to segment the construction operation video of each target construction machine includes: Extracting three-dimensional temporal features of the construction operation video of each target construction machine; The three-dimensional time series features are sequentially subjected to one-dimensional convolution processing, self-attention processing, and feedforward processing to obtain fusion region features corresponding to each target construction machine; Using a denoising network model, determining the fusion region feature and the noise distribution at the s-th time step, and determining the action category at the s-th time step; According to the action category corresponding to each time step, the construction operation video of each target construction machine is segmented into multiple construction operation segments arranged in chronological order.

6. The method according to claim 5, characterized in that The method of using the denoising network model to determine the fusion region feature and the noise distribution at the s-th time step, and determining the action category at the s-th time step, includes: Extract the features of the noise distribution at the s-th time step to obtain the noise distribution features at the s-th time step; The noise distribution characteristics are fused with the time encoding vector to obtain the noise time distribution characteristics of the sth time step; Extract the noise time distribution characteristics of the s-th time step to obtain the noise representation of the s-th time step; The noise representation of the s-th time step is concatenated with the fused regional features as the query vector and the key vector. The noise representation of the s-th time step is used as the value vector and input into the cross attention layer. The cross attention layer calculates the attention weight between the query vector and the key vector, and performs weighted summation on the value vector according to the attention weight to obtain the cross attention result. Fuse the cross attention result with the noise representation of the s-th time step to obtain a fused representation; The fused representation is processed using feed-forward connection layers and convolutional layers to obtain the action category at the s-th time step.

7. The method according to any one of claims 1 to 6, characterized in that: The construction simulation model includes: a construction unit intelligent agent object, a construction resource scheduling intelligent agent object, a construction resource intelligent agent object, and a construction process simulation model corresponding to a plurality of construction unit intelligent agent objects; the construction process simulation model includes: a construction process simulation model; the construction process simulation model includes: a queue element and a delay element; the queue element is used to simulate the queuing and waiting process of the construction resource intelligent agent objects in the construction process; the delay element is used to simulate the duration of the construction process; The simulation process of the construction simulation model includes: the construction unit intelligent agent object sends a resource request message to the construction resource scheduling intelligent agent object according to the corresponding construction process simulation model; the construction resource scheduling intelligent agent object sends a task issuing message to at least one selected target construction resource intelligent agent object according to the resource request message; after receiving the task issuing message, at least one target construction resource intelligent agent object moves to the target construction work surface where the construction unit intelligent agent object is located; the construction resource intelligent agent objects that meet the set conditions at the target construction work surface are placed in a queue represented by a queue element for queuing and waiting; the construction resource intelligent agent objects that do not need to wait in queue or are taken out of the queue element at the target construction work surface are used to perform construction simulation of the construction process; after completing the simulation of the construction process of the construction unit intelligent agent object, the corresponding delay element ends; The method further comprises: Based on the video images of the construction site, the target detection technology, target tracking technology and clustering technology are used to determine the construction resource distribution information corresponding to the construction work surface of the construction site; According to the construction resource distribution information, the queue elements and delay elements in the corresponding construction process simulation model are instantiated.

8. The method according to claim 7, characterized in that The method of determining the construction resource distribution information corresponding to the construction work surface of the construction work site by using target detection technology, target tracking technology and clustering technology based on the video image of the construction work site includes: Performing target detection on the video image to obtain detection frames and resource categories corresponding to a plurality of construction resources in the video image; Clustering multiple construction resources in the video image according to the width and height corresponding to the detection frame to obtain a plurality of clusters; the detection frames in the same cluster belong to the same construction work surface; According to the resource categories corresponding to the detection frames in the same cluster, the construction resource distribution information corresponding to the construction working surface of the construction working site is determined.

9. The method according to claim 7, characterized in that: The construction simulation model also includes: a connection relationship between different construction process simulation models; the connection relationship includes: parallel connection; The method further comprises: Determine the number of construction work surfaces at the construction work site based on the video images of the construction work site; According to the number of construction work surfaces, the number of parallel construction process models is instantiated.

10. The method according to claim 7, characterized in that The setting conditions include: For a first construction resource intelligent agent object at a target construction work surface, the first construction resource intelligent agent object corresponds to a first resource category, and the number of first type of construction resource intelligent agent objects at the target construction work surface exceeds the maximum number of simultaneous operations; and / or For the first construction resource intelligent body object at the target construction work surface, the first construction resource intelligent body object corresponds to the first resource category, and the construction resource intelligent body object of the first resource category cooperates with the construction resource intelligent body object of the second resource category to jointly simulate the construction process.

11. The method according to claim 7, characterized in that The construction resource distribution information includes: at least one resource category and the quantity of resources corresponding to each resource category; The instantiation of queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information includes: If the number of resources of a resource category in the construction resource distribution information exceeds the maximum number of simultaneous workers, the construction resource agent objects exceeding the maximum number of simultaneous workers are first sorted according to the arrival time or priority to obtain a first sorting result; The first sorting result is saved to the first queue corresponding to the queue element in the corresponding construction process simulation model.

12. The method according to claim 11, characterized in that The method further comprises: When a construction resource intelligent agent object of a resource category completes the simulation of the construction process and the current number of tasks of a resource category is less than the maximum number of simultaneous tasks, the intelligent construction resource intelligent agent object of the corresponding resource category is taken out from the first queue for the simulation of the construction process.

13. The method according to claim 7, characterized in that The construction resource distribution information includes: at least one resource category and the quantity of resources corresponding to each resource category; The instantiation of queue elements and delay elements in the corresponding construction process simulation model according to the construction resource distribution information includes: According to the collaborative relationship between the first resource category and the second resource category, the first construction resource intelligent agent object and the second construction resource intelligent agent object corresponding to the construction resource distribution information are matched to obtain a plurality of matching pairs; a matching pair includes: a first construction resource intelligent agent object and a second construction resource intelligent agent object; the number of the matching pairs does not exceed the maximum number of simultaneous operations; According to the arrival time or priority, the first construction resource agent objects that are not in the matching pair are sorted in a second manner to obtain a second sorting result; Performing a third sorting on the second construction resource agent objects that are not in the matching pair according to the arrival time or priority to obtain a third sorting result; The second sorting result is saved to the second queue corresponding to the queue element in the corresponding construction process simulation model, and the third sorting result is saved to the third queue corresponding to the queue element in the corresponding construction process simulation model.

14. A construction progress prediction device based on long-term video analysis, characterized in that: The device comprises: The video analysis module is used to analyze the relevant operation videos of the construction operation to obtain the equipment category, equipment identification and detection frame of the target construction machinery; the target construction machinery is the construction machinery that restricts the progress of the construction process; the duration of the relevant operation video is greater than the duration threshold; the relevant operation video is a historical operation video of the construction operation, and / or the relevant operation video is a historical operation video of the same type of operation as the construction operation; A video cropping module is used to crop the relevant operation video from the spatial dimension and the temporal dimension according to the equipment identification and the detection frame of the target construction machinery to obtain the construction operation video of each target construction machinery; The action segmentation module is used to segment the construction operation video of each target construction machine using the action segmentation model to obtain multiple construction operation segments arranged in chronological order; wherein two construction operation segments adjacent in time correspond to different action categories; A duration data determination module, used to determine duration data of an action category according to a plurality of construction operation segments arranged in chronological order; A factual sample determination module is used to determine the factual duration sample corresponding to the action category according to the duration data; the factual duration sample includes: a data set consisting of multiple duration values ​​generated by the action category in the frame images at different moments; A random sample generation module, used for generating random duration samples corresponding to action categories according to factual duration samples corresponding to action categories; An assignment module is used to assign a construction process parameter of a target construction machinery in a construction simulation model of a construction operation according to a duration value corresponding to the random duration sample; the construction process parameter includes: a unit operation time corresponding to the target construction machinery; The construction simulation module is used to simulate the construction simulation model according to the construction process parameters of the target construction machinery to obtain the construction progress prediction result.

15. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 13.

16. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images