Video-based engineering quantity prediction method and device, equipment, medium and product

Abstract

The invention discloses a video-based engineering quantity prediction method and device, equipment, a medium and a product, and the method comprises the steps: obtaining video data of a historical time period of an engineering site, extracting the video data to obtain a video frame sequence, determining region nodes and a space-time connection relation of the region nodes based on the video frame sequence, generating an engineering topological graph of the engineering site, and predicting the engineering quantity of the engineering site according to the engineering topological graph. And obtaining node features of the region nodes, determining global representation of the video data in the historical time period, performing construction action prediction based on the global representation of the video data in the historical time period, and obtaining a construction action sequence of the project site in the future time period. And according to the construction action sequence in the future time period and the predetermined unit volume corresponding to the construction action, obtaining the predicted project amount of the project site in the future time period. According to the method, the construction action chain of the project thread in the future time period is accurately predicted through the project site video data, and the accurate project amount can be effectively predicted in advance in combination with the unit amount of the construction action.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of engineering management, and particularly relates to a video-based engineering quantity prediction method, device, equipment, medium and product. BACKGROUND

[0002] With the rapid development of smart construction sites and the increasing demand for intelligence, the management and scheduling of construction sites are gradually moving towards automation and intelligence. Among them, engineering quantity, as one of the core indicators of construction management, the traditional engineering quantity statistical method mainly relies on manual measurement or device sensors, which has problems such as low efficiency and response lag. Therefore, how to realize efficient prediction of engineering quantity has become an urgent need for current technical development.

[0003] At present, the automatic management of engineering quantity in construction sites mainly passes through two levels of action recognition and three-dimensional modeling. In the action recognition level, specific actions such as "loading" and "unloading" in the construction site are recognized to infer the current situation of the construction site. In the three-dimensional modeling level, a three-dimensional model of the construction site is constructed to provide geometric spatial information of the construction site, helping to understand the spatial layout and equipment position of the construction site. However, this method only recognizes the executed construction action or only constructs the geometric space of the construction site, which cannot accurately map the regional function and future engineering quantity, resulting in insufficient accuracy of engineering quantity statistics and response lag, affecting the management efficiency of the construction site. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a video-based engineering quantity prediction method, device, equipment, medium and product, which can solve the problem that the current method cannot accurately map the regional function and future engineering quantity, resulting in insufficient accuracy of engineering quantity statistics and response lag, affecting the management efficiency of the construction site.

[0005] In order to solve the above technical problems, the present application is implemented as follows: In a first aspect, the embodiments of the present application provide a video-based engineering quantity prediction method, which comprises: Obtaining video data of a historical time period of an engineering site, and extracting a video frame sequence from the video data; wherein the video frame sequence comprises at least one construction action segment; Based on the video frame sequence, determining a regional node and a spatio-temporal connection relationship of the regional node, and generating an engineering topology graph of the engineering site; According to the access relationship between the regional node in the engineering topology graph and the construction action segment, extracting the access feature of the regional node to obtain the node feature of the regional node; determine a global representation of the video data of the historical time period according to the node features of the region nodes, perform construction action prediction based on the global representation of the video data of the historical time period to obtain a construction action sequence of the construction site in a future time period; obtain predicted engineering quantities of the construction site in the future time period according to the construction action sequence of the future time period and a predetermined unit quantity corresponding to a construction action.

[0006] Optionally, the generating of the engineering topology of the construction site based on the sequence of video frames and the determination of the region nodes and the spatio-temporal connection relationship of the region nodes comprises: filtering out similar frames from the sequence of video frames, and classifying similar frames in the same group as a region node; performing similarity calculation on a current video frame to be processed in the sequence of video frames and an existing region node to determine maximum similarity between the current video frame and the existing region node; determining a region node corresponding to the current video frame and a spatio-temporal connection relationship of the region node according to the maximum similarity, a preset node similarity threshold and a similarity boundary interval; adding directed edges to all region nodes corresponding to the sequence of video frames according to the spatio-temporal connection relationship to generate an engineering topology of the construction site.

[0007] Optionally, the filtering out of similar frames from the sequence of video frames and the classifying of similar frames in the same group as a region node comprises: sampling at least one pair of video frames from the sequence of video frames, determining whether a sampling time difference between the pair of video frames is less than a preset inter-frame time threshold, and if so, determining that the pair of video frames are similar frames and classifying similar frames in the same group as a region node; or sampling at least one pair of video frames from the sequence of video frames, extracting key points of the pair of video frames, determining matching key points existing in the pair of video frames according to distances of feature vectors corresponding to the key points, and if a number of the matching key points is greater than or equal to a preset threshold, determining that the pair of video frames are similar frames and classifying similar frames in the same group as a region node.

[0008] Optionally, the performing of similarity calculation on a current video frame to be processed in the sequence of video frames and an existing region node to determine maximum similarity between the current video frame and the existing region node comprises: obtaining an initial topology, and classifying region nodes of similar frames in the same group as existing region nodes of the initial topology; performing similarity calculation on the current video frame to be processed in the sequence of video frames and the existing region nodes to respectively obtain similarity scores of the current video frame and each of the existing region nodes. determining a maximum similarity between the current video frame and the existing region node from the similarity scores of the current video frame and each of the existing region nodes.

[0009] Optionally, the determining the region node corresponding to the current video frame and the spatio-temporal connection relationship of the region nodes according to the maximum similarity, a preset node similarity threshold, and a similarity boundary interval comprises: comparing the maximum similarity with the preset node similarity threshold, and a difference between the node similarity threshold and the similarity boundary interval, respectively; if the maximum similarity is greater than the node similarity threshold, merging the current video frame to the existing region node corresponding to the maximum similarity; if the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval, newly creating a region node for the current video frame; determining the spatio-temporal connection relationship of the region nodes based on the region node corresponding to the current video frame and the construction action.

[0010] Optionally, the extracting the access feature of the region node according to the access relationship between the region node and the construction action segment in the engineering topology map to obtain the node feature of the region node comprises: determining the access relationship between the region node and the construction action segment in the engineering topology map; extracting the feature of the construction action segment in the access relationship corresponding to the region node to obtain the access feature of the region node; performing mean processing on the access feature of the region node to obtain the node feature of the region node.

[0011] Optionally, the determining the global representation of the video data of the historical time period according to the node feature of the region node, and performing construction action prediction based on the global representation of the video data of the historical time period to obtain the construction action sequence of the future time period of the construction site comprises: inputting the node feature of the region node into a graph convolutional neural network to obtain a node representation of the region node; performing mean processing on the node representation of the region node to obtain the global representation of the video data of the historical time period; inputting the global representation of the video data of the historical time period into a pre-trained classifier to perform construction action prediction to obtain the construction action sequence of the future time period of the construction site.

[0012] Optionally, before the inputting the global representation of the video data of the historical time period into the pre-trained classifier to perform construction action prediction to obtain the construction action sequence of the future time period of the construction site, the method further comprises: obtain a set of construction actions visited by the region node, and generate an action interaction label of the region node according to the set of construction actions; extract an image feature vector from the access segment of the region node, pair the image feature vector with the action interaction label, and predict an availability vector of the region node; use the action interaction label and the availability vector of the region node as training samples, train by using cross-entropy loss, and obtain a classifier for outputting a sequence of construction actions in a future time period.

[0013] Optionally, the obtaining of the predicted engineering quantity of the engineering site in the future time period according to the sequence of construction actions in the future time period and the predetermined unit square quantity corresponding to the construction action includes: determining an action type and an execution frequency of the construction action according to the sequence of construction actions in the future time period; mapping the sequence of construction actions in the future time period to an engineering quantity based on the action type and the execution frequency of the construction action and the predetermined unit square quantity corresponding to the construction action, and obtaining the predicted engineering quantity of the engineering site in the future time period.

[0014] Optionally, after the obtaining of the predicted engineering quantity of the engineering site in the future time period according to the sequence of construction actions in the future time period and the predetermined unit square quantity corresponding to the construction action, the method further includes: obtaining a completed engineering quantity of the engineering site in a historical time period according to a construction action segment of the engineering site in the historical time period and the predetermined unit square quantity corresponding to the construction action; pre-scheduling engineering equipment resources to a target region node according to the predicted engineering quantity and the completed engineering quantity.

[0015] In a second aspect, an engineering quantity prediction device based on video is provided, and the device includes: an acquisition data module configured to acquire video data of an engineering site in a historical time period, and extract a video frame sequence from the video data; the video frame sequence includes at least one construction action segment; a generation topology module configured to determine a region node and a spatiotemporal connection relationship of the region node based on the video frame sequence, and generate an engineering topology graph of the engineering site; a feature extraction module configured to extract an access feature of the region node according to an access relationship between the region node and the construction action segment in the engineering topology graph, and obtain a node feature of the region node. an action prediction module configured to determine a global representation of the video data of the historical time period according to the node features of the region nodes, perform construction action prediction based on the global representation of the video data of the historical time period, and obtain a construction action sequence of the construction site in a future time period; an engineering quantity prediction module configured to obtain predicted engineering quantities of the construction site in the future time period according to the construction action sequence of the future time period and a predetermined unit square quantity corresponding to a construction action.

[0016] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method for predicting engineering quantities based on video according to the first aspect.

[0017] In a fourth aspect, a readable storage medium is provided, which stores a program or instructions, and the program or instructions, when executed by a processor, implement the steps of the method for predicting engineering quantities based on video according to the first aspect.

[0018] In a fifth aspect, a computer program product is provided, which includes a program or instructions, and the program or instructions, when executed by a processor, implement the steps of the method for predicting engineering quantities based on video according to the first aspect.

[0019] The method for predicting engineering quantities based on video provided in the embodiments of the present application can obtain video data of a historical time period of a construction site, extract a video frame sequence from the video data, determine region nodes and a spatio-temporal connection relationship of the region nodes based on the video frame sequence, generate an engineering topology graph of the construction site, extract access features of the region nodes according to an access relationship between the region nodes and construction action segments in the engineering topology graph, obtain node features of the region nodes, determine a global representation of the video data of the historical time period according to the node features of the region nodes, perform construction action prediction based on the global representation of the video data of the historical time period, obtain a construction action sequence of the construction site in a future time period, and obtain predicted engineering quantities of the construction site in the future time period according to the construction action sequence of the future time period and a predetermined unit square quantity corresponding to a construction action. The embodiments of the present application can effectively filter out spatial structures related to construction actions by capturing time sequence information and functional expressions of regions of video data of a construction site, generate a topology graph reflecting dynamic interactions of construction action regions of the construction site, obtain construction action execution features of the regions based on video understanding, accurately predict a construction action chain of the construction site in a future time period, output predicted engineering quantities in combination with unit square quantities of the construction actions, replace a lagging response of traditional manual statistics of engineering quantities, and can effectively predict accurate engineering quantities in advance, thereby realizing intelligent management of the construction site.

[0020] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application and implement the same according to the contents of the specification, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0021] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not meant to limit the present application. Moreover, the same reference numerals in the drawings indicate the same or similar components. In the drawings: Figure 1 is a step flow chart of a video-based engineering quantity prediction method provided by an embodiment of the present application; Figure 2 is Figure 1 is a flow chart of step 102 in a video-based engineering quantity prediction method provided by an embodiment of the present application; Figure 3 is a step flow chart of another video-based engineering quantity prediction method provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of a video-based engineering quantity prediction device provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0023] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0024] The video-based engineering quantity prediction method, device, equipment, medium and product provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings, specific embodiments and application scenarios.

[0025] To solve the problem that only the executed construction actions are identified or only the geometric space of the construction site is constructed, the area function cannot be accurately mapped to the future engineering quantity, resulting in insufficient accuracy of engineering quantity statistics and lag of response, and affecting the management efficiency of the construction site, the embodiments of the present application process the video data of the engineering site to obtain an engineering topology graph composed of area nodes and directed edges, and establish the relationship between the functions of the areas in different construction environments, complete the conversion of the engineering site from "visual space" to "functional space", use the function representation of the area nodes in the engineering topology graph, predict the possible construction actions of the construction equipment and personnel on some areas in the dynamic construction environment reflected by the construction site video, combine the unit quantity of construction work, accurately predict the future engineering quantity of the construction site, realize the dynamic calculation and prediction of engineering quantity driven by video understanding, and timely and effectively obtain the engineering quantity of the future time period, so as to manage the construction site in advance.

[0026] Reference Figure 1 The step flowchart of the video-based engineering quantity prediction method provided by the embodiments of the present application is shown, and the method can include: Step 101, obtaining video data of a historical time period of an engineering site, and extracting video frame sequences from the video data; wherein the video frame sequences include at least one construction action segment.

[0027] It should be noted that the execution subject of the embodiments of the present application can be a control end of an engineering management system, an engineering monitoring platform, etc. The control end obtains video data of a historical time period of an engineering site, and extracts video frame sequences from the video data. Specifically, the video data can be frame-extracted to obtain video frame sequences containing multiple video frames, wherein the video frame sequences include at least one construction action segment, and the construction action segment refers to a video frame segment containing a construction action, and the construction action includes an action performed by a construction equipment or a construction personnel, such as loading, unloading, excavating, transporting, leveling, etc. For example, the video data of the engineering site can be obtained by a drone aerial photography or a fixed monitoring device, and the video frame sequences can be extracted by frame extraction. The engineering site can be the site of a construction project such as a water conservancy project or a large-scale earthwork allocation, and the video data of the historical time period can include video data collected in a certain length of time in the past as a data basis for prediction in the future time period, and the length of the historical time period is not specifically limited in this embodiment.

[0028] Step 102, determining area nodes and the spatio-temporal connection relationship of the area nodes based on the video frame sequences, and generating an engineering topology graph of the engineering site.

[0029] In the embodiment of the application, based on the video frame sequence, the region nodes and the spatio-temporal connection relationship of the region nodes are determined, and the engineering topology graph of the engineering site is generated. In the specific implementation, based on the video frame sequence, similar frames are first selected, and similar frames in the same group are classified as a region node. Similarity calculation is performed on the current video frame to be processed in the video frame sequence and the existing region nodes, the region node of the current video frame is determined, and the spatio-temporal connection relationship of all region nodes is determined. Thus, the video frame sequence corresponding to all region nodes is added with directed edges according to the spatio-temporal connection relationship, and the engineering topology graph of the engineering site is generated.

[0030] In the embodiment of the application, based on the region node corresponding to the current video frame and the construction action association of the video frames in the region node, the spatio-temporal connection relationship of the region node is determined. After obtaining all region nodes and the spatio-temporal connection relationship thereof, the video frame sequence corresponding to all region nodes is added with directed edges according to the spatio-temporal connection relationship, and the engineering topology graph of the engineering site is generated. The region node reflects the region where the construction action occurs, such as the loading area, the unloading area, etc. The directed edge represents the spatio-temporal association of the movement mode of the construction equipment and the construction personnel between different regions.

[0031] In step 103, the access feature of the region node is extracted according to the access relationship between the region node and the construction action segment in the engineering topology graph, and the node feature of the region node is obtained.

[0032] In the embodiment of the application, the region where the construction action occurs in the engineering site may be accessed by multiple groups of construction action segments, that is, the same region appears multiple construction actions at different times and is collected into a video segment. Therefore, in order to reflect the access situation of the construction action of different region nodes, the embodiment extracts the access feature of the region node according to the access relationship between the region node and the construction action segment in the engineering topology graph, and obtains the node feature of the region node.

[0033] In the specific implementation, the access relationship between the region node and the construction action segment in the engineering topology graph is first determined, that is, according to the access segment in the region node, the feature of the construction action segment in the access relationship corresponding to the region node is extracted, the construction action segment and the construction action are recognized, the access feature of the region node is obtained, and the access feature of the region node is processed by mean value to obtain the node feature of the region node.

[0034] In step 104, the global representation of the video data of the historical time period is determined according to the node feature of the region node, the construction action prediction is performed based on the global representation of the video data of the historical time period, and the construction action sequence of the future time period of the engineering site is obtained.

[0035] In the embodiments of the present application, in order to predict the construction actions that will occur in the future time period of the engineering site based on the engineering topology graph, the global representation of the video data of the historical time period is determined according to the node features of the region nodes, so as to perform action prediction by using the global representation. Specifically, the node features of all region nodes are input into a graph convolutional neural network (GCN) to fuse the information of the region nodes and obtain node representations. The node representations of the region nodes are processed by averaging to obtain the global representation of the video data of the historical time period. Then, the construction action prediction is performed based on the global representation of the video data of the historical time period to obtain the construction action sequence of the future time period of the engineering site.

[0036] It should be noted that the global representation of the video data of the historical time period is input into a pre-trained classifier to perform construction action prediction to obtain the construction action sequence of the future time period of the engineering site. The classifier is a model or functional module that can input the region node availability vector, i.e., the construction action sequence of the future time period of the engineering site, and is pre-trained. The construction action sequence reflects the construction actions and the number of actions that are likely to occur in the future for each region node.

[0037] In step 105, the predicted engineering quantity of the future time period of the engineering site is obtained according to the construction action sequence of the future time period and the pre-determined unit square quantity corresponding to the construction action.

[0038] In the embodiments of the present application, the pre-determined unit square quantity corresponding to the construction action refers to the work quantity that can be completed by a single execution of the construction action, such as the single loading work quantity of the muck truck, the single flattening work quantity of the bulldozer, and the single unloading work quantity of the muck truck. After the construction action sequence of the future time period is determined, the construction action sequence of the future time period is mapped to the engineering quantity by combining the action type and the execution times of the construction actions in the construction action sequence of the future time period with the unit square quantity corresponding to the construction action, to obtain the predicted engineering quantity of the future time period of the engineering site. In the dynamic construction environment reflected by the construction site video, the future possible construction actions of the construction equipment and personnel on some regions are predicted, the unit square quantity of the construction work is combined, the engineering quantity of the construction site is accurately predicted, and the dynamic calculation and prediction of the engineering quantity driven by video understanding are realized.

[0039] The engineering quantity prediction method based on video provided in the embodiments of the present application can effectively screen the spatial structure related to the construction action by capturing the time sequence information and the functional representation of the region of the engineering site video data, generate a topology graph reflecting the dynamic interaction of the construction action region of the engineering site, obtain the construction action execution features of the region based on video understanding, accurately predict the construction action chain of the future time period of the engineering thread, combine the unit square quantity of the construction action, output the predicted engineering quantity, replace the lagging response of the traditional manual engineering quantity statistics, and can effectively predict the accurate engineering quantity in advance, so as to realize the intelligent management of the engineering site.

[0040] Refer to Figure 2 , which shows Figure 1 a flowchart of step 102 in a video-based engineering quantity prediction method provided, and this method is basically the same as the video-based engineering quantity prediction method provided in the first embodiment of the present application. Step 102 may include: Step 201, based on the video frame sequence, screen out similar frames, and classify the same group of similar frames into one region node; Step 202, for the current video frame to be processed in the video frame sequence, perform similarity calculation with the existing region nodes to determine the maximum similarity between the current video frame and the existing region nodes; Step 203, according to the maximum similarity, a preset node similarity threshold, and a similarity boundary interval, determine the region node corresponding to the current video frame and the spatio-temporal connection relationship of the region nodes; Step 204, add directed edges to all the region nodes corresponding to the video frame sequence according to the spatio-temporal connection relationship to generate an engineering topology graph of the engineering site.

[0041] In the embodiment of the present application, to construct an engineering topology graph of the engineering site, first, based on the video frame sequence, screen out similar frames, and classify the same group of similar frames into one region node. Among them, the video frame sequence is denoted as F=(f1,f2,...,fT). A pre-trained region localization network is used to process each video frame in the video frame sequence. The region localization network is used to output the similarity between two video frames, so as to determine that video frames with similar content belong to the same physical or logical region. Therefore, a group of similar frames of the video frame sequence is used as the first region node of the engineering topology graph. The engineering topology graph includes a region node set and a directed edge set. For the current video frame to be processed in the video frame sequence, perform similarity calculation with the existing region nodes to determine the maximum similarity between the current video frame and the existing region nodes. The existing region nodes are the region nodes classified from multiple groups of determined similar frames. Thus, according to the maximum similarity, a preset node similarity threshold, and a similarity boundary interval, determine the region node corresponding to the current video frame and the spatio-temporal connection relationship of the region nodes, and add directed edges to all the region nodes corresponding to the video frame sequence according to the spatio-temporal connection relationship to generate an engineering topology graph of the engineering site.

[0042] It should be noted that by using a pre-trained region localization network to process the video frame sequence and construct an engineering topology map, the average cosine similarity between the feature vectors of the current video frame and all representative frames in the region node can be calculated by the region localization network, and the similarity between the current video frame to be processed and each existing region node in the graph can be calculated and output. From the similarities between the current video frame and each existing region node in the graph, find the maximum similarity to determine the region node most similar to the current video frame.

[0043] In this embodiment, after determining the maximum similarity between the current video frame and existing region nodes, i.e., determining the region node most similar to the current video frame, it is necessary to further determine whether the current video frame should be merged into a region node or a new region node should be created. Combining the preset node similarity threshold and similarity boundary interval, the region node corresponding to the current video frame and the spatiotemporal connection relationship of the region node are determined. Here, the node similarity threshold refers to the similarity threshold used in video frame sequence analysis to determine whether two video frames belong to the same region node. Specifically, it is a preset value and is not limited here. When the similarity between two video frames is higher than the node similarity threshold, they belong to the same region node, and the current video frame can be merged into the region node. When the similarity is lower than the node similarity threshold, they do not belong to the same region node, and a new region node needs to be created. To ensure more accurate processing of creating a new region node for the current video frame, a refined judgment of the similarity boundary interval is introduced. In particular, when the maximum similarity is less than the node similarity threshold, if the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval, that is, when the similarity between the video frame and the region node is near the node similarity threshold, then a new region node is considered for the current video frame. If the maximum similarity is greater than the difference between the node similarity threshold and the similarity boundary interval, it indicates that the current video frame is similar to the region node, but to avoid redundancy of nodes in the topology graph, this part of the video frame is not processed.

[0044] In this embodiment, based on the region node corresponding to the current video frame and the construction action association of the video frames in the region node, the spatiotemporal connection relationship of the region node is determined. After obtaining all region nodes and their spatiotemporal connection relationships, directed edges are added to all region nodes corresponding to the video frame sequence according to the spatiotemporal connection relationships to generate an engineering topology map of the construction site, which provides a structured representation for subsequent understanding of regional functions. Among them, the region node reflects the region where construction actions occur, such as loading area, unloading area, etc., and the directed edge represents the spatiotemporal association of the movement mode of construction equipment and construction personnel between different areas.

[0045] Based on the collected video data, this application embodiment constructs an engineering topology map that reflects the areas where construction actions occur and the relationships between construction actions between areas, thereby realizing the transformation of the engineering site from "visual space" to "functional space".

[0046] In some embodiments of this application, step 201, based on a video frame sequence, filters out similar frames and categorizes the same group of similar frames into a region node. Specifically, this may include the following steps: S11, sample at least one pair of video frames from the video frame sequence, determine whether the sampling time difference between the video frame pairs is less than a preset inter-frame time threshold, if so, determine that the video frame pairs are similar frames, and classify the same group of similar frames into one region node; or, S12, sample at least one pair of video frames from the video frame sequence, extract key points of the video frame pairs, determine the matching key points of the video frame pairs based on the distance between the feature vectors corresponding to the key points, and determine the video frame pairs as similar frames if the number of matching key points is greater than or equal to a preset threshold, and classify the same group of similar frames into a region node.

[0047] In this embodiment, similar frames are selected based on the video frame sequence, and the same group of similar frames is classified into a region node. This can be achieved through two parallel and complementary judgment mechanisms. If either condition is met, the two frames can be determined to be similar frames and classified into the same region node. Specifically, similar frames are selected based on temporal proximity or visual similarity and classified into the same region. Temporal proximity can capture the spatial continuity reflected by the tendency to stay in a specific operation area; visual similarity is used to identify frame pairs that repeat in the background even though the foreground object changes significantly. Conversely, if the frame pair has low visual similarity and a long time interval, or is only an incidental viewpoint in the scene where no substantial operation has occurred, it is classified as a different frame and awaits further judgment.

[0048] In specific implementation, similar frames are selected based on visual similarity. At least one pair of video frames is sampled from the video frame sequence, and it is determined whether the sampling time difference between the video frame pairs is less than a preset inter-frame time threshold. If so, the video frame pairs are determined to be similar frames, and the same group of similar frames is classified into one region node. Based on visual similarity, keypoint isomorphism can be used to determine similar frames. At least one pair of video frames is sampled from the video frame sequence, and keypoints of the video frame pairs are extracted. According to the distance between the feature vectors corresponding to the keypoints, the matching keypoints of the video frame pairs are determined. If the number of matching keypoints is greater than or equal to a preset threshold, the video frame pairs are determined to be similar frames, and the same group of similar frames is classified into one region node. Specifically, firstly, SuperPoint is used to extract the keypoints of the video frames, and then ResNet-152 is used to extract the frame-level features, i.e., feature vectors, of the keypoints. The Euclidean distance between the feature vectors corresponding to the keypoints is calculated. If the distance between the feature vectors corresponding to the keypoints is less than a set threshold, it is considered that there are matching keypoints between the two frames. If the number of matching keypoints is greater than or equal to the preset threshold, the video frame pairs are determined to be similar frames.

[0049] It's important to note that SuperPoint is a deep learning model used to detect keypoints in images, such as corners and edge intersections, and generates a descriptor for each keypoint to describe the visual features around it. ResNet-152 is a deep convolutional network used to extract global features from the entire image. Keypoints are extracted from two frames separately, and descriptors are generated. The Euclidean distance between the descriptors of the two frames is calculated as the distance between the feature vectors corresponding to the keypoints. A smaller distance indicates similar visual features. When a sufficient number of keypoint pairs can be successfully matched, for example, more than 10 matching keypoints, it proves that the background structures of the two frames are highly consistent, and they are determined to be similar frames.

[0050] This application embodiment filters similar frames by temporal proximity or visual similarity, accurately identifying areas from video data that reflect construction activities, so as to construct an engineering topology map of the construction site.

[0051] In some embodiments of this application, step 202 involves calculating the similarity between the current video frame to be processed in the video frame sequence and existing region nodes to determine the maximum similarity between the current video frame and existing region nodes. Specifically, this may include the following steps: S21, Obtain the initial topology graph, and use the region nodes of the same group of similar frames as the existing region nodes of the initial topology graph; S22, For the current video frame to be processed in the video frame sequence, perform similarity calculation with the existing region nodes to obtain the similarity score between the current video frame and each existing region node. S23, determine the maximum similarity between the current video frame and existing regional nodes from the similarity scores between the current video frame and each existing regional node.

[0052] In this embodiment, an initial topology map is obtained, and the region nodes that classify similar frames in the same group are used as existing region nodes in the initial topology map. For the current video frame to be processed in the video frame sequence, a pre-trained region localization network can be used to calculate the similarity between the current video frame and each existing region node, and obtain the similarity score between the current video frame and each existing region node. From the similarity scores between the current video frame and each existing region node, the maximum similarity between the current video frame and the existing region node is determined. The maximum similarity is used to further determine whether the current video frame can be added to the region node corresponding to the maximum similarity.

[0053] It should be noted that the region localization network is a twin neural network, consisting of two neural network branches with shared weights. It takes two video frames as input and outputs a similarity score between the two video frames. If the two video frames belong to the same region or the same action scene, the output similarity score is close to 1; if they belong to different regions or unrelated scenes, the output similarity score is close to 0.

[0054] The training data for the region localization network consists of all similar frame pairs found through filtering. Using the trained region localization network, the similarity between the current video frame and each existing region node is calculated, yielding a similarity score between the current video frame and each existing region node. This score is calculated using the following formula:

[0055] In this context, MLP stands for Multilayer Perceptron, which maps video frame differences to a similarity space, and σ is the Sigmoid function, which outputs a similarity score. (f) represents the video frame features extracted by a CNN (such as ResNet-18), and the absolute value of the difference between the video frame features is used to measure the discrepancy.

[0056] In this embodiment, for the current video frame to be processed in the video frame sequence, a similarity calculation is performed with existing region nodes, where n∈N. The similarity scores between the current video frame and each existing region node are obtained. From these similarity scores, the maximum similarity between the current video frame and each existing region node is determined using the following formula:

[0057]

[0058] This application embodiment calculates the similarity between the current video frame to be processed in the video frame sequence and existing regional nodes to determine the association between the video frame and the regional nodes, so as to generate an engineering topology map that accurately reflects the functional areas of the engineering site.

[0059] In some embodiments of this application, step 203 determines the region nodes corresponding to the current video frame and the spatiotemporal connectivity of the region nodes based on the maximum similarity, a preset node similarity threshold, and a similarity boundary interval. Specifically, this may include the following steps: S31, compare the maximum similarity with the preset node similarity threshold, and the difference between the node similarity threshold and the similarity boundary interval; S32, If the maximum similarity is greater than the node similarity threshold, then merge the current video frame into the existing region node corresponding to the maximum similarity. S33, If the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval, then create a new region node for the current video frame; S34. Based on the regional nodes corresponding to the current video frame and the construction actions, determine the spatiotemporal connection relationship of the regional nodes.

[0060] In this embodiment, the maximum similarity is compared with a preset node similarity threshold and the difference between the node similarity threshold and the similarity boundary interval. If the maximum similarity is greater than the node similarity threshold, it indicates that the current video frame has a high scene similarity with the region node. In this case, the current video frame is merged into the existing region node corresponding to the maximum similarity. If the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval, a new region node is created for the current video frame. Finally, based on the region node corresponding to each current video frame to be processed in the video frame sequence and the construction action, the spatiotemporal connection relationship of all region nodes is determined to generate an engineering topology map.

[0061] It should be noted that if the maximum similarity is greater than the node similarity threshold, the current video frame is merged into the existing region node corresponding to the maximum similarity. The merging position of the current video frame within the region node needs to be determined. Since the same region node may be accessed by multiple construction action segments—for example, if the excavation face region node has one excavation operation on day 1 and another on day 2—this region node corresponds to two construction action segment accesses. Therefore, the region node includes access segments of multiple sets of construction actions. When merging the current video frame, if the current video frame is continuous with the video frame of the previous access segment of the region node, the current video frame is extended into that access segment; otherwise, a new access segment is created in the region node to place the current video frame. If the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval, a new region node is directly created for the current video frame.

[0062] This application utilizes similarity indicators to accurately convert video data into regional nodes and construction actions, determining the spatiotemporal connection relationships of all regional nodes to generate an engineering topology map.

[0063] In this embodiment of the application, step 103 extracts the access features of the regional nodes based on the access relationship between regional nodes and construction action segments in the engineering topology diagram, thereby obtaining the node features of the regional nodes. Specifically, this may include: S41, Determine the access relationship between regional nodes and construction action segments in the engineering topology diagram; S42, extract features from the construction action segments in the access relationship corresponding to the regional nodes to obtain the access features of the regional nodes; S43, the access characteristics of the regional nodes are averaged to obtain the node characteristics of the regional nodes.

[0064] In this embodiment, since the regional nodes include multiple sets of access segments of construction actions, this embodiment needs to determine the access characteristics of each regional node in order to clarify the construction actions that occur on each regional node. Therefore, the access relationship between regional nodes and construction action segments in the engineering topology diagram is first determined. That is, based on the access segments in the regional nodes, the construction action segments in the corresponding access relationships of the regional nodes are feature extracted to identify the construction action segments and construction actions, thereby obtaining the access characteristics of the regional nodes. The access characteristics of the regional nodes are then averaged to obtain the node characteristics of the regional nodes.

[0065] It should be noted that feature extraction is performed on the construction action segments in the access relationship corresponding to the regional nodes to obtain the access features of the regional nodes. The access features are a feature vector. The mean of the access features of the regional nodes is then applied to obtain the node features of the regional nodes, as shown below:

[0066]

[0067] Where v represents the access segment of the region node. (ft) is the frame feature vector, xv is the access feature of the region node, and xn is the node feature of the region node.

[0068] This application embodiment determines the access characteristics of each regional node to clarify the construction actions occurring on each regional node. By utilizing the functional representation of regional nodes in the engineering topology diagram, it can predict the construction actions that construction equipment and personnel may perform in some areas in the future within the dynamic construction environment reflected in the construction site video.

[0069] In this embodiment, step 104 determines the global representation of video data for a historical time period based on the node characteristics of the regional nodes, and predicts construction actions based on the global representation of the video data for the historical time period to obtain the sequence of construction actions at the construction site for the future time period. Specifically, this may include: S51, input the node features of the region nodes into the graph convolutional neural network to obtain the node representation of the region nodes; S52, the node representations of the regional nodes are averaged to obtain a global representation of the video data for the historical time period; S53 inputs the global representation of video data for historical time periods into a pre-trained classifier to predict construction actions and obtain the sequence of construction actions for future time periods at the construction site.

[0070] In this embodiment, the node features of the regional nodes are input into a graph convolutional neural network (GCN) to obtain the node representation of the regional nodes. Specifically, the node features of all regional nodes are input into the GCN, and the information of the regional nodes is fused to obtain the node representation gn. The node representation of the regional nodes is then averaged to obtain the global representation xG of the video data for the historical time period. The specific formula is as follows:

[0071]

[0072] The global representation of historical video data is input into a pre-trained classifier to predict construction actions, resulting in a sequence of construction actions for future time periods at the construction site. This sequence reflects the predicted future construction actions and their frequency for each area node. The sequence is represented as a D-dimensional binary vector. ∈ ,in, .

[0073] This application embodiment uses the functional representation of regional nodes in the engineering topology diagram to accurately predict the construction actions that construction equipment and personnel may perform in some areas in the future, in order to accurately determine the engineering quantity corresponding to the construction actions, in the dynamic construction environment reflected in the construction site video.

[0074] As a specific implementation of this application embodiment, before inputting the global representation of the video data of the historical time period into a pre-trained classifier to predict construction actions and obtain the construction action sequence of the future time period at the construction site in step S53, it may further include: S54, obtain the set of construction actions that the regional node has been accessed, and generate action interaction tags for the regional node based on the set of construction actions; S55: Extract image feature vectors from the access segments of the region nodes, pair the image feature vectors with action interaction labels, and predict the availability vectors of the region nodes. S56 uses the action interaction labels and availability vectors of the regional nodes as training samples and employs cross-loss entropy for training to obtain a classifier for outputting construction action sequences for future time periods.

[0075] In this embodiment, each region node represents not only a scene at a specific moment, but also includes the interactive actions available in that region, such as digging, loading, unloading, and leveling. Since a single video clip often only shows one way of using a region, this embodiment requires combining cross-video and cross-environment video understanding to complete all potential interactive possibilities of the region nodes. For example, in a construction scene, an excavator can dig in this region, and a dump truck can load soil in this region; the interactive possibilities provided by this region are called availability. To represent the interactive possibilities of a region, the problem is defined as a multi-label classification task, that is, mapping the features of a frame to an A-dimensional binary indicator vector. Where A represents the total number of possible interaction types (e.g., mining, loading, transporting, unloading, leveling, etc.). If a node in a certain region has the ability to perform a certain type of interaction action, the corresponding dimension takes the value of 1; otherwise, it takes the value of 0.

[0076] To train the aforementioned availability classifier, for each node n∈N in the engineering topology map G(N, E), its node-level action interaction label yn is calculated, which is the set of construction actions observed by that node in all access segments. For each access to a node n, a frame of image is sampled from that access segment, and an image feature vector x is extracted. This image feature vector is paired with the corresponding node action interaction label n as a training sample. The availability vector y^n is predicted using a two-layer multilayer perceptron (MLP). The action interaction labels and availability vectors of the node are used as training samples, and cross-loss entropy is used for training to obtain a classifier for outputting the construction action sequence for future time periods. The training objective is that even if the "slag truck unloading" scene does not appear in a certain video segment, the system can infer that the "slag yard unloading area" has unloading availability. The availability vector y^n is represented as:

[0077] This application embodiment trains a classifier that can predict the sequence of construction actions in the future time period of the project site based on the functional representation of regional nodes in the engineering topology graph by analyzing the availability of regional nodes, thereby improving the accuracy and efficiency of construction action prediction.

[0078] In some embodiments of this application, step 105 obtains the predicted engineering volume for the future time period based on the sequence of construction actions in the future time period and the pre-determined unit volume corresponding to the construction actions. Specifically, it may include the following steps: S61, Based on the sequence of construction actions in the future time period, determine the action type and execution frequency of the construction actions; S62, based on the action type and execution frequency of construction actions, as well as the pre-determined unit volume corresponding to the construction actions, maps the sequence of construction actions in the future time period into the engineering quantity, and obtains the predicted engineering quantity of the construction site in the future time period.

[0079] In this embodiment, the predicted construction action sequence is mapped to the engineering quantity. That is, based on the construction action sequence in the future time period and the pre-determined unit volume of the construction action, the predicted engineering quantity of the construction site in the future time period is obtained. For example, in the earthwork excavation scenario, the predicted abstract construction action sequence is converted into a quantified earthwork operation quantity and applied to construction management. That is, based on the construction action sequence in the future time period, the action type and execution number of the construction action are determined. Based on the action type and execution number of the construction action and the pre-determined unit volume of the construction action, the construction action sequence in the future time period is mapped to the engineering quantity to obtain the predicted engineering quantity of the construction site in the future time period.

[0080] It should be noted that the pre-determined unit volume corresponding to the construction action refers to the amount of work that can be completed in a single execution of the construction action, such as the loading volume of a dump truck in a single operation, the leveling volume of a bulldozer in a single operation, and the unloading volume of a dump truck in a single operation. The unit volume is m³ / operation. Based on the type and number of executions of the construction actions, as well as the pre-determined unit volume corresponding to the construction actions, the predicted work volume for the future time period at the construction site is calculated. For example, the loading volume is the product of the predicted number of loading operations and the loading volume of a single dump truck operation.

[0081] This application embodiment accurately predicts the future workload of the construction site by predicting the construction actions that construction equipment and personnel may perform in some areas in the future, and combining this with the unit volume of construction work, thus realizing the dynamic calculation and prediction of the workload driven by video understanding.

[0082] Reference Figure 3 This diagram illustrates a flowchart of another video-based quantity prediction method provided in an embodiment of this application. This method is essentially the same as the video-based quantity prediction method provided in the first embodiment of this application, except that the method may further include: Step 101: Obtain video data from historical time periods at the construction site, and extract video frame sequences from the video data; wherein, the video frame sequence includes at least one construction action segment; Step 102: Based on the video frame sequence, determine the regional nodes and their spatiotemporal connections, and generate an engineering topology map of the engineering site. Step 103: Based on the access relationship between regional nodes and construction action segments in the engineering topology diagram, extract the access features of regional nodes to obtain the node features of regional nodes. Step 104: Based on the node characteristics of the regional nodes, determine the global representation of the video data for the historical time period, and predict the construction actions based on the global representation of the video data for the historical time period to obtain the construction action sequence for the future time period at the construction site. Step 105: Based on the sequence of construction actions in the future time period and the pre-determined unit volume of the construction actions, obtain the predicted engineering volume of the project site in the future time period.

[0083] Steps 101 to 105 above are the same as those in the previous sequence and will not be repeated here.

[0084] Step 106: Based on the construction action segments of the historical time period at the project site and the pre-determined unit volume corresponding to the construction action, obtain the completed work volume of the historical time period at the project site.

[0085] Step 107: Based on the predicted workload and the completed workload, pre-schedule engineering equipment resources to the target area nodes.

[0086] In this embodiment, the completed work volume for the historical time period of the project site is obtained based on the construction action segments of the project site and the pre-determined unit volume corresponding to the construction action. Based on the determined predicted work volume and the completed work volume, the project equipment resources are pre-scheduled to the target area node according to the predicted work volume and the completed work volume, so as to realize the linkage between the predicted work volume and the resource scheduling. The target area node is the construction area where the predicted work volume is greater than the preset value. After the target area node is determined, the corresponding construction equipment or construction personnel are pre-scheduled to the area node according to the predicted work volume of the target area node. For example, the muck trucks, excavators and unloading point resources are pre-allocated to achieve pre-response, effective project resource scheduling and dynamic optimization.

[0087] It should be noted that, in some embodiments, the total amount of work completed in the historical time period and the predicted amount of work in the future time period at the construction site can be determined based on the completed and predicted amounts of work. Resource scheduling can then be performed according to the total amount of work. These details will not be elaborated upon here. The completed amount of work, the predicted amount of work, and the total amount of work are calculated using the following formulas:

[0088] Here, Q event(i) represents the amount of work corresponding to the i-th construction action.

[0089] Compared with the prior art, the implementation method of this application, based on the beneficial effects brought by the first implementation method, links the predicted workload with resource scheduling, and pre-schedules the corresponding construction equipment or personnel to the target area node according to the predicted workload, thereby achieving pre-response, effective engineering resource scheduling and dynamic optimization, and further improving the management effect of the engineering site.

[0090] To enable those skilled in the art to more clearly understand the video-based engineering quantity prediction method described in the above embodiments, the following explanation uses engineering quantity prediction in a construction excavation engineering scenario as an example: Using video data captured by a drone as input, firstly, through a regional positioning network constructed by a Siamese neural network, the construction-related activity areas such as excavators, muck trucks, and construction faces in the video frames are automatically divided into regional nodes. Using a topological availability graph construction algorithm, directed edges are generated to form the access relationships and spatiotemporal relationships between regional nodes, thus forming the topological relationships between construction areas. Based on the construction actions that may occur in the construction areas identified in the input video data, a graph convolutional neural network is used to infer and predict the construction actions of regional nodes in the future time period, thereby predicting the engineering quantity in advance and scheduling resources.

[0091] Example: In a construction project, an engineering topology map containing "excavation area - loading area - slag yard" was constructed using drone video. Even if some video clips did not show "slag truck unloading", it was still possible to infer that the area had unloading function through cross-environment links, and predict that after the excavator truck loaded the slag truck, the slag truck would drive to the unloading area next.

[0092] S1: Use drones to take aerial photos of the slag yard at a certain construction site every day to obtain continuous video data of excavators and slag trucks operating in the construction area, and obtain a video frame sequence (f1,f2,...,fT).

[0093] S2: Video clips of the same excavator loading in the morning on a sunny day and unloading in the afternoon on a rainy day. Although the lighting and environment are different, the same area, namely the excavator's operating area, can still be identified through the regional positioning network.

[0094] S3: The areas reflected in the video frames, namely "excavation face," "loading point," and "slag yard unloading point," are respectively formed into regional nodes, and directed edges are formed through the driving trajectory of the slag truck. Although the "excavator operation area" in different construction action segments may have different appearances (different sizes, different colors), they can be linked into the same type of functional area because their functions are consistent.

[0095] S4: Even if the video does not show the "slag truck reversing and unloading", it can be inferred that the "slag yard unloading area" has unloading availability. After detecting "excavator loading" and "slag truck departure", the most likely next action is predicted to be "slag truck driving to the unloading area and completing unloading".

[0096] S5: Each construction action corresponds to a unit volume, which yields the predicted project volume. For example, one excavator loading is approximately equal to one bucket of earth and stone, which can be set according to the excavator model, such as 3.5 m³ / bucket; one dump truck unloading is approximately equal to one truckload of earth and stone, which can be set according to the vehicle model and load capacity, such as 20 m³ / truck. The predicted project volume is then calculated.

[0097] S6: If it is predicted that there will be 10 excavator loading operations in the loading area in the next 30 minutes, the estimated workload will be 35m³. Sufficient slag truck transportation capacity needs to be pre-scheduled. The capacity of the slag yard or the allocation of reserved unloading points can also be adjusted in advance, and the number of slag trucks dispatched can be adjusted to avoid local congestion or waiting.

[0098] For example, during the construction of a water conservancy project, it was observed that there were 8 loading operations by excavators and 8 departures of muck trucks in a historical period. Based on the predicted amount of work to be done by 8 unloading operations of muck trucks, it can be inferred that 160 m³ of earth and rock needs to be transported away in the future. This can prompt the pre-arrangement of multiple muck trucks to operate in a cyclical manner.

[0099] Reference Figure 4 The diagram shows a structural schematic of a video-based engineering quantity prediction device according to an embodiment of this application. The device includes: The data acquisition module 301 is used to acquire video data from a historical time period at the construction site and extract a video frame sequence from the video data; wherein the video frame sequence includes at least one construction action segment. The topology generation module 302 is used to determine the regional nodes and their spatiotemporal connections based on the video frame sequence, and generate an engineering topology map of the engineering site. The feature extraction module 303 is used to extract the access features of the regional nodes based on the access relationship between the regional nodes and the construction action segments in the engineering topology diagram, and obtain the node features of the regional nodes. The action prediction module 304 is used to determine the global representation of video data for a historical time period based on the node characteristics of the regional nodes, and to predict construction actions based on the global representation of video data for a historical time period to obtain the construction action sequence for a future time period at the construction site. The project quantity prediction module 305 is used to obtain the predicted project quantity for the future time period based on the construction action sequence for the future time period and the pre-determined unit volume corresponding to the construction actions. Optionally, the topology generation module 302 includes: The filtering submodule is used to filter out similar frames based on the video frame sequence and classify the same group of similar frames into a region node; The first calculation submodule is used to perform similarity calculation between the current video frame to be processed in the video frame sequence and existing region nodes, and to determine the maximum similarity between the current video frame and the existing region nodes. The first determining submodule is used to determine the region nodes corresponding to the current video frame and the spatiotemporal connection relationship of the region nodes based on the maximum similarity, the preset node similarity threshold and the similarity boundary interval. The generation submodule is used to add directed edges to all regional nodes corresponding to the video frame sequence according to the spatiotemporal connection relationship, thereby generating the engineering topology map of the engineering site.

[0100] Optionally, the filtering submodule includes: The first judgment unit is configured to sample at least one pair of video frames from the video frame sequence, determine whether the sampling time difference between the video frame pairs is less than a preset inter-frame time threshold, and if so, determine that the video frame pairs are similar frames, and classify the same group of similar frames into one region node; or, The second judgment unit is used to sample at least one set of video frame pairs from the video frame sequence, extract key points of the video frame pairs, determine matching key points of the video frame pairs based on the feature vector distance corresponding to the key points, and determine that the video frame pairs are similar frames if the number of matching key points is greater than or equal to a preset threshold, and classify the same set of similar frames into a region node.

[0101] Optionally, the first computing submodule includes: The acquisition unit is used to acquire the initial topology map and to classify the region nodes of the same group of similar frames as the existing region nodes of the initial topology map. The calculation unit is used to perform similarity calculation between the current video frame to be processed in the video frame sequence and the existing region nodes, and obtain the similarity score between the current video frame and each of the existing region nodes respectively. The first determining unit is used to determine the maximum similarity between the current video frame and the existing regional nodes from the similarity scores between the current video frame and each of the existing regional nodes.

[0102] Optionally, the first determining submodule includes: The comparison unit is used to compare the maximum similarity with a preset node similarity threshold and the difference between the node similarity threshold and the similarity boundary interval. The merging unit is used to merge the current video frame into the existing region node corresponding to the maximum similarity if the maximum similarity is greater than the node similarity threshold. A new unit is created to create a new region node for the current video frame if the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval. The second determining unit is used to determine the spatiotemporal connection relationship of the regional nodes based on the regional nodes corresponding to the current video frame and the construction actions.

[0103] Optionally, the feature extraction module 303 includes: The second determining submodule is used to determine the access relationship between the regional nodes in the engineering topology diagram and the construction action segments; The extraction submodule is used to extract features from the construction action segments in the access relationship corresponding to the regional nodes to obtain the access features of the regional nodes. The processing submodule is used to perform mean processing on the access characteristics of the regional nodes to obtain the node characteristics of the regional nodes.

[0104] Optionally, the action prediction module 304 includes: The node representation submodule is used to input the node features of the region nodes into the graph convolutional neural network to obtain the node representation of the region nodes; The global representation submodule is used to perform mean processing on the node representations of the regional nodes to obtain a global representation of the video data for the historical time period; The prediction submodule is used to input the global representation of the video data of the historical time period into a pre-trained classifier to predict the construction actions and obtain the sequence of construction actions in the future time period of the project site.

[0105] Optionally, the action prediction module 304 further includes: A tag generation submodule is used to obtain the set of construction actions accessed by the regional node, and generate action interaction tags for the regional node based on the set of construction actions. An availability prediction submodule is used to extract image feature vectors from the access segments of the region node, pair the image feature vectors with the action interaction labels, and predict the availability vectors of the region node. The training submodule is used to train a classifier that uses the action interaction labels and availability vectors of the regional nodes as training samples and employs cross-loss entropy to output the construction action sequence for future time periods.

[0106] Optionally, the quantity prediction module 305 includes: The third determining submodule is used to determine the action type and execution number of the construction action based on the construction action sequence of the future time period; The mapping submodule is used to map the sequence of construction actions in the future time period into the quantity of work based on the action type and execution number of the construction actions and the pre-determined unit volume of the construction actions, so as to obtain the predicted quantity of work in the future time period of the construction site.

[0107] Optionally, the device further includes: The "Determine Project Quantity" module is used to obtain the completed project quantity for the historical time period of the project site based on the construction action segments of the project site during the historical time period and the pre-determined unit volume corresponding to the construction action. The resource scheduling module is used to pre-schedule engineering equipment resources to target area nodes based on the predicted engineering workload and the completed engineering workload.

[0108] The video-based quantity prediction device provided in this application can realize all the processes implemented by the video-based quantity prediction method in the above embodiments of this application. To avoid repetition, it will not be described again here.

[0109] The video-based engineering quantity prediction device provided in this application can effectively filter out the spatial structures related to construction actions by capturing the temporal information and functional expression of video data from the construction site, and generate a topology map reflecting the dynamic interaction of construction action areas on the construction site. Based on video understanding, it can obtain the execution characteristics of construction actions in the area, accurately predict the construction action chain of the construction site in the future time period, and output the predicted engineering quantity by combining the unit volume of construction actions. It replaces the lagging response of traditional manual statistical engineering quantity, and can predict the accurate engineering quantity in advance and effectively, so as to realize the intelligent management of the construction site.

[0110] Reference Figure 5 This application also provides an electronic device, such as... Figure 5 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. Processor 401, memory 403 for storing processor-executable instructions; The processor 401 is configured to execute the instructions to implement the video-based quantity prediction method as described below: Video data from historical time periods at the construction site is acquired, and video frame sequences are extracted from the video data; each video frame sequence includes at least one segment of a construction action. Based on the video frame sequence, the regional nodes and their spatiotemporal connections are determined, and an engineering topology map of the engineering site is generated. Based on the access relationship between regional nodes and construction action segments in the engineering topology diagram, the access features of regional nodes are extracted to obtain the node features of regional nodes. Based on the node characteristics of regional nodes, a global representation of video data for historical time periods is determined. Based on the global representation of video data for historical time periods, construction actions are predicted to obtain the sequence of construction actions for future time periods at the construction site. Based on the sequence of construction actions in the future time period and the pre-determined unit volume of the construction actions, the predicted amount of work done at the construction site in the future time period is obtained.

[0111] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0112] The communication interface is used for communication between the aforementioned terminal and other devices.

[0113] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0114] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0115] In another embodiment provided in this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the video-based engineering quantity prediction method described in any of the above embodiments.

[0116] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0117] In another embodiment of this application, a computer program product is also provided, including a program or instructions that, when executed by a processor, implement the steps in the video-based engineering quantity prediction method described in any of the above embodiments of this application.

[0118] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0119] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0120] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A video-based method for predicting engineering quantities, characterized in that, The method includes: Video data from historical time periods at the construction site is acquired, and video frame sequences are extracted from the video data; wherein, the video frame sequence includes at least one construction action segment; Based on the video frame sequence, regional nodes and their spatiotemporal connections are determined, and an engineering topology map of the engineering site is generated. Based on the access relationship between the regional nodes and the construction action segments in the engineering topology diagram, the access features of the regional nodes are extracted to obtain the node features of the regional nodes. Based on the node characteristics of the regional nodes, a global representation of the video data for historical time periods is determined. Based on the global representation of the video data for historical time periods, construction actions are predicted to obtain a sequence of construction actions for future time periods at the construction site. Based on the sequence of construction actions for the future time period and the pre-determined unit volume corresponding to the construction actions, the predicted engineering volume for the future time period at the project site is obtained.

2. The method according to claim 1, characterized in that, The step of determining regional nodes and their spatiotemporal connections based on the video frame sequence, and generating an engineering topology map of the engineering site, includes: Based on the video frame sequence, similar frames are selected and the same group of similar frames are classified into one region node; For the current video frame to be processed in the video frame sequence, a similarity calculation is performed with existing region nodes to determine the maximum similarity between the current video frame and the existing region nodes; Based on the maximum similarity, the preset node similarity threshold, and the similarity boundary interval, the region nodes corresponding to the current video frame and the spatiotemporal connection relationship of the region nodes are determined. By adding directed edges to all regional nodes corresponding to the video frame sequence according to the spatiotemporal connection relationship, an engineering topology map of the engineering site is generated.

3. The method according to claim 2, characterized in that, The step of filtering out similar frames based on the video frame sequence and grouping the same group of similar frames into a region node includes: At least one pair of video frames is sampled from the video frame sequence. It is determined whether the sampling time difference between the video frame pairs is less than a preset inter-frame time threshold. If so, the video frame pairs are identified as similar frames, and the same group of similar frames is classified into one region node; or... At least one pair of video frames is sampled from the video frame sequence, key points of the video frame pair are extracted, and matching key points of the video frame pair are determined based on the feature vector distance corresponding to the key points. If the number of matching key points is greater than or equal to a preset threshold, the video frame pair is determined to be similar frames, and the same group of similar frames is classified into a region node.

4. The method according to claim 2, characterized in that, The step of performing similarity calculations between the current video frame to be processed in the video frame sequence and existing region nodes to determine the maximum similarity between the current video frame and the existing region nodes includes: Obtain the initial topology graph, and use the region nodes that are classified into the same group of similar frames as the existing region nodes of the initial topology graph; For the current video frame to be processed in the video frame sequence, a similarity calculation is performed with the existing region nodes to obtain the similarity score between the current video frame and each of the existing region nodes. The maximum similarity between the current video frame and each of the existing region nodes is determined from the similarity scores between the current video frame and each of the existing region nodes.

5. The method according to claim 2, characterized in that, The step of determining the region nodes corresponding to the current video frame and the spatiotemporal connectivity of the region nodes based on the maximum similarity, a preset node similarity threshold, and a similarity boundary interval includes: The maximum similarity is compared with a preset node similarity threshold, and the difference between the node similarity threshold and the similarity boundary interval. If the maximum similarity is greater than the node similarity threshold, then the current video frame is merged into the existing region node corresponding to the maximum similarity. If the maximum similarity is less than the difference between the node similarity threshold and the similarity boundary interval, then a new region node is created for the current video frame; Based on the regional nodes corresponding to the current video frame and the construction actions, the spatiotemporal connection relationship of the regional nodes is determined.

6. The method according to claim 1, characterized in that, The step of extracting the access features of the regional nodes based on the access relationships between regional nodes and construction action segments in the engineering topology diagram, and obtaining the node features of the regional nodes, includes: Determine the access relationship between the regional nodes in the engineering topology diagram and the construction action segments; Feature extraction is performed on the construction action segments in the access relationship corresponding to the regional nodes to obtain the access features of the regional nodes; The node characteristics of the regional nodes are obtained by averaging the access characteristics of the regional nodes.

7. The method according to claim 1, characterized in that, The process of determining a global representation of video data for a historical time period based on the node characteristics of the regional nodes, and predicting construction actions based on this global representation to obtain a sequence of construction actions for future time periods at the construction site includes: The node features of the region nodes are input into a graph convolutional neural network to obtain the node representation of the region nodes; The node representations of the region nodes are averaged to obtain a global representation of the video data for the historical time period. The global representation of the video data for the historical time period is input into a pre-trained classifier to predict construction actions, thereby obtaining the sequence of construction actions for future time periods at the construction site.

8. The method according to claim 7, characterized in that, Before inputting the global representation of the video data for the historical time period into a pre-trained classifier to predict construction actions and obtain the sequence of construction actions for future time periods at the construction site, the method further includes: Obtain the set of construction actions that the area node has accessed, and generate action interaction tags for the area node based on the set of construction actions; Image feature vectors are extracted from the access segments of the region nodes, and the image feature vectors are paired with the action interaction labels to predict the availability vectors of the region nodes. The action interaction labels and availability vectors of the regional nodes are used as training samples, and cross-loss entropy is used for training to obtain a classifier for outputting construction action sequences for future time periods.

9. The method according to claim 1, characterized in that, The step of obtaining the predicted engineering volume for the future time period based on the construction action sequence for the future time period and the pre-determined unit volume corresponding to the construction actions includes: Based on the sequence of construction actions for the future time period, determine the action type and number of times the construction actions will be performed; Based on the action type and execution frequency of the construction actions, as well as the pre-determined unit volume corresponding to the construction actions, the sequence of construction actions in the future time period is mapped into the engineering quantity, thereby obtaining the predicted engineering quantity of the project site in the future time period.

10. The method according to claim 1, characterized in that, After obtaining the predicted project volume for the future time period based on the construction action sequence for the future time period and the pre-determined unit volume corresponding to the construction actions, the method further includes: Based on the construction action segments of the historical time period at the project site and the pre-determined unit volume corresponding to the construction action, the completed work volume of the historical time period at the project site is obtained. Based on the predicted workload and the completed workload, engineering equipment resources are pre-scheduled to the target area nodes.

11. A video-based engineering quantity prediction device, characterized in that, The device includes: The data acquisition module is used to acquire video data from the construction site over a historical time period and extract video frame sequences from the video data; wherein the video frame sequence includes at least one construction action segment; A topology generation module is used to determine regional nodes and their spatiotemporal connections based on the video frame sequence, and to generate an engineering topology map of the engineering site. The feature extraction module is used to extract the access features of the regional nodes based on the access relationship between the regional nodes and the construction action segments in the engineering topology diagram, and obtain the node features of the regional nodes. The action prediction module is used to determine the global representation of video data for a historical time period based on the node characteristics of the regional nodes, and to predict construction actions based on the global representation of video data for a historical time period to obtain the construction action sequence for a future time period at the construction site. The project quantity prediction module is used to obtain the predicted project quantity for the future time period based on the construction action sequence for the future time period and the pre-determined unit volume corresponding to the construction action.

12. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the video-based quantity prediction method as described in any one of claims 1-10.

13. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video-based quantity prediction method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a program or instructions that, when executed by a processor, implement the steps of the video-based quantity prediction method as described in any one of claims 1 to 10.

Images