Intelligent construction site supervision method and system, program product and storage medium

Abstract

The invention provides an intelligent construction site supervision method and system, a program product and a storage medium, and relates to the technical field of construction site supervision. A plurality of IOT devices with a communication function can comprehensively collect environment and personnel identity data, video image data are processed and then personnel and device interaction time points are accurately positioned, and comprehensive data are constructed based on index association environment and identity data and then used for training an intelligent decision-making model. Thus, all technical features act synergistically, the situation of data isolation is broken, data redundancy is reduced, data integration and association are close, an accurate and organized data basis is provided for decision making, the accuracy and reliability of intelligent construction site supervision decision making are improved, decision making errors caused by data disorder are avoided, and the construction site management efficiency and quality are effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of construction site supervision, and in particular to a smart construction site supervision method, system, program product and storage medium. Background Art

[0002] In the wave of continuous innovation and development in the field of construction engineering, smart construction sites, as a core concept to improve the level of modernization of construction management, aim to optimize personnel deployment, equipment operation, environmental control and other aspects of the construction site through comprehensive, accurate and intelligent data collection and in-depth analysis, so as to achieve the goal of improving construction quality, strengthening safety assurance and shortening construction period.

[0003] However, related smart construction site supervision technologies: numerous sensors are scattered in various corners of the construction site, each independently collecting specific environmental parameter data and storing them in isolated databases.

[0004] Therefore, when relevant technologies make decisions based on these data, the data is too redundant, resulting in ineffective decision-making. Summary of the invention

[0005] The present application provides a smart construction site supervision method, system, program product and storage medium, which are used to break the data isolation, reduce data redundancy, and improve the accuracy and reliability of smart construction site supervision decisions.

[0006] In the first aspect, the present application provides a smart construction site supervision method, including: using multiple IOT devices with communication functions to collect environmental data of the construction site and identity information of construction workers; based on the video image data of the construction site, identifying the personnel and equipment in the video image data, and obtaining video frames with both personnel and equipment; for the video image data of the construction site, determining the type and location information of the equipment and personnel by performing target detection on the video frame, and then determining the time point when the equipment and personnel interact based on the time series information; using the time point as an index, associating the corresponding environmental data and the corresponding identity information to construct comprehensive data based on interaction events; using the comprehensive data as training data to train the intelligent decision-making model.

[0007] By adopting the above technical solutions, multiple IOT devices with communication functions can comprehensively collect environmental and personnel identity data. After processing, the video image data can accurately locate the time point of interaction between personnel and equipment. Based on this index, the environmental and identity data are associated to build comprehensive data, which is then used to train the intelligent decision-making model. In this way, the various technical features work together to break the data isolation situation, reduce data redundancy, and make data integration closely related, providing an accurate and organized data foundation for decision-making, improving the accuracy and reliability of smart construction site supervision decisions, avoiding decision-making errors due to data clutter, and effectively improving the efficiency and quality of construction site management.

[0008] In combination with some embodiments of the first aspect, in some embodiments, for the video image data of the construction site, after the step of determining the type and location information of the equipment and personnel by performing target detection on the video frames, and then determining the time point when the equipment and personnel interact based on the time series information, the method also includes: extracting personnel and equipment from the video frames using the time point as an index; using a behavior analysis algorithm to determine whether the extracted personnel and equipment both have behavioral actions for the image information of the extracted personnel and equipment, and the behavior analysis algorithm is used to identify the body movements of the personnel and the movement status of the components of the equipment; if the extracted personnel and equipment do not both have behavioral actions , it is determined that the extracted personnel and equipment do not interact at the time point; if the extracted personnel and equipment both have behavioral actions, it is determined whether the corresponding environmental data has changed; if the corresponding environmental data has not changed, it is determined that the extracted personnel and equipment do not interact at the time point; if the corresponding environmental data has changed, historical data is extracted, and the corresponding environmental data changes are matched and analyzed with the environmental data change patterns of similar personnel and equipment interaction scenarios in the historical data using a data matching algorithm; if the matching analysis is successful, the extracted personnel and equipment interact at the time point; if the matching analysis is unsuccessful, the extracted personnel and equipment do not interact at the time point.

[0009] By adopting the above technical solution, the image information of personnel and equipment is extracted with time points as indexes and behavioral analysis is performed. By combining the changes in environmental data with historical data matching, the complex interactive relationship between personnel and equipment can be deeply analyzed. The behavioral analysis algorithm identifies actions, and the data matching algorithm compares environmental changes. In the multi-person and multi-device interactive scene, the real interaction situation is effectively screened out, and the corresponding environment and identity information are accurately associated. The complex interactive association problem is overcome, so that the constructed comprehensive data is more in line with the actual construction, providing strong support for subsequent accurate decision-making, and ensuring the orderly and efficient construction.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the step of determining whether the corresponding environmental data has changed specifically includes: comparing the corresponding environmental data within a preset time window before and after a time point, and using data difference calculation and threshold judgment methods to determine whether the corresponding environmental data has changed.

[0011] By adopting the above technical solution and using data difference calculation and threshold judgment methods to compare the environmental data in the preset time window before and after a time point, it is possible to efficiently and accurately determine whether the environmental data has changed, thereby improving the accuracy of the entire smart construction site supervision system in judging the interaction situation, thereby optimizing resource allocation and construction arrangements.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the corresponding environmental data and the corresponding identity information are associated with each other based on the time point as an index, and the steps of constructing comprehensive data based on interaction events specifically include: tracing back the start and end time of the interaction forward and backward based on the time point to determine the interaction time interval; according to the interaction time interval, performing data retrieval on the environmental data to obtain the corresponding environmental data, and performing data retrieval on the identity information to obtain the corresponding identity information; and associating the corresponding environmental data and the corresponding identity information.

[0013] By adopting the above technical solution, the interaction time interval is determined by tracing the start and end time of the interaction based on the time point, and the associated environment and identity information is retrieved accordingly. Accurately defining the time interval can accurately locate the data range closely related to the interaction, avoid the mixing of irrelevant data, and make the associated environment and identity information more targeted. The close connection of each step makes the comprehensive data construction more systematic, enhances the data relevance, and provides a clear and definite data basis for intelligent decision-making.

[0014] In combination with some embodiments of the first aspect, in some embodiments, for the video image data of the construction site, the type and location information of the equipment and personnel are determined by performing target detection on the video frames, and then the time point when the equipment and personnel interact is determined based on the time series information. The steps specifically include: based on the video image data of the construction site, respectively extracting the comprehensive coordinates of the personnel and equipment, the comprehensive coordinates including the location coordinates and the time coordinates; when the time coordinates of the personnel and the equipment are consistent and the difference in the location coordinates is lower than the interaction distance threshold corresponding to the equipment, it is determined that there is an interaction between the personnel and the equipment.

[0015] By adopting the above technical solution, the interaction judgment is made more accurate by extracting the comprehensive coordinates of personnel and equipment and judging the interaction based on the time and location coordinates. The consistency of time coordinates ensures the synchronization of interaction, and the difference in location coordinates is compared with the interaction distance threshold to determine the rationality of the space. This coordinate-based judgment method is intuitive and accurate, effectively filtering out misjudged interactions, improving the accuracy of interaction recognition, and providing a reliable basis for subsequent data association and decision-making.

[0016] In combination with some embodiments of the first aspect, in some embodiments, based on the video image data of the construction site, the comprehensive coordinates of personnel and equipment are extracted respectively, and the comprehensive coordinates include the steps of position coordinates and time coordinates, specifically including: detecting personnel and equipment in the video frame, determining the image coordinates, and obtaining the timestamp of the video frame where the personnel and equipment are located as the time coordinate; converting the image coordinates into actual geographic coordinates based on the camera calibration parameters and the construction site geographic information to obtain the position coordinates; summarizing the position coordinates and time coordinates of the personnel and equipment to obtain the comprehensive coordinates.

[0017] By adopting the above technical solution, the video frame is detected to obtain the image coordinates and timestamps and converted into actual geographic coordinates to obtain comprehensive coordinates. The image coordinates and timestamps facilitate the positioning of personnel and equipment in the video data. After conversion into actual geographic coordinates, the data corresponds to the actual spatial position of the construction site, enhancing the practicality and intuitiveness of the data.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of associating corresponding environmental data and corresponding identity information based on time points as indexes and constructing comprehensive data based on interaction events, the method also includes: for the associated corresponding environmental data and corresponding identity information, combining the skill qualifications and equipment operation requirements in the identity information, judging whether the personnel have the ability to operate the equipment; if not, denying the rationality of the interaction, marking and storing the suspected abnormal interaction data.

[0019] By adopting the above technical solution, the operating ability of personnel can be judged in combination with the skill qualifications in the identity information and the equipment operation requirements, and abnormal interactions can be discovered in time. This judgment can be made after data association, which can eliminate potential risks caused by inconsistent personnel skills in advance and ensure the safe operation of equipment and construction quality. Suspected abnormal interaction data is marked and stored to facilitate subsequent traceability analysis, optimize personnel training and equipment management strategies, improve the safety and standardization of smart site management, and avoid accidents and delays caused by improper personnel operation.

[0020] In the second aspect, the present application provides a smart construction site supervision system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call computer instructions to enable the smart construction site supervision system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, the present application provides a computer program product comprising instructions, which, when executed on a smart construction site supervision system, enables the smart construction site supervision system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a smart construction site supervision system, enables the smart construction site supervision system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Multiple IOT devices with communication functions can comprehensively collect environmental and personnel identity data. After processing, the video image data can accurately locate the time point of interaction between personnel and equipment. Based on this index, the environmental and identity data are associated to build comprehensive data, which is then used to train the intelligent decision-making model. In this way, the synergy of various technical features breaks the data isolation situation, reduces data redundancy, and makes data integration closely related, providing an accurate and organized data foundation for decision-making, improving the accuracy and reliability of smart construction site supervision decisions, avoiding decision-making errors due to data clutter, and effectively improving the efficiency and quality of construction site management.

[0024] 2. Extracting personnel and equipment image information and performing behavior analysis based on time points, combined with environmental data changes and historical data matching, can deeply analyze the complex interaction between personnel and equipment. The behavior analysis algorithm identifies actions, and the data matching algorithm compares environmental changes. In multi-person and multi-device interaction scenarios, it effectively screens out real interactions, accurately associates corresponding environments and identity information, overcomes the problem of complex interaction association, and makes the constructed comprehensive data more in line with actual construction, providing strong support for subsequent accurate decision-making and ensuring orderly and efficient construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the smart construction site supervision method in the embodiment of the present application; Figure 2 is another flowchart of the smart construction site supervision method in an embodiment of the present application; Figure 3 It is a schematic diagram of an exemplary hardware structure of the smart construction site supervision system in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0027] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0028] See also Figure 1 , Figure 1 This is a flow chart of the smart construction site supervision method in the embodiment of the present application; S101, using multiple IoT devices with communication functions to collect environmental data of the construction site and identity information of construction workers; Among them, IOT devices with communication functions refer to devices built on the Internet of Things technology that can exchange data with other devices or systems, such as various sensors, smart terminals, etc., which are used to represent the source of information collected about the construction site. Environmental data refers to all data information on the construction site except for the data of construction workers, such as the operating parameters of various mechanical equipment, equipment failure information, equipment maintenance records, construction material data, construction area data, etc. These data can reflect the physical environment conditions of the construction site. The identity information of construction workers refers to data that can determine the personal identity and related attributes of construction workers.

[0029] In some embodiments, when construction activities begin at the construction site, these IOT devices with communication functions are activated. Various sensors and data acquisition terminals are reasonably arranged in different areas of the construction site, such as setting up equipment operation parameter collectors in the equipment room to obtain equipment data, installing inventory management sensors in the material warehouse to collect material data, and arranging regional information collection devices in key construction areas to collect regional data, etc., to collect environmental data. At the same time, an identity recognition device is set at the entrance of the construction site, such as a system combining a face recognition device and a work card reader. When construction personnel enter the construction site, their identity information is automatically collected and the data is transmitted to the data processing center through the network.

[0030] S102, identifying personnel and equipment in the video image data according to the video image data of the construction site, and obtaining a video frame having both personnel and equipment; Among them, the video image data of the construction site refers to the continuous image information captured and recorded by surveillance cameras installed at different locations on the construction site, which is used to represent the dynamic visual scene of the construction site.

[0031] In some embodiments, when the surveillance camera at the construction site completes the collection and storage of video image data for a period of time (such as a day or several hours), the data processing system will start the analysis process of these data. The video image data is read frame by frame in chronological order, and then target recognition is performed to filter out video frames that contain both personnel and equipment recognition results for subsequent interactive analysis.

[0032] S103, for the video image data of the construction site, determine the type and location information of the equipment and personnel by performing target detection on the video frames, and then determine the time point when the equipment and personnel interact based on the time series information; In some embodiments, after obtaining the video frames with both personnel and equipment filtered out by step S102, these video frames are processed frame by frame to deeply analyze the behavior and interaction relationship between personnel and equipment. The SSD (Single Shot Multi Box Detector) algorithm is used, which detects targets of different sizes based on feature maps of different scales. First, feature extraction is performed on the video frames, and multi-scale feature maps are obtained using basic convolutional network structures such as the VGG (Visual Geometry Group) network. Then, the preset default boxes are used on these feature maps for target matching and classification to determine the types and preliminary location information of personnel and equipment, and then the overlapping detection boxes are removed by the non-maximum suppression algorithm to obtain accurate location information. After that, time series information is constructed based on the timestamps of the video frames, and the position changes and action changes of personnel and equipment in the continuous time series are analyzed. For example, when the hand movements of personnel and the operating parts of the equipment show signs of contact within several consecutive frames and meet the specific interaction logic, the time point when the equipment and personnel interact is determined. In the whole process, the layout diagram of the construction site, the equipment operation specifications and other prior knowledge are also combined to make a reasonable judgment on the behavior of personnel and equipment to improve the accuracy of interaction judgment.

[0033] In some specific embodiments, step S103 specifically includes: S1031. Extracting the comprehensive coordinates of personnel and equipment respectively according to the video image data of the construction site, the comprehensive coordinates including position coordinates and time coordinates; In some embodiments, video frame images are read sequentially frame by frame from the stored video image data, and these images are used as the basic units for subsequent analysis. Then, for each video frame, the target detection algorithm in computer vision technology (such as a target detection model based on a convolutional neural network) is used to identify and locate the personnel and equipment therein, find the approximate area of ​​the personnel and equipment in the image, and then determine its image coordinates, and obtain the timestamp corresponding to the video frame at the same time, which is used as the time coordinate. Then, based on the calibration parameters of the camera and the pre-entered construction site geographic information, the image coordinates are converted into actual geographic coordinates through a specific coordinate conversion algorithm to obtain accurate position coordinates. Finally, the acquired position coordinates and time coordinates of the personnel and equipment are sorted and summarized to form complete comprehensive coordinate information, which provides key data support for the subsequent judgment of whether personnel and equipment interact.

[0034] In some embodiments, step S1031 specifically includes: S10311. Detect people and equipment in the video frame, determine the image coordinates, and obtain the timestamp of the video frame where the people and equipment are located as the time coordinate; In some embodiments, after acquiring the video image data, the video frame images are read frame by frame in chronological order and input into a pre-trained target detection model. This target detection model can be built based on a deep learning framework and has been fully trained using a large amount of image data labeled with construction site personnel and equipment. The model will perform feature extraction, classification, and positioning operations on the input video frame images, and output the location area where the personnel and equipment are located in the image in the form of a bounding box. The image coordinates are determined by calculating the coordinate values ​​of the upper left corner vertex or center point of the bounding box in the image coordinate system. At the same time, the timestamp information corresponding to the video frame is directly extracted from the metadata of the video image data and used as the time coordinate. For example, for a video frame showing a worker operating a crane, the target detection model outputs the bounding box coordinates (50,80) where the worker is located (here it is assumed to be the coordinates of the upper left corner vertex of the bounding box, in pixels) as the image coordinates, and the timestamp "2024-12-1411:00:00.000" of the video frame is used as the time coordinate, so that the coordinate information of the personnel and equipment at the spatiotemporal level is preliminarily obtained.

[0035] S10312, converting the image coordinates into actual geographic coordinates according to the camera calibration parameters and the construction site geographic information to obtain the position coordinates; In some embodiments, after obtaining the image coordinates of personnel and equipment in the video frame and the timestamp of the video frame, in order to accurately know their actual geographical location at the construction site, it is necessary to perform step S10312 for coordinate conversion. First, it is necessary to calibrate the camera at the construction site in advance, use professional calibration tools, shoot multiple sets of calibration plate images of known feature points, and use the correspondence between the actual coordinates of the feature points in the image (real coordinates pre-measured on the calibration plate) and the image coordinates after imaging to calculate the internal parameters (such as focal length, principal point coordinates, etc.) and external parameters (such as rotation angle, translation vector, etc.) of the camera, and store and record these parameters. Then, the geographic information of the construction site is obtained, usually by importing existing digital map data or constructing a geographic data set containing information such as the construction site boundary and key landmark point coordinates through on-site measurement. Then, according to the determined camera calibration parameters and the construction site geographic information, a coordinate conversion algorithm (such as an algorithm based on the perspective transformation principle) is used to convert the image coordinates (in pixels) of personnel and equipment into actual geographic coordinates (measured in actual length units) according to a certain mathematical transformation relationship.

[0036] S10313. Summarize the position coordinates and time coordinates of personnel and equipment to obtain comprehensive coordinates.

[0037] It can be seen that the image coordinates and timestamps are obtained by detecting the video frame and converted into actual geographic coordinates to obtain comprehensive coordinates. Image coordinates and timestamps facilitate the positioning of personnel and equipment in the video data. After conversion into actual geographic coordinates, the data corresponds to the actual spatial position of the construction site, enhancing the practicality and intuitiveness of the data.

[0038] S1032: When the time coordinates of the person and the device are consistent and the difference in the position coordinates is lower than the interaction distance threshold corresponding to the device, it is determined that the person and the device are interacting.

[0039] In some embodiments, first, the comprehensive coordinate information corresponding to all personnel and equipment will be traversed, and the combination of each group of personnel and equipment will be judged. For the time coordinate, directly compare the timestamp strings of the two (strictly compare according to the time format) to see if they are completely consistent. If not, it is directly determined that the group of personnel and equipment does not have any interaction behavior at the current moment; if the time coordinates are consistent, then calculate the difference between the position coordinates of the two. For the two-dimensional coordinate situation, the spatial distance between the two is obtained by calculating the square root of the sum of the squares of the horizontal and vertical coordinate differences (i.e., the Euclidean distance calculation formula). For the three-dimensional coordinate situation, the calculation formula is expanded accordingly. Then, the interaction distance threshold corresponding to the device is obtained. This threshold is usually stored in a configuration file or database table. Its value is determined comprehensively based on the specific type of the equipment, operating specifications, and past construction experience. For example, different types of cranes, different power welding machines, etc. have their own adaptive interaction distance thresholds. Finally, the calculated position coordinate difference is compared with the corresponding interaction distance threshold. If the difference is less than the threshold, it is determined that this group of people and equipment have interacted at the corresponding time point, and the relevant interaction information (such as personnel identity, equipment number, interaction time, location coordinates, etc.) is recorded for further analysis and utilization; if the difference is greater than the threshold, it is determined that there is no interaction between the two at that time point, and the next group of people and equipment combinations will continue to be judged until all combinations are traversed.

[0040] It can be seen that by extracting the comprehensive coordinates of personnel and equipment and judging the interaction based on the time and location coordinates, the interaction judgment is made more accurate. The consistency of the time coordinates ensures the synchronization of the interaction, and the difference in the location coordinates is compared with the interaction distance threshold to determine the rationality of the space. This coordinate-based judgment method is intuitive and accurate, effectively filtering out misjudged interactions, improving the accuracy of interaction recognition, and providing a reliable basis for subsequent data association and decision-making.

[0041] S104, using the time point as an index, associating the corresponding environment data with the corresponding identity information to construct comprehensive data based on the interaction event; Among them, the time point as index refers to the use of the previously determined time point of the interaction between personnel and equipment as a key identifier, which is used to quickly locate and filter data information related to the interaction event in the data. The corresponding environmental data refers to all data information containing equipment tags collected near the time point of the interaction between personnel and equipment; the corresponding identity information refers to the identity attribute data of the construction workers involved in the interaction event, including their type of work, skill level and other information. The comprehensive data based on the interaction event refers to a set of data formed by integrating the environmental data, identity information data and other relevant data (such as interaction duration, interaction frequency, etc.) associated with the interaction event between personnel and equipment, which is used for subsequent analysis and decision-making.

[0042] In some embodiments, after determining the time point of interaction between personnel and equipment, the data processing system will perform data retrieval and association operations in the data repository with this time point as the core. First, determine a time range, for example, with the interaction time point as the center, the environmental data within a few minutes before and after is used as the corresponding environmental data, and extract the equipment operating parameters, material usage data, regional spatial data, etc. within this time period from the environmental data repository. Then, retrieve the identity information of the personnel participating in the interaction at this interaction time point from the personnel information database, including their name, work number, type of work, skill level, etc. Then, integrate these corresponding environmental data and identity information, and further calculate some auxiliary information, such as the operation time of the personnel on the equipment, the changes in the operating parameters of the equipment during this time period, etc., to construct comprehensive data based on interaction events, so as to provide comprehensive data support for subsequent intelligent decision-making.

[0043] In some specific embodiments, step S104 specifically includes: S1041, tracing back the start and end time of the interaction based on the time point, and determining and defining the interaction time interval; The start and end time of the interaction refers to the earliest time when people and devices start meaningful interaction and the latest time when they end the interaction, which are used to define the start and end boundaries of the entire interaction process in the time dimension.

[0044] In some embodiments, starting from the video image data, with the interaction time point as the center, check the changes in the actions of people and equipment in the video frames before this time. When it is found that the person begins to make preparatory actions related to the interaction with the equipment (such as walking towards the equipment, adjusting their own posture, etc.) and the corresponding parts of the equipment also begin to have corresponding response actions, this time is marked as the start time; similarly, check the video frames after the interaction time point until the person and the equipment complete the interaction action and no longer have direct related behaviors with each other (such as workers leaving the equipment operation area, the equipment stops the action caused by the interaction, etc.), and this time is determined as the end time. Finally, the interval formed by the start time and the end time is determined as the defined interaction time interval, and further retrieves relevant data information based on this.

[0045] S1042, according to the interaction time interval, performing data retrieval on the environment data to obtain corresponding environment data, and performing data retrieval on the identity information to obtain corresponding identity information; In some embodiments, after determining the interaction time interval, the database storing environmental data is connected, and a query statement is constructed according to the interaction time interval to extract all environmental data records within the time interval from the database. These records contain specific values ​​of various environmental data indicators at different times. At the same time, the database storing personnel identity information is connected, and a query statement is also constructed based on the interaction time interval and auxiliary conditions such as the construction area that may be involved to find the identity information records of the personnel who participated in the interaction within the time period, so as to prepare for subsequent data association and integration.

[0046] S1043. Associate the corresponding environmental data with the corresponding identity information.

[0047] It can be seen that the interaction time interval is determined by tracing the start and end time of the interaction based on the time point, and the associated environment and identity information is retrieved accordingly. Accurately defining the time interval can accurately locate the data range closely related to the interaction, avoid the mixing of irrelevant data, and make the associated environment and identity information more targeted. The close connection of each step makes the comprehensive data construction more systematic, enhances the data relevance, and provides a clear and definite data basis for intelligent decision-making.

[0048] S105. Use the comprehensive data as training data to train the intelligent decision-making model.

[0049] It can be seen that multiple IOT devices with communication functions can comprehensively collect environmental and personnel identity data. After processing, the video image data can accurately locate the time point of interaction between personnel and equipment. Based on this index, the environment and identity data are associated to build comprehensive data, which is then used to train the intelligent decision-making model. In this way, the synergy of various technical features breaks the situation of data isolation, reduces data redundancy, and makes data integration closely related, providing an accurate and organized data basis for decision-making, improving the accuracy and reliability of smart construction site supervision decisions, avoiding decision-making errors due to data clutter, and effectively improving the efficiency and quality of construction site management.

[0050] In actual use, the interaction scenarios between construction site personnel and equipment are highly complex. It is not a simple one-to-one correspondence mode under ideal conditions, but it is common for multiple people to operate a piece of equipment at the same time. At the same time, it is not uncommon for one person to interact with multiple devices in a short period of time. This complex interaction situation makes it extremely difficult to judge whether personnel and equipment are interacting simply based on conventional methods. The judgment method used previously may not be able to accurately determine its interaction status. It may be misjudged as no interaction, but in fact some key interaction actions are not recognized; or it may be judged as interaction, but there are invalid or non-substantial interaction behaviors; or in the complex interweaving scenarios of multiple people and multiple devices, it is difficult to clarify the real and effective interaction relationship between each person and the equipment, which leads to deviations and errors in the subsequent data association and decision-making based on these inaccurate interaction judgments, and cannot effectively meet the requirements of smart construction sites for supervision and efficient decision-making.

[0051] Therefore, in some embodiments, step S103 further includes: See also Figure 2 , Figure 2 is another flowchart of the smart construction site supervision method in an embodiment of the present application; S201, extracting people and equipment from video frames using time points as indexes; In some embodiments, this step is executed after the operation of determining the time point of interaction between personnel and equipment based on video frames is completed in step S103. The data processing system will obtain the information of each previously determined interaction time point, and then traverse all video frames in the stored video image data. For each time point, a smaller time window range is set (for example, video frames 1-2 seconds before and after the time point are centered), and the video frames falling within this time window are filtered out through the timestamp marks corresponding to the video frames. These filtered video frames are video frames containing image information related to personnel and equipment extracted with the time point as the index, and they constitute the basic data set for subsequent in-depth analysis of interactive behaviors.

[0052] S202, using a behavior analysis algorithm to determine whether the extracted personnel and equipment both have behavioral actions, with respect to the image information of the extracted personnel and equipment, wherein the behavior analysis algorithm is used to identify the body movements of personnel and the movement status of components of equipment; Among them, the behavioral analysis algorithm refers to a set of logical and computational rules built based on computer vision, machine learning and other technologies. It aims to identify people's body movements (such as raising hands, bending over, walking, etc.) and the movement status of equipment components (such as moving a certain operating lever of the equipment, extending the robotic arm, etc.) through the analysis of image information. It is a key tool for analyzing whether there is actual interaction between people and equipment.

[0053] In some embodiments, after the corresponding video frames are extracted after completing step S201, each extracted video frame containing the image information of personnel and equipment is processed. The video frames are input into a pre-trained behavior analysis algorithm model, which can be a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN) based on deep learning. First, CNN is used to extract the features of the personnel and equipment in the image. For example, for personnel, the key feature points of various parts of the body can be identified, and for equipment, the shape and position features of its key components can be obtained. Then, RNN is used to perform a time series analysis of these feature changes in continuous video frames, so as to determine whether the personnel have changes in body movements and whether the components of the equipment have corresponding changes in motion states, and finally determine whether both the personnel and the equipment have behavioral actions.

[0054] S203: If the extracted personnel and equipment do not all have behavioral actions, it is determined that the extracted personnel and equipment do not interact at the time point; In some embodiments, when step S202 completes the judgment on whether there are behavioral actions between personnel and equipment, corresponding judgment result data will be obtained. If the judgment result is that there is no behavioral action, that is, it indicates that the personnel and equipment do not both have behavioral actions, then it is directly determined that there is no interactive behavior between the personnel and the equipment at this corresponding time point. This judgment logic is based on the following considerations: real and effective interaction between personnel and equipment is often accompanied by changes in the actions of both parties. If one or both parties do not take any action, it is difficult to determine that there is a substantial interaction, so it is directly determined that there is no interaction to avoid subsequent erroneous data association and analysis.

[0055] S204: If both the extracted personnel and equipment have behavioral actions, determine whether the corresponding environmental data has changed; In some embodiments, after step S202 determines that both personnel and equipment have behavioral actions, it is necessary to further examine environmental factors. First, all environmental data records within a preset time interval before and after the time point are obtained from the environmental data repository. Then, for different types of environmental data, the corresponding data difference calculation method is adopted. For example, for numerical environmental data (such as temperature, humidity, etc.), the absolute value of the difference between the corresponding data before and after the time point is calculated. For categorical environmental data (such as different functional divisions of the construction area, etc.), it is determined whether there is a category change. Through these specific calculation and comparison operations, it is determined whether there is a change in the environmental data, which provides more basis for more accurate subsequent judgment of whether personnel and equipment are actually interacting.

[0056] In some specific embodiments, step S204 specifically includes: comparing the corresponding environmental data in a preset time window before and after the time point, and using data difference calculation and threshold judgment methods to determine whether the corresponding environmental data has changed.

[0057] In some embodiments, the duration of the preset time window is reasonably set according to factors such as the characteristics of the construction scene and past experience. For example, for some large-scale equipment that operates relatively stably and has a slow impact on the environment, the time window can be set slightly longer; while for small-scale equipment that operates relatively quickly and provides timely environmental feedback. Then, all records of the corresponding environmental data in the time window are extracted from the environmental data repository. These records contain the specific values ​​of various environmental data indicators at different times. Then, for the same environmental data indicator, the data value after the time point and the data value before the time point are calculated in turn to obtain a series of difference results. Then determine the appropriate threshold value based on the construction type, equipment characteristics, etc. Finally, compare the calculated difference with the corresponding threshold value to determine whether the corresponding environmental data has changed. If the difference of a certain environmental data indicator exceeds its corresponding threshold value, it is determined that the environmental data has changed near this time point.

[0058] It can be seen that by using data difference calculation and threshold judgment methods to compare environmental data in the preset time window before and after a time point, it is possible to efficiently and accurately determine whether the environmental data has changed, thereby improving the accuracy of the entire smart construction site supervision system in judging interaction situations, thereby optimizing resource allocation and construction arrangements.

[0059] S205, if the corresponding environmental data does not change, it is determined that the extracted personnel and equipment do not interact at the time point; In some embodiments, when it is determined through comparison that the corresponding environmental data has not changed, an interaction judgment is made based on this result. Because under normal circumstances, if there is real and effective interaction between personnel and equipment, it will often be reflected in the environmental data. For example, the operation of the equipment will generate heat to change the surrounding temperature, and the operation of the equipment by personnel may cause the surrounding air flow to change the humidity. When the environmental data does not change, it means that the behavior of the personnel and the equipment is likely to occur independently, and there is no interactive connection that will cause environmental changes. Therefore, at this time, it will be determined that there is no interaction between personnel and equipment at this point in time, and this judgment result will be recorded and integrated with the previous time points, personnel and equipment related information, etc., so as to facilitate the subsequent overall analysis of the interaction situation and the data association, accurately exclude such situations that seem to have behavioral actions but actually have no interaction, and avoid interference with the overall interaction judgment.

[0060] S206. If the corresponding environmental data has changed, extract the historical data, and use the data matching algorithm to match and analyze the corresponding environmental data change situation with the environmental data change pattern in the historical data under the same personnel and equipment interaction scenario; In some embodiments, when it is determined that the corresponding environmental data has changed, historical data is extracted and matched and analyzed. First, the data processing system will build query conditions in the historical data repository based on key information such as the current personnel's job type, the type of equipment being operated, the construction area in which it is located, and the current approximate time range (such as daytime construction period or nighttime construction period, etc.), and filter out environmental data record sets in the same personnel and equipment interaction scenarios that meet these conditions from the massive historical data through the database query statement. Then, the current environmental data change situation (such as how much the temperature has increased, how much the humidity has changed, etc.) is organized into a data vector form, and the environmental data change pattern in the historical data record set is also vectorized for subsequent algorithm analysis. Then, the selected data matching algorithm (such as the cosine similarity algorithm) is used to calculate the similarity value between the current environmental data change vector and the environmental data change vector corresponding to each record in the historical data, and a set of similarity results is obtained.

[0061] S207, if the matching analysis is successful, the extracted person and device interact at the time point; In some embodiments, based on multiple factors such as the previous behavioral actions of both personnel and equipment and the changes in environmental data that conform to the interaction mode, it is logically determined that at this specific point in time, meaningful interaction has indeed occurred between personnel and equipment. For example, workers' operation of the equipment has caused changes in environmental data, and this change is similar to previous similar interaction scenarios, so it is determined that interaction has occurred.

[0062] In some embodiments, although the personnel and equipment have previously taken actions and the environmental data has changed, since the change pattern is quite different from the typical pattern in the interaction scenario, it is logically determined that no meaningful interaction has occurred between the personnel and the equipment at this specific time point. For example, changes in environmental data may be caused by other external factors (such as sudden weather changes, other non-related construction activities nearby, etc.), rather than interactions between personnel and equipment, so it is determined that there is no interaction.

[0063] It can be seen that by extracting personnel and equipment image information and performing behavior analysis based on time points, and combining environmental data changes with historical data matching, we can deeply analyze the complex interaction relationship between personnel and equipment. The behavior analysis algorithm identifies actions, and the data matching algorithm compares environmental changes. In multi-person and multi-device interaction scenarios, it effectively screens out real interactions, accurately associates corresponding environments and identity information, overcomes the problem of complex interaction association, and makes the constructed comprehensive data more in line with actual construction, providing strong support for subsequent accurate decision-making and ensuring orderly and efficient construction.

[0064] It can be seen that by extracting personnel and equipment image information and performing behavior analysis based on time points, and combining environmental data changes with historical data matching, we can deeply analyze the complex interaction relationship between personnel and equipment. The behavior analysis algorithm identifies actions, and the data matching algorithm compares environmental changes. In multi-person and multi-device interaction scenarios, it effectively screens out real interactions, accurately associates corresponding environments and identity information, overcomes the problem of complex interaction association, and makes the constructed comprehensive data more in line with actual construction, providing strong support for subsequent accurate decision-making and ensuring orderly and efficient construction.

[0065] In some embodiments, after step S104, the method further includes: S1041. For the associated corresponding environment data and corresponding identity information, judging whether the personnel have the ability to operate the equipment in combination with the skill qualification and equipment operation requirements in the identity information; In some embodiments, after the association of environmental data and identity information has been completed, in order to analyze the rationality of the interaction behavior more deeply, step S1041 will be executed. First, the skill qualification related content of the personnel is extracted from the associated identity information, which may involve checking the personnel skill certificate number, the skill range corresponding to the type of work, and the skill level and other detailed information. For example, by querying the skill qualification record of the personnel in the personnel information management system, the electrician skill level is obtained as intermediate, and the electrician operation skill range and proficiency of the personnel are known. Then, the operation requirements of the equipment related to this interaction are obtained. These requirements are usually stored in the equipment management database. According to the model, category and other information of the equipment, the corresponding operation specification document is found, which records in detail the qualification certificates, professional skills and physical conditions required to operate the equipment. For example, it is judged whether the certificate held by the personnel is consistent with the certificate type required by the equipment, whether the personnel's skill proficiency can cope with the complexity of the equipment operation, etc., and finally the judgment result of whether the personnel has the ability to operate the equipment is obtained, which provides a basis for the subsequent judgment of the rationality of the interaction.

[0066] S1042. If not, the rationality of the interaction is denied, and the suspected abnormal interaction data is marked and stored.

[0067] In some embodiments, the content to be marked is determined, that is, all relevant data information involved in this interactive behavior is determined as suspected abnormal interactive data, including various specific data extracted from previously associated environmental data and identity information, such as the operating parameters of the equipment at that time (such as the lifting weight of the crane, the angle of the crane arm, etc.), the environmental data of the construction area (such as temperature, humidity, etc.), the identity information of the participants in the interaction (such as work number, type of work, skill level, etc.) and the time point of the interaction, etc., and these data are organized into a data set or data record. Then, a suitable marking method is selected, such as adding a new field in the data record to mark the data as abnormal interactive data, or adding a specific identification string to the data file name to distinguish it as an abnormal data file. Next, the storage location is determined to facilitate subsequent viewing, analysis, and as a basis for improving construction management measures.

[0068] It can be seen that by combining the skill qualifications in the identity information with the equipment operation requirements to judge the operating ability of personnel, abnormal interactions can be discovered in time. This judgment can be made after data association to eliminate potential risks caused by inconsistent personnel skills in advance and ensure the safe operation of equipment and construction quality. Marking and storing suspected abnormal interaction data facilitates subsequent traceability analysis, optimizes personnel training and equipment management strategies, improves the safety and standardization of smart site management, and avoids accidents and delays caused by improper personnel operation.

[0069] The following introduces an exemplary smart construction site supervision system 300 provided in an embodiment of the present application. Figure 3 It is a schematic diagram of an exemplary hardware structure of the smart construction site supervision system 300 provided in an embodiment of the present application.

[0070] In some embodiments, the smart construction site supervision system 300 is a computer device or the smart construction site supervision system 300 includes a computer device. The computer device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, the method in the embodiment of the present application is implemented.

[0071] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0072] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0073] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0074] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0075] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A smart construction site supervision method, characterized in that: include: Use multiple IoT devices with communication functions to collect environmental data on the construction site and identity information of construction workers; According to the video image data of the construction site, personnel and equipment in the video image data are identified to obtain a video frame having both personnel and equipment; For the video image data of the construction site, the type and location information of the equipment and personnel are determined by performing target detection on the video frames, and then the time point when the equipment and personnel interact is determined based on the time series information; According to the time point as an index, the corresponding environment data and the corresponding identity information are associated to construct comprehensive data based on the interaction event; The comprehensive data is used as training data to train the intelligent decision-making model.

2. The method according to claim 1, characterized in that After the step of determining the type and location information of the equipment and personnel by performing target detection on the video frames for the video image data of the construction site, and then determining the time point when the equipment and personnel interact based on the time series information, the method further includes: Using the time point as an index, extracting people and equipment from the video frame; For the extracted image information of personnel and equipment, a behavior analysis algorithm is used to determine whether the extracted personnel and equipment both have behavioral actions, wherein the behavior analysis algorithm is used to identify the body movements of personnel and the movement status of components of equipment; If the extracted person and device do not both have behavioral actions, it is determined that the extracted person and device do not interact at the time point; If both the extracted personnel and equipment have behavioral actions, determine whether the corresponding environmental data has changed; If the corresponding environmental data does not change, it is determined that the extracted person and the device do not interact at the time point; If the corresponding environmental data has changed, historical data is extracted, and the corresponding environmental data changes are matched and analyzed with the environmental data change patterns in the historical data under similar human-device interaction scenarios using a data matching algorithm; If the matching analysis is successful, the extracted person and device interacted at the said time point; If the matching analysis is unsuccessful, the extracted person and device do not interact at the said point in time.

3. The method according to claim 2, characterized in that The step of determining whether the corresponding environmental data has changed specifically includes: By comparing the corresponding environmental data in a preset time window before and after the time point, it is determined whether the corresponding environmental data has changed by using data difference calculation and threshold judgment method.

4. The method according to claim 1, characterized in that: The step of associating the corresponding environment data with the corresponding identity information based on the time point as an index to construct comprehensive data based on the interaction event specifically includes: Based on the time point, trace back the start and end time of the interaction to determine the defined interaction time interval; According to the interaction time interval, data retrieval is performed on the environment data to obtain corresponding environment data, and data retrieval is performed on the identity information to obtain corresponding identity information; Corresponding environmental data and corresponding identity information are associated.

5. The method according to claim 1, characterized in that The steps of determining the type and location information of equipment and personnel by performing target detection on video frames for the video image data of the construction site, and then determining the time point when the equipment and personnel interact based on the time series information specifically include: Extracting the comprehensive coordinates of personnel and equipment respectively according to the video image data of the construction site, wherein the comprehensive coordinates include position coordinates and time coordinates; When the time coordinates of the person and the device are consistent, and the difference in the position coordinates is lower than the interaction distance threshold corresponding to the device, it is determined that the person and the device are interacting.

6. The method according to claim 5, characterized in that The step of extracting the comprehensive coordinates of personnel and equipment respectively according to the video image data of the construction site, wherein the comprehensive coordinates include position coordinates and time coordinates, specifically comprises: Detecting people and equipment in the video frame, determining image coordinates, and obtaining timestamps of the video frames where the people and equipment are located as the time coordinates; According to the camera calibration parameters and the construction site geographic information, the image coordinates are converted into actual geographic coordinates to obtain the position coordinates; The position coordinates and the time coordinates of the personnel and equipment are aggregated to obtain comprehensive coordinates.

7. The method according to claim 1, characterized in that After the step of associating the corresponding environment data with the corresponding identity information based on the time point as an index to construct comprehensive data based on the interaction event, the method further includes: For the associated corresponding environmental data and corresponding identity information, combine the skill qualifications in the identity information with the equipment operation requirements to determine whether the personnel have the ability to operate the equipment; If not, the rationality of the interaction is denied, and the suspected abnormal interaction data is marked and stored.

8. A smart construction site supervision system, characterized in that: The smart construction site supervision system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the smart construction site supervision system to execute the method described in any one of claims 1-7.

9. A computer program product comprising instructions, characterized in that When the computer program product runs on a smart construction site supervision system, the smart construction site supervision system executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the smart construction site supervision system, the smart construction site supervision system executes the method as described in any one of claims 1-7.

Images