Construction operation monitoring system based on eye tracking

Abstract

The invention provides a construction operation monitoring system based on eye tracking, which comprises a sight line acquisition assembly, the sight line acquisition assembly comprises an auxiliary mechanism and an eye tracking device, the auxiliary mechanism captures an eye image of an operator in the eye tracking device, and obtains sight line data by using an image algorithm to realize real-time monitoring of eye movement; the environment acquisition assembly is used for acquiring environment parameters in construction so as to form environment data; and the monitoring model evaluates the safety risk and the operation efficiency of the operation of the operator by using a learning algorithm through the input environmental parameters and the sight line data, and guides the operator to carry out correct operation steps when the safety risk exists. According to the invention, real-time monitoring and real-time feedback of worker operation are realized, so that the safety and working efficiency of a construction site are remarkably improved, safety accidents are reduced, workers are guided to optimize operation, invalid labor is reduced, and the skill proficiency of new workers is accelerated.

Description

Technical Field

[0001] The invention relates to the technical field of monitoring in construction operations, and in particular to a construction operation monitoring system based on eye tracking in a mine. Background Art

[0002] The existing technology in the field of construction site monitoring mainly lacks real-time monitoring and personalized feedback of individual workers' operations. The current system often only focuses on improving safety or efficiency, and fails to comprehensively analyze line of sight data and on-site environment, thus failing to improve safety and work efficiency at the same time. In the existing technology, the following problems exist in construction monitoring: On the one hand, although video surveillance systems can provide comprehensive on-site monitoring, they usually cannot provide sufficient real-time data analysis and personalized feedback to support workers' immediate decision-making and operational improvements. In addition, these systems often require a lot of manpower for data analysis and supervision, increasing operating costs.

[0003] On the other hand, gaze tracking technology has demonstrated its superiority in other fields, especially in understanding user intent and improving system interactivity. However, the application of this technology in construction sites has not been fully explored. Existing gaze tracking solutions are mainly focused on consumer electronics and medical devices, and their design is not considered for the complex and ever-changing working environment of construction sites. Summary of the invention

[0004] The present invention aims to realize real-time monitoring and instant feedback of workers' operations by integrating eye tracking and augmented reality technology, thereby significantly improving the safety and work efficiency of construction sites, reducing safety accidents, guiding workers to optimize operations, reducing ineffective labor, and accelerating the skill proficiency of new workers.

[0005] In order to solve the above technical problems, the technical solutions provided by the present invention are as follows: The construction operation monitoring system based on eye tracking includes: A sight acquisition component, the sight acquisition component includes an auxiliary mechanism and a gaze tracking device, the auxiliary mechanism captures the eye image of the operator in the gaze tracking device, and uses an image algorithm to obtain sight data to achieve real-time eye movement monitoring; An environment collection component, which is used to obtain environmental parameters during construction to form environmental data; The monitoring model uses input environmental parameters and line of sight data and a learning algorithm to evaluate the safety risks and operating efficiency of the operator's operation, and guides the operator to perform correct operating steps when there is a safety risk.

[0006] Furthermore, the auxiliary mechanism includes an infrared LED lamp and an infrared camera located outside the gaze tracking device, and the infrared camera is used to collect infrared light reflected by the eye to generate a continuous eye image.

[0007] Furthermore, the image algorithm specifically uses the algorithm to identify the pupil center and its reflection point in the preprocessed eye image and calculate the line of sight direction.

[0008] Furthermore, the monitoring model includes a feature extraction unit, which is used to extract the gaze point, gaze duration and gaze frequency of the operator's line of sight.

[0009] Furthermore, it also includes training optimization of the monitoring model, specifically: using labeled historical data to train the convolutional neural network model, and optimizing the model parameters through the back propagation algorithm and gradient descent method to improve the accuracy of risk assessment.

[0010] Furthermore, the monitoring module includes a multimodal real-time feedback device, specifically including visual feedback, auditory feedback and tactile feedback, and the visual feedback is achieved by a remotely controlled gaze tracking device.

[0011] Furthermore, the auditory feedback is realized by voice prompting, and the tactile feedback is specifically to control the eye tracking device so that it acts on the operator's touch to provide feedback.

[0012] Furthermore, it also includes a positioning system, which specifically uses an inertial measurement unit to supplement the positioning information obtained by GPS to obtain the positioning of the operator.

[0013] Furthermore, it also includes the construction of a dynamic three-dimensional model, specifically, by using three-dimensional scanning and photogrammetry technology to update the positioning data in real time and build a three-dimensional model of the construction site to prompt operators with component information at corresponding locations.

[0014] Furthermore, it also includes a collaborative working unit, which enables the eye tracking devices of several operators to work collaboratively through a communication mechanism, a location sharing protocol and a collaborative task execution mechanism.

[0015] The present invention has the following beneficial effects: Through real-time monitoring and intelligent analysis, the central processing unit of the present invention can timely discover and prevent possible safety accidents, while improving work efficiency.

[0016] In the present invention, the personalized feedback mechanism helps workers to quickly adapt to work according to their own operating habits and on-site environment, reducing training costs and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a framework diagram of the construction operation monitoring system based on eye tracking provided by the present invention. DETAILED DESCRIPTION

[0018] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0019] Refer to the attached Figure 1 As shown, the solution in the present invention should be used in a certain construction project, where a worker is performing high-altitude operations at the construction site. For example, by analyzing its line of sight data and environmental data, the system finds that the worker frequently looks at unstable areas, and the duration exceeds the safety threshold. At this time, relevant reminders can be made. The processing process at this time is as follows: the central processing unit immediately activates the early warning mechanism, displays a red warning sign in the worker's field of view through the AR device, and reminds the worker to pay attention to safety through headphones. The entire feedback optimization is: the system records the worker's response time and subsequent operations, combines risk scoring, and adjusts the feedback strategy to ensure that potential risks can be more accurately predicted and prevented in the future.

[0020] Through the implementation details of the above-mentioned central processing unit, the present invention can provide a construction operation monitoring system based on eye tracking for the construction site, which not only improves safety but also improves work efficiency through personalized operation guidance.

[0021] Specifically, it includes the following systems: a sight line acquisition component, the sight line acquisition component includes an auxiliary mechanism and a sight line tracking device, the auxiliary mechanism captures the eye image of the operator in the sight line tracking device, and uses an image algorithm to obtain sight line data to achieve real-time eye movement monitoring; An environment collection component, which is used to obtain environmental parameters during construction to form environmental data; The monitoring model uses input environmental parameters and line of sight data and a learning algorithm to evaluate the safety risks and operating efficiency of the operator's operation, and guides the operator to perform correct operating steps when there is a safety risk.

[0022] Using the device of the present invention, it includes the following steps: Step 1: Collect the sight line collection components.

[0023] The line of sight acquisition component in this embodiment uses an infrared LED light and a high-sensitivity infrared camera to capture eye images, calculates the line of sight direction through an image processing algorithm, and realizes real-time eye movement monitoring.

[0024] In the present invention, the eye tracking technology forms an auxiliary mechanism with an infrared lighting module, specifically, multiple infrared LED lights are arranged around the monitoring area to ensure that the worker's eyes can be evenly illuminated under variable lighting conditions, thereby obtaining a clear eye image. The addition of auxiliary mechanisms can also improve the quality of eye images and provide reliable data for subsequent line of sight direction calculations. It is preferred to use an infrared camera with a high frame rate (at least 120 frames per second) to capture the infrared light reflected by the eye, generate continuous eye images, and then capture a sufficient amount of image data to provide a basis for real-time analysis.

[0025] The sight line acquisition component in this embodiment follows the following steps: Preprocessing stage: Perform denoising, contrast enhancement and other operations on eye images to improve image quality.

[0026] Feature extraction stage: Use algorithms to identify the pupil center and its reflection point in the eye image and calculate the line of sight direction.

[0027] The mathematical formula at this time is: sight direction = f (pupil center coordinates, reflection point coordinates) where the function (f) represents the sight direction calculation algorithm, which calculates the sight direction based on the coordinates of the pupil center and the reflection point.

[0028] The line of sight direction can then be accurately calculated, providing real-time feedback for subsequent operations.

[0029] Line of sight data application module: The calculated line of sight data is combined with the on-site environmental data to evaluate the safety and efficiency of workers' operations. Through real-time monitoring and evaluation, workers are provided with immediate feedback to guide their operations.

[0030] For example, at a construction site, workers need to operate heavy machinery in accordance with safety regulations. Through eye tracking technology, the system monitors in real time whether the worker's line of sight deviates from the operating area. If the worker's line of sight leaves the operating area for more than a preset time threshold, the system will determine it as a potential safety risk. At this time, the system immediately displays a warning sign in the worker's field of vision through an AR headset and issues an auditory warning through headphones to prompt the worker to concentrate. The system also records the time of occurrence, duration, and response of each warning, which will be used for subsequent analysis of workers' operating habits and optimization of safety warning strategies.

[0031] Through the above implementation details and steps, the eye tracking technology of the present invention can provide efficient and accurate eye movement monitoring for construction sites, thereby significantly improving work safety and efficiency.

[0032] Step 2: Processing by the central processing unit containing the monitoring model.

[0033] ) Data processing: The central processing unit receives and integrates eye movement data from the image capture and analysis module as well as on-site environmental data, such as worker location, operating status of surrounding equipment, and construction site layout.

[0034] 2) Algorithm application: Use machine learning algorithms, especially convolutional neural networks (CNNs), to conduct in-depth analysis of data to identify safety risks and efficiency bottlenecks in the operation process.

[0035] Formula: Risk score = CNN (line of sight data, environmental data) CNN stands for the convolutional neural network model, which takes line of sight data and environmental data as input and outputs a risk score, which reflects the degree of security risk of the current operation.

[0036] 3) Feedback: Based on the risk assessment results, the system provides workers with customized operational suggestions and early warning information, aiming to guide workers to take correct operational steps and avoid potential risks.

[0037] Step 3: Construction of monitoring model.

[0038] In this embodiment, the detection model uses a CNN model to process and analyze data, and generates risk assessments and operational recommendations through data preprocessing, feature extraction, model training and optimization.

[0039] In this embodiment, the detection model is constructed as follows: 3.1 Data preprocessing: The collected raw eye movement data and environmental data are first cleaned to remove outliers and noise.

[0040] Normalize the data to ensure that the data is analyzed at the same scale, improving the efficiency and accuracy of model training.

[0041] 3.2 Feature extraction: The CNN model is used to automatically extract key features from the preprocessed data, including the worker's gaze point, gaze duration, gaze frequency, etc.

[0042] Through its convolutional and pooling layer structures, the CNN model is able to identify complex data patterns and provide support for subsequent risk assessment.

[0043] Mathematical formula: Feature=CNNconv(Xnorm) where CNNconv represents the convolutional layer operation in the convolutional neural network, and Feature is the extracted feature vector.

[0044] 3.3 Model training and optimization: The CNN model is trained using labeled historical data, which includes cases of safe and risky operations. The model parameters are optimized through the back-propagation algorithm and gradient descent method to improve the accuracy of risk assessment.

[0045] Mathematical formula: Among them, θ represents the model parameters, α is the learning rate, J(θ) is the loss function, is the gradient of the loss function with respect to the parameters.

[0046] 3.4 Risk assessment and generation of operational recommendations: The trained model will process newly collected data in real time and output risk scores and action recommendations.

[0047] Depending on the risk score, the system will provide corresponding warnings and guidance measures, such as adjusting work steps or providing safety tips.

[0048] In actual use, when the model detects that a worker stares at a potential risk area for a long time, the system will generate a high-risk score and display a safety operation guide through the AR device to guide the worker to take preventive measures. In this embodiment, the adaptive learning model can self-optimize according to real-time data to improve the accuracy and timeliness of risk assessment. The application of the CNN model significantly improves the efficiency and depth of feature extraction, enabling the system to accurately identify safety risks from complex eye movement data. The real-time generated operational suggestions at this time help workers respond quickly to potential risks and reduce the occurrence of safety accidents.

[0049] Step 4: Feedback mechanism.

[0050] The real-time feedback in this embodiment provides visual, auditory, and tactile feedback through an AR head-mounted device to ensure that warnings and instructions are effectively conveyed in a noisy environment.

[0051] In order to ensure that safety warnings and operating instructions are effectively conveyed in noisy environments such as construction sites, the system adopts a multimodal feedback mechanism that integrates three feedback methods: visual, auditory, and tactile, to adapt to different work scenarios and individual differences.

[0052] The specific steps of the feedback mechanism are as follows: 4.1 Visual Feedback: The system dynamically overlays warning signs and operating instructions in the worker's direct field of vision through augmented reality (AR) headsets. The AR device uses advanced image recognition technology to ensure that feedback information is synchronized with the worker's visual focus and on-site environment in real time.

[0053] During use, when the system detects potential safety risks, such as if a worker's eyes stay in a dangerous area for a long time, the AR device will immediately display a red warning sign at the relevant location to remind the worker to pay attention to safety.

[0054] 4.2 Auditory Feedback: Auditory feedback is provided via headphones, providing workers with clear voice instructions and warnings.

[0055] The system adjusts the volume based on on-site noise levels and worker preferences to ensure clear information delivery.

[0056] In actual use, before workers perform high-risk operations, the system will issue voice prompts through headphones such as "Please stop the operation immediately and check the surrounding environment" to guide workers to take safety measures.

[0057] 4.3 Haptic feedback: The system provides tactile signals, such as vibration reminders, through wearable devices as a supplement to visual and auditory feedback. Tactile feedback is particularly suitable for environments with excessive noise or obstructed vision, ensuring that workers receive safety reminders in a timely manner. When workers approach a dangerous area, the wearable device will vibrate as a non-invasive reminder.

[0058] In this embodiment, suppose that a worker is operating an aerial work machine, and the system discovers through eye tracking that he stares at a moving hook for 5 consecutive seconds, which may indicate that the worker is uncertain about the movement trajectory of the hook.

[0059] The system immediately displays a yellow warning box at the hook position through the AR headset and issues a voice prompt through the headphones: "Pay attention to the mobile device above."

[0060] At the same time, the worker's wearable device begins to vibrate to reinforce the warning signal.

[0061] Workers adjust their positions and operations based on these multimodal feedbacks to avoid potential safety accidents.

[0062] In this embodiment, through this multimodal feedback mechanism, the system not only improves the reliability of information transmission, but also enhances the adaptability and safety of workers in complex environments.

[0063] Step 5: Dynamic environmental mapping and augmented reality overlay In this embodiment, GPS and IMU technologies are used for real-time positioning, dynamic mapping of the construction site is constructed, and an augmented reality view is superimposed through spatial positioning technology.

[0064] The core of this embodiment is to use the global positioning system (GPS) and inertial measurement unit (IMU) technology to locate workers and equipment on the construction site in real time with high precision. Combined with advanced image processing and 3D modeling technology, the system can build a dynamic map that is synchronized with the site in real time. In addition, through precise spatial positioning technology, the system directly superimposes safety tips, operating instructions and warning information in the worker's field of vision to form an augmented reality (AR) view.

[0065] The positioning details in this embodiment are as follows: 5.1 Real-time positioning technology: Use GPS to determine the approximate location of workers and equipment, and use IMU to compensate for tiny errors in rapid movement to ensure the continuity and accuracy of positioning.

[0066] Mathematical expression: GPS positioning can be expressed as P GPS (t)=(x,y,z), where (x,y,z) is the three-dimensional space coordinate and t is the timestamp. IMU data is used for positioning correction and can be expressed as ΔP IMU (t)=(Δx,Δy,Δz).

[0067] 5.2 Dynamic mapping construction: Through 3D scanning and photogrammetry technology, combined with real-time updated positioning data, a dynamic 3D model including the construction site topography, equipment layout and construction progress is constructed. In this embodiment, the 3D scanning data of the construction site is updated regularly, and key information such as on-site equipment, material storage areas, and construction areas are identified and marked through algorithms.

[0068] 5.3 Augmented reality view overlay: The camera on the head-mounted AR device is used to capture real-time images, and spatial positioning is achieved through SLAM (Simultaneous Localization and Mapping) technology, so that virtual information can be accurately superimposed on the worker's actual field of vision.

[0069] Specifically, when a worker looks toward a potentially dangerous area, the system immediately overlays a dynamic red warning border on the AR device, while displaying the safe distance and avoidance measures.

[0070] The specific examples in this embodiment are as follows: Suppose a worker needs to perform excavation work in an area filled with pipes and cables.

[0071] The system determines the worker's location through GPS and IMU and builds a dynamic 3D model of the surrounding environment on their AR device.

[0072] When a worker approaches the location of an underground cable, the system superimposes a virtual cable marker and warning information in his field of view through the AR device.

[0073] Workers adjust their excavation positions based on this intuitive augmented reality view to avoid damaging cables and ensure construction safety.

[0074] Step 6: Multi-mode work support.

[0075] In this embodiment, the collaborative work mode supports interoperability between multiple workers' devices to promote team collaboration and operational efficiency.

[0076] In order to strengthen the collaboration among team members and improve operational efficiency, the system is designed with a multi-device collaborative working mode. In this mode, each worker's device can communicate with other devices, share key information, and complete tasks together.

[0077] This embodiment includes the following details: 6.1 Communication mechanism between devices: Use wireless networking technology, such as Wi-Fi or Bluetooth, to ensure high-speed data transmission between devices.

[0078] Mathematical expression: The connectivity of communication between devices can be represented by a network in graph theory, where nodes represent devices and edges represent communication channels. Let G=(V,E) be a communication network, where V is the set of devices and E is the set of communication links connecting two devices.

[0079] 6.2 Information Sharing Agreement: The information shared between devices includes but is not limited to personal location, gaze tracking data, environmental warnings, and operational guidance. When a worker’s device detects a potential risk, the information is immediately broadcast to all connected devices, ensuring that team members have access to critical information in real time.

[0080] 6.3 Collaborative task execution: The system supports real-time task allocation and progress updates, ensuring that team members have a clear understanding of their respective responsibilities and the overall project status.

[0081] How it works: From a central console or project management app, task assignments and progress updates can be synced to each worker’s AR device in real time. Specific implementation method: Suppose a construction site team consists of multiple workers with different responsibilities, including excavators, electricians, and supervisors.

[0083] As the excavator approaches a location marked as an area where cables are located, his device sends location and line-of-sight tracking data to the electrician's device in a collaborative mode.

[0084] After receiving the data, the electrician uses the AR device to view the excavator's perspective and provide guidance to avoid damaging the cables.

[0085] At the same time, the equipment can monitor the progress of tasks and the status of team collaboration in real time to ensure that operational efficiency and safety standards are maintained. Through this collaborative working mode, team members can collaborate efficiently, reduce communication costs, and improve the construction efficiency of the entire construction site.

[0086] This embodiment has the following characteristics: 1) By integrating gaze tracking with augmented reality (AR) technology, real-time monitoring of workers’ gaze and instant feedback based on gaze are achieved. This integration improves the real-time nature of monitoring and the personalization of feedback, thereby significantly improving the safety and efficiency of on-site work.

[0087] 2) Using machine learning algorithms to analyze the collected sight data and on-site environmental data in real time can quickly identify safety risks and improper operations. Compared with existing technologies, this real-time analysis and feedback mechanism can provide personalized guidance based on the specific situation of each worker, such as highlighting dangerous areas or providing operational improvement suggestions, greatly improving operational safety and efficiency.

[0088] 3) By using advanced machine learning algorithms such as convolutional neural networks (CNNs) to analyze line of sight data, the present invention can accurately identify the areas where workers' lines of sight are focused and their potential risks. In this regard, machine learning algorithms have significantly improved the accuracy and efficiency of data analysis compared to traditional data processing methods. In addition, the system can automatically learn and adapt to the line of sight behaviors of different workers, further enhancing the relevance and accuracy of personalized feedback.

[0089] 4) The adaptive learning model of the present invention can automatically adjust the analysis parameters according to the actual application scenario and the collected data, which means that the system can continuously optimize its performance over time and with the accumulation of more data. This dynamic adjustment mechanism makes the system not only effective when it is first deployed, but also able to continuously adapt to new working environments and challenges, maintaining high efficiency and high accuracy monitoring and feedback capabilities.

[0090] 5) This system adopts a modular design concept, and each technical component (such as gaze tracking module, machine learning analysis module, feedback mechanism, etc.) is designed as an independent module. This design facilitates the interchange and upgrade of modules to adapt to new technological developments. The system's software architecture supports the introduction of new algorithms and functions through remote updates. With the continuous advancement of machine learning and augmented reality technology, software updates can be pushed regularly to include more efficient data processing algorithms, more accurate gaze tracking technology, or new user interaction functions.

[0091] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0092] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. The construction operation monitoring system based on eye tracking is characterized by: include, A sight acquisition component, the sight acquisition component includes an auxiliary mechanism and a gaze tracking device, the auxiliary mechanism captures the eye image of the operator in the gaze tracking device, and uses an image algorithm to obtain sight data to achieve real-time eye movement monitoring; An environment collection component, which is used to obtain environmental parameters during construction to form environmental data; The monitoring model uses input environmental parameters and line of sight data and a learning algorithm to evaluate the safety risks and operating efficiency of the operator's operation, and guides the operator to perform correct operating steps when there is a safety risk.

2. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: The auxiliary mechanism includes an infrared LED lamp and an infrared camera located outside the gaze tracking device. The infrared camera is used to collect infrared light reflected by the eyes to generate continuous eye images.

3. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: The image algorithm specifically uses the algorithm to identify the pupil center and its reflection point in the preprocessed eye image and calculate the sight direction.

4. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: The monitoring model includes a feature extraction unit, which is used to extract the gaze point, gaze duration and gaze frequency of the operator's line of sight.

5. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: It also includes training optimization of the monitoring model, specifically: using labeled historical data to train the convolutional neural network model, and optimizing the model parameters through the back propagation algorithm and gradient descent method to improve the accuracy of risk assessment.

6. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: The monitoring module includes a multimodal real-time feedback device, specifically including visual feedback, auditory feedback and tactile feedback, and the visual feedback is achieved by a remotely controlled gaze tracking device.

7. The construction operation monitoring system based on eye tracking according to claim 6, characterized in that: The auditory feedback is realized through voice prompts, and the tactile feedback is specifically to control the eye tracking device to cause it to act on the operator's touch for feedback.

8. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: It also includes a positioning system, which specifically uses an inertial measurement unit to supplement the positioning information obtained by GPS to obtain the positioning of the operator.

9. The construction operation monitoring system based on eye tracking according to claim 8, characterized in that: It also includes the construction of a dynamic three-dimensional model, which is to update the positioning data in real time through three-dimensional scanning and photogrammetry technology, and build a three-dimensional model of the construction site to prompt operators with component information at corresponding locations.

10. The construction operation monitoring system based on eye tracking according to claim 1, characterized in that: It also includes a collaborative working unit, which enables the eye tracking devices of several operators to work collaboratively through a communication mechanism, a location sharing protocol and a collaborative task execution mechanism.

Images