A smart construction site safety inspection device and system based on image recognition

Abstract

This invention discloses a smart construction site safety inspection device and system based on image recognition, belonging to the field of machine inspection technology. The system acquires audio, video, and environmental parameters from the construction site through a multimodal acquisition unit. Combined with image recognition, eye tracking, semantic fusion, and permission verification technologies, it achieves full-process automation of target interaction personnel identification, effective voice command filtering, target device positioning, and control command generation and execution. The device provides hardware support for the system, ensuring the stable operation of each functional module. The two work together to comprehensively cover construction site inspection scenarios, solving the problem of single data acquisition in existing inspection equipment. Furthermore, this invention combines eye tracking and a three-level device physical-semantic map to achieve precise positioning of target devices. Even with multiple candidate devices, it can quickly pinpoint user intent, improving the level of interactive intelligence.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a smart construction site safety inspection device and system based on image recognition. Background Technology

[0002] Inspection robots often operate in high-noise environments (equipment noise, airflow noise, wind noise, etc.), and the accuracy and reliability of their human-robot interaction directly affect inspection efficiency and operational safety. Current technologies mainly rely on single voice interaction or a single wake-word recognition mode, which has significant shortcomings in practical industrial applications and is difficult to meet the needs of complex working conditions. 1. In collaborative work scenarios involving multiple personnel, robots cannot effectively identify the specific identity and intent of the person issuing the instruction. They are easily disturbed by conversations among those around them, resulting in erroneous responses to instructions from non-target personnel, a typical "cocktail party effect" that seriously affects the work order. 2. In emergency or complex situations, operators' instructions are often unstructured. A single voice modality cannot be accurately interpreted in combination with the specific physical orientation on site, leading to deviations in instruction execution and even safety hazards.

[0003] In view of the shortcomings of the existing technologies, there is an urgent need for an inspection solution that can solve the problems of multi-target confusion and semantic ambiguity, and improve the accuracy of inspection robots. Summary of the Invention

[0004] This invention provides a smart construction site safety inspection system and device based on image recognition. The system acquires audio, video, environmental parameters, and other data from the construction site through a multimodal acquisition unit. Combining image recognition, eye tracking, semantic fusion, and permission verification technologies, it achieves full-process automation of target interaction personnel identification, effective voice command filtering, target equipment positioning, control command generation, and execution. The device provides hardware support for the system, ensuring the stable operation of each functional module. The two work together to solve the pain points of existing construction site inspections. This invention provides the following technical solution: a smart construction site safety inspection system based on image recognition, the inspection system comprising: S1: Real-time sensor data is acquired through a multimodal acquisition unit installed on the inspection robot; the multimodal acquisition unit includes: a microphone array for acquiring audio stream data, an RGB visible light camera and an infrared thermal imaging camera for acquiring video stream data, an environmental situation sensor for acquiring environmental parameter data, and a depth sensor for assisting positioning; S2: Calculate the sound source location based on the audio stream data, identify the personnel target based on the video stream data, and spatiotemporally correlate the sound source location with the personnel target location to determine the target interaction personnel; S3: Extract the visual features of lip movements of the target interactive person and the speech activity features in the audio stream data, calculate the visual-auditory temporal consistency, and filter valid speech commands based on the temporal consistency; S4: Extract the gaze focus of the target interactive person through the eye-tracking unit and head-tracking unit on the inspection robot, and determine the target device by combining it with the preset device physical-semantic map; S5: Perform semantic fusion between the effective voice command and the target device to generate preliminary control commands; S6: Perform permission verification on the preliminary control command. If the verification passes, output the final control command; if the verification fails, refuse to execute and provide a prompt message. The overall process framework of the image recognition-based smart construction site safety inspection system is defined, and the core working steps of the system are clarified, covering the entire process of multimodal data acquisition, target interaction personnel identification, effective voice command screening, target equipment identification, control command generation and permission verification. Step S2 includes: Acquire audio stream data collected by a microphone array, and use a time difference of arrival algorithm to calculate at least one sound source azimuth angle and its sound source confidence. The system acquires video stream data from an RGB visible light camera and an infrared thermal imaging camera, uses a target detection network to identify at least one person target, and calculates the position and visual confidence of each person target in the image coordinate system. Based on the pre-calibrated microphone array-camera extrinsic parameters, the azimuth angles of each sound source are mapped to the image coordinate system to generate sound source azimuth lines; Calculate the pixel distance between the position of each person target and the azimuth line of each sound source, and calculate the matching score based on the pixel distance, sound source confidence, and visual confidence. Select the person with the highest matching score that is greater than the preset threshold as the candidate target interaction person; determine the person with the highest overall matching score among the candidate target interaction persons as the target interaction person. Specifically, this involves extracting the voiceprint and facial feature vectors of the candidate target interacting personnel. Query the preset voiceprint-face association database and calculate the cross-modal matching degree between the voiceprint feature vector and the face feature vector; Extract the visual features of lip movements of candidate target interactors and verify their temporal consistency with the speech activity features in the audio stream data to obtain the lip movement-speech matching degree. The matching score, cross-modal matching degree, and lip movement-speech matching degree are weighted and fused to obtain a comprehensive matching score; When the overall matching score is greater than the preset fusion threshold, the candidate target interaction person is determined as the target interaction person.

[0005] After identifying the target audience in step S2, the following steps are also included: A Kalman filter is used to continuously track the position and velocity of the target interacting person in three-dimensional space; A unique identifier is assigned to the target interaction user, and the target interaction user's facial feature vector, voiceprint feature vector, and appearance feature vector are associated and stored with the identifier. When the target interacting person briefly leaves the field of vision and then reappears, the original identity is reused by re-association through location prediction and feature matching. When multiple people are detected, the interaction intent score is calculated based on each person's body orientation, gaze direction, lip movement state, and sound source location matching degree. The person with the highest interaction intent score is selected as the current primary target.

[0006] Step S2 also includes: When the sound source location calculation fails or the sound source confidence is lower than the first threshold, it is downgraded to pure vision mode, and the target interaction person is determined based on the person facing the robot and whose lip movement state is speaking in the visual detection results. When the person target detection fails or the visual confidence level is lower than the second threshold, the mode is downgraded to pure acoustic mode, and the position of the target person in space is determined based on the azimuth angle of the sound source and the distance information obtained by the depth sensor. When the spatial matching score between the sound source location and the target person's location is lower than the third threshold, an active confirmation mechanism is triggered. A confirmation request is output through the interactive interface, and the target person to be interacted with is determined based on the user's confirmation feedback.

[0007] An interaction intent score is maintained for each person in the field of vision, and the interaction intent score is updated in real time based on body orientation angle, gaze direction, lip movement state and sound source location matching degree. When the interaction intent score of a non-primary target person exceeds the interaction intent score of the current primary target person and the duration exceeds a preset time threshold, target switching is performed, and the non-primary target person is identified as the new primary target. When switching targets, a switching prompt message is displayed through the interactive interface. In step S3, the calculation of visual-auditory temporal consistency includes the following sub-steps: S31: Analyze the facial ROI region of the target interactive person using a 3D convolutional neural network to obtain the start and end time points of lip movements [t1, t2]. S32: Perform speech activity detection on the audio stream data, extract speech activity features, and obtain the speech start and end time points [t3, t4]; S33: Calculate the time overlap rate between [t1, t2] and [t3, t4]. If the time overlap rate is greater than a preset threshold, the audio stream data is determined to be a valid voice command; if the time overlap rate is less than the preset threshold, it is determined to be environmental noise and discarded. The specific process of extracting the gaze focus of the target interacting person in step S4 is as follows: Obtain the eye rotation angle output by the eye tracking unit and the head pose angle output by the head tracking unit; Synthesize the line-of-sight direction vector in absolute space; Kalman filtering is applied to the line-of-sight vector sequence to suppress high-frequency jitter; A sliding window mid-value filter is applied to the filtered line-of-sight point sequence to output a stable line-of-sight focus. A stable gaze focus is input into a temporal convolutional network for multiple consecutive frames, and the gaze type is classified and output. The gaze type includes saccade state, gaze state and tracking state. The current gaze focus is marked as a valid gaze point only when the gaze type is a gaze state and the duration exceeds a preset duration threshold; Spatiotemporal clustering is performed on continuous valid gaze points to generate gaze clusters, with each gaze cluster corresponding to a candidate target device. Step S4, which involves determining the target device by combining a preset device physical-semantic map, specifically includes: Construct a three-level device physical-semantic graph, the three levels being scenario-level, device-level, and component-level; Obtain the robot's real-time pose information, determine the current scene area, and load the device list corresponding to the scene area; Project the viewpoint into three-dimensional space, calculate the distance to each device's bounding box, and generate a device-level candidate list and corresponding distance weights; If there is a unique candidate in the device-level candidate list or the difference in candidate weights is greater than a preset threshold, then the candidate is locked as the target device. If there are multiple candidate devices and the weight difference is less than a preset threshold, then component-level matching is entered. The distance between the gaze focus and the key components of each device is calculated, and the target device is determined by combining the gaze duration. If there are still multiple candidate devices after component-level matching, the semantic parsing result of the effective voice command is obtained. Extract device type keywords and location descriptors from the semantic parsing results; The device type keywords are matched with the semantic attributes of the candidate devices, and candidate devices that meet the preset matching conditions are filtered. The selected candidate devices are spatially matched with location descriptors to determine the target device; Record the historical target devices and their spatial locations from the most recent N interactions (the historical target devices are the target devices identified during the interaction process of the inspection system, which are automatically recorded and stored by the system). When the Euclidean distance between the spatial location of the current candidate device and the spatial location of the historical target device is less than a preset distance threshold, the matching weight of the candidate device is increased. When the duration of fixation at the current point exceeds a preset long-term threshold, the historical target device is overwritten and updated to the current candidate device. Step S5 includes: The valid voice command is parsed to identify pronouns, omitted components, and ambiguous expressions. When a pronoun is identified, the target device determined in step S4 is obtained, and the pronoun is replaced with the device identifier of the target device. When an omitted component is identified, the omitted information is completed based on the task context or historical operation records. When ambiguous expressions are identified, multiple rounds of dialogue are initiated through the interactive interface to disambiguate and generate complete and structured preliminary control instructions. In step S5, the environmental parameter data includes noise level in decibels, light intensity, and wind speed. The specific method for dynamically adjusting the modal weights is as follows: When the noise level is greater than 85 dB or the wind speed is greater than level 5, the decision weight of the auditory modality is adjusted to 30%, and the decision weight of the visual lip movement and eye tracking modalities is adjusted to 70%. When the illumination is less than 50 Lux, switch to infrared vision mode and combine thermal imaging contours to perform lip movement detection or gesture recognition.

[0008] Furthermore, the decision weights of each sensing modality are dynamically adjusted based on environmental parameter data, specifically including: Acquire environmental acoustic parameters, environmental optical parameters, user distance parameters, user posture parameters, and task context parameters to construct a five-dimensional context-aware space; The five-dimensional context-aware parameters are input to the fuzzy logic controller, which outputs auditory modality weights, visual modality weights, and context modality weights according to a preset fuzzy rule base. The centroid method is used to defuzzify the output of the fuzzy logic controller to obtain the precise weight values ​​of each mode; The specific process of performing permission verification on the preliminary control command in step S6 is as follows: Obtain the identity of the target user and query the corresponding user permission configuration file; Obtain the status information of the target device to determine its sensitivity level and current operating status; Based on user permission configuration, device sensitivity level, and operation type of initial control command, multi-dimensional permission verification is performed. The multi-dimensional verification includes at least user identity verification (verifying the legitimacy of the user's identity), device permission verification (verifying whether the user has the permission to operate the device), operation risk verification (verifying whether the operation command has security risks), and contextual security verification (verifying whether the current environment and device status are suitable for executing the command). When all dimensions pass the validation, the final control command is output; Before executing the final control command, an operation scenario snapshot is collected and stored. The operation scenario snapshot includes at least: the identity of the target interactive personnel, the content of the operation command, the execution timestamp, environmental parameter data, device status data, and audio and video clips. The snapshot of the operation scenario is associated and stored in the local log database, and synchronized to the remote audit server when the robot is connected to the network. A smart construction site safety inspection device based on image recognition, the inspection device includes an inspection robot, the inspection robot is equipped with a multimodal acquisition unit, an edge computing unit, an execution and feedback unit, an eye tracking unit and a head tracking unit, the multimodal acquisition unit, the execution and feedback unit, the eye tracking unit and the head tracking unit are electrically connected to the edge computing unit respectively; The multimodal acquisition unit includes a ring microphone array, a wide-angle RGB visible light camera, an infrared thermal imaging camera, a depth sensor, and an environmental situation sensor; the ring microphone array is a ring structure with ≥6 microphones and is set in the robot's head; the environmental situation sensor is used to collect noise decibel values, light intensity, and wind speed data; The edge computing unit includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the inspection system as described in any one of claims 1-7. The execution and feedback unit is used to receive and execute the final control command output by the edge computing unit, and at the same time provide feedback on the interaction status information. The execution and feedback unit includes a robot motion control system, a robotic arm, and an interactive display screen; the interactive display screen is used to display the command recognition status, permission verification results, and prompt information, and is also used to receive manual confirmation operations from the user.

[0009] Compared with the prior art, the beneficial effects of the present invention are: Compared with the prior art, the present invention has the following advantages: 1. This invention achieves multimodal data fusion acquisition, combining RGB visible light, infrared thermal imaging, audio, environmental parameters, and depth data to comprehensively cover construction site inspection scenarios and solve the problem of single data acquisition in existing inspection equipment; 2. This invention achieves accurate identification of target interactive personnel and effective voice commands by correlating the sound source with the location of the person in time and verifying the consistency between lip movement and speech time. It has strong anti-interference ability and avoids the influence of environmental noise and irrelevant personnel. 3. This invention combines eye tracking and a three-level device physical-semantic graph to achieve precise positioning of the target device. Even with multiple candidate devices, it can quickly pinpoint the user's intent and improve the level of intelligent interaction. 4. This invention employs multi-dimensional permission verification to strengthen on-site safety management and prevent unauthorized and high-risk operations. At the same time, it enables operation traceability through operation scenario snapshots, facilitating subsequent auditing and accident investigation. 5. The device of this invention has a reasonable structure, with each module working in concert, adapting to the harsh environment of the construction site, supporting offline operation and automatic charging, with strong battery life, enabling 24-hour uninterrupted inspection, greatly improving inspection efficiency and reducing labor costs. Attached Figure Description

[0010] Figure 1 This is a flowchart of an image recognition-based smart construction site safety inspection system according to the present invention. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to specific embodiments, so that those skilled in the art can understand and implement it. It should be noted that the embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention. Any modifications or equivalent substitutions based on the present invention without departing from the concept of the present invention shall fall within the scope of protection of the present invention. Example 1:

[0012] In this embodiment, the smart construction site safety inspection system and device are applied to large-scale construction sites. Inspection robots are deployed on the construction site, covering key areas such as construction areas, equipment areas, and personnel passages. The multimodal acquisition unit collects audio and video data and environmental parameter data of the construction site in real time, the edge computing unit processes the data locally (avoiding network latency), and the execution and feedback unit responds to control commands to realize the automation and intelligence of construction site inspection. The remote audit server is deployed at the construction site monitoring center to synchronously store snapshots of operation scenarios, realizing remote auditing and monitoring.

[0013] In this embodiment, the preset parameters are as follows (which can be adjusted according to actual site requirements): the preset threshold for sound source confidence is 0.7, the preset threshold for visual confidence is 0.8, the preset threshold for matching score is 0.75; the preset threshold for time overlap rate is 0.6; the preset threshold for gaze duration is 1.5s; the preset threshold for device candidate weight difference is 0.3; the preset threshold for Euclidean distance is 0.5m; the preset duration threshold is 3s; the ring microphone array is an 8-microphone ring structure; N is 5 (the last 5 interaction records). See attached... Figure 1 As shown, the specific operation flow of the image recognition-based smart construction site safety inspection system in this embodiment is as follows: S1: The inspection robot moves along the preset inspection route, and the multi-modal acquisition unit works synchronously. The microphone array collects audio stream data from the construction site (including personnel voices, equipment operating noise, environmental noise, etc.), the RGB visible light camera collects color video stream data from the site (for personnel identification and equipment appearance identification), the infrared thermal imaging camera collects infrared video stream data from the site (for equipment temperature monitoring and personnel location), the environmental situation sensor collects noise decibel values, light intensity, and wind speed data from the site, and the depth sensor collects depth information of objects on site (to assist in positioning). All real-time sensing data is transmitted to the edge computing unit. The multimodal acquisition unit's data acquisition frequency is set as follows: 16kHz sampling rate for audio stream, 30 frames / second for video stream, and environmental parameters are acquired every 100ms. The edge computing unit uses an NVIDIA Jetson AGX Xavier chip, supporting real-time data processing (latency ≤100ms). The system supports offline working mode; when the robot is not connected to the network, the operation scenario snapshot is stored in local memory and automatically synchronized to the remote audit server after connecting to the network. The preset device physical-semantic map can be imported through the construction site equipment ledger and supports manual and automatic updates (adding devices through image recognition). S2: The edge computing unit processes the audio stream data to calculate the sound source location and simultaneously performs personnel target recognition on the video stream data, spatiotemporally associating the sound source location with the personnel target location to determine the target interactive person who is speaking (i.e., the person initiating the control command). The specific implementation process of step S2 (identifying the target audience) is as follows: (1) Acquire audio stream data collected by microphone array (8-microphone ring structure), and use the time difference of arrival (TDOA) algorithm to calculate at least one sound source azimuth angle (accuracy ±1°) and its sound source confidence (range 0-1). For example, two sound source azimuth angles are calculated, namely 30° (confidence 0.85) and 150° (confidence 0.6). (2) Acquire video stream data collected by RGB visible light camera and infrared thermal imaging camera, use YOLOv8 target detection network to identify at least one person target on site, calculate the position (in pixels) and visual confidence (range 0-1) of each person target in the image coordinate system. For example, 3 person targets are identified, with positions of (320, 240) (confidence 0.88), (640, 480) (confidence 0.92), and (960, 360) (confidence 0.7). (3) Based on the pre-calibrated microphone array-camera extrinsic parameters (obtained by Zhang Zhengyou calibration method, including rotation matrix and translation vector), map the azimuth angle of each sound source to the image coordinate system to generate the azimuth line of the sound source (for example, after mapping the azimuth angle of the 30° sound source, a straight line from the center of the image to the 30° direction is generated in the image coordinate system). (4) Calculate the pixel distance between the position of each person target and the azimuth line of each sound source (for example, the pixel distance between the person target (640, 480) and the azimuth line of the 30° sound source is 15 pixels, and the pixel distance between the person target (640, 480) and the azimuth line of the 150° sound source is 80 pixels). Based on the pixel distance (the smaller the distance, the higher the weight), the sound source confidence, and the visual confidence, calculate the matching score using the weighted summation formula: Matching score = 0.3 × sound source confidence + 0.4 × visual confidence + 0.3 × (1 - pixel distance / image diagonal length). For example, the matching score of the person target (640, 480) with the 30° sound source is 0.82, and the matching score with the 150° sound source is 0.65. (5) Select the person with the highest matching score and greater than the preset threshold (0.75) as the candidate target interaction person. Here, the matching score of the person target (640, 480) is 0.82 > 0.75, so it is selected as the candidate target interaction person. Since there is only one candidate target interaction person, it is directly determined as the target interaction person. The specific implementation of the TDOA algorithm is as follows: by calculating the time difference of the audio signal arriving at different microphones, and combining the geometric position of the microphone array, the azimuth angle of the sound source is solved; the input of the YOLOv8 object detection network is a video frame (resolution 1920×1080), and the output is the bounding box coordinates and visual confidence of the person target; the image diagonal length is calculated according to the camera resolution (the diagonal length of an image with a resolution of 1920×1080 is √(1920²+1080²)≈2203 pixels); if there are multiple candidate target interaction persons (all matching scores > 0.75), the one with the highest comprehensive matching score (the average matching score of all sound sources with the person) is taken as the target interaction person. S3: The edge computing unit extracts the visual features of the target user's lip movements and the speech activity features in the audio stream data. It calculates the visual-auditory temporal consistency between the two and filters out the real and effective voice commands (excluding environmental noise). The specific implementation process of calculating visual-auditory temporal consistency in step S3 is as follows: S31: Analyze the ROI region of the target interactive person's face using a 3D convolutional neural network (C3D network) (extracted by a face detection algorithm; the ROI region is the area around the lips on the face, with a size of 128×128 pixels). Analyze 10 consecutive video frames to identify the start and end times of lip movements and obtain the start and end time points of lip movements [t1, t2], for example, t1=10.2s, t2=12.5s (lip movement duration 2.3s). S32: Perform speech activity detection (VAD algorithm) on the audio stream data, set the speech activity threshold to 0.02 (calibrated according to ambient noise), identify the time periods in the audio stream where speech exists, extract speech activity features (Mel frequency cepstral coefficients MFCC), and obtain the speech start and end time points [t3, t4], for example, t3=10.3s, t4=12.4s (speech duration 2.1s). S33: Calculate the time overlap rate between [t1, t2] and [t3, t4]. The time overlap rate = overlap time length / (maximum of lip movement duration and speech duration). Here, the overlap time is 12.4s - 10.3s = 2.1s, the maximum value is 2.3s, and the overlap rate = 2.1 / 2.3 ≈ 0.91, which is greater than the preset threshold of 0.6. Therefore, the audio stream data is determined to be a valid speech command. If the overlap rate is less than 0.6 (for example, lip movement time [t1 = 10.2s, t2 = 10.5s], speech time [t3 = 11.0s, t4 = 11.3s], overlap rate = 0), it is determined to be environmental noise and discarded. The C3D network takes as input a video clip of the target person's facial ROI (30 frames / second, 3 seconds in length) and outputs as the start and end times of lip movements. The network is trained and optimized using a training set (containing lip movement videos of different people and different speaking speeds), achieving a lip movement recognition accuracy of ≥95%. The VAD algorithm uses an adaptive threshold that can be adjusted in real time according to environmental noise (the higher the noise level, the higher the threshold), avoiding misinterpretation of environmental noise as speech. The speech activity feature MFCC is extracted in 13 dimensions for subsequent speech command parsing. S4: By using the eye-tracking unit and head-tracking unit, extract the focus of the target user's gaze, and combine it with the preset physical-semantic map of the construction site equipment to determine the target equipment that the target user wants to control. The specific implementation process of extracting the gaze focus of the target interactive person in step S4 is as follows: (1) Obtain the eye rotation angle (horizontal rotation angle, vertical rotation angle) output by the eye tracking unit (using infrared eye tracking module, sampling rate 100Hz) and the head attitude angle (pitch angle, yaw angle, roll angle) output by the head tracking unit (using IMU inertial measurement unit, sampling rate 100Hz). (2) Based on the eye rotation angle and head posture angle, a coordinate transformation algorithm is used to synthesize the gaze direction vector in absolute space (with the inspection robot as the origin, the x-axis as the horizontal direction, the y-axis as the vertical direction, and the z-axis as the forward direction). For example, the synthesized gaze direction vector is (0.2, 0.1, 0.97). (3) Perform Kalman filtering on the line-of-sight vector sequence (50 consecutive frames, 0.5s duration), set the process noise covariance Q=diag([0.01, 0.01, 0.01]) and the observation noise covariance R=diag([0.1, 0.1, 0.1]) to suppress high-frequency line-of-sight jitter (jitter amplitude after filtering ≤0.5°). (4) Perform sliding window mid-range filtering (window size is 5 frames) on the filtered gaze point sequence (spatial points obtained by combining gaze direction vector and depth data) to output stable gaze focus (spatial coordinates), for example, the stable gaze focus coordinates are (5.2m, 2.3m, 10.5m). (5) Input the stable gaze focus of 20 consecutive frames (duration 0.67s) into the temporal convolutional network (TCN) and classify and output the gaze type. The TCN network is trained with a training set (containing gaze data of three states: saccade, gaze, and tracking) and the classification accuracy is ≥98%. For example, the output gaze type is "gaze state". (6) If the gaze type is determined to be a gaze state and the duration (1.8s) exceeds the preset duration threshold (1.5s), the current gaze focus is marked as a valid gaze point. (7) Spatiotemporal clustering is performed on consecutive effective gaze points (e.g., 10 consecutive frames, duration 0.33s) using the K-means clustering algorithm, K=1, to generate gaze clusters, which correspond to a candidate target device (i.e. the device on which the gaze is focused). The eye-tracking unit has a detection accuracy of 0.1°, and the head-tracking unit has a pose angle accuracy of 0.05°. The coordinate transformation algorithm is implemented through a pre-calibrated eye-head coordinate transformation matrix to ensure the accuracy of the gaze direction vector. Kalman filtering is used to suppress gaze deviations caused by eye tremors and minor head movements, and sliding window midpoint filtering is used to further smooth the gaze focus data. The input of the TCN network is the spatial coordinate sequence of the gaze focus, and the output is the probability of three gaze types. The one with the highest probability is taken as the final gaze type.

[0014] Example 2: In this embodiment, the specific implementation process of determining the target device in step S4 by combining the preset device physical-semantic map is as follows: (1) Construct a three-level physical-semantic map of equipment, where the scene level is the various areas of the construction site (such as the construction area, equipment area, and office area), the equipment level is the equipment in each area (such as the tower crane in the construction area, the power distribution box in the equipment area, and the surveillance camera in the office area), and the component level is the key components of each equipment (such as the hook of the tower crane, the switch of the power distribution box, and the lens of the surveillance camera). The map contains information such as the spatial coordinates of the equipment, equipment type, semantic attributes (such as "tower crane" corresponding to "lifting equipment" and "high-altitude operation"), and component location. (2) Obtain the real-time pose information of the inspection robot (by GPS+IMU fusion positioning, with a positioning accuracy of ±0.1m), determine the current scene area as the equipment area, and load the equipment list corresponding to the equipment area (including 5 devices such as power distribution box, frequency converter, and water pump). (3) Project the effective gaze point (spatial coordinates (5.2m, 2.3m, 10.5m)) into three-dimensional space, calculate the distance to the bounding box of each device in the device list (the device bounding box is the preset three-dimensional bounding box of the device), generate a device-level candidate list and the corresponding distance weight (the closer the distance, the higher the weight), for example: distribution box (distance 0.3m, weight 0.85), frequency converter (distance 1.2m, weight 0.6), water pump (distance 2.5m, weight 0.4). (4) In the equipment-level candidate list, the weight of the distribution box (0.85) and the weight of the second-ranked frequency converter (0.6) are 0.25, which is less than the preset threshold of 0.3, so it enters the component-level matching; calculate the distance between the effective gaze point and the key components (switch, indicator light, terminal block) of the distribution box, where the distance to the switch is 0.15m and the gaze duration is 1.8s. Combining the distance and gaze duration, the distribution box is determined as a candidate device. (5) After component-level matching, only one candidate device (distribution box) is determined, and the candidate is directly locked as the target device; if there are multiple candidate devices, the semantic parsing results of the effective voice command (such as "open the left distribution box") are obtained, the device type keyword "distribution box" and the location description word "left side" are extracted, the semantic attributes of "distribution box" are matched with the candidate devices, the distribution box type devices are filtered out, and the target device is determined by combining the spatial location of "left side". (6) Record the historical target devices and their spatial locations in the last 5 interactions. For example, the historical target device is "right distribution box" (spatial coordinates (5.8m, 2.3m, 10.5m)). The Euclidean distance between the current candidate device (left distribution box) and the historical target device is √[(5.8-5.2)²+(2.3-2.3)²+(10.5-10.5)²]=0.6m, which is greater than the preset threshold of 0.5m, so the matching weight is not increased. (7) The current gaze duration is 1.8s, which does not exceed the preset long time threshold of 3s and does not cover the historical target device. The target device is finally determined to be the left-side power distribution box. The device physical-semantic graph is stored in the Neo4j graph database, supporting fast querying and updating; the device bounding box is obtained through LiDAR scanning and generated in combination with the device's 3D model; semantic parsing uses the BERT model, which can accurately extract keywords and descriptive words from the instructions; Euclidean distance calculation is used to combine historical interaction habits to improve the efficiency of target device determination, and can quickly match the target device when the user operates the same area of ​​devices multiple times. S5: The edge computing unit semantically fuses valid voice commands with the target device, analyzes the user's control intent, and generates preliminary control commands (such as "start the tower crane" or "close the distribution box"). The specific implementation process of generating preliminary control commands in step S5 is as follows: (1) Parse valid voice commands. For example, if the user’s voice command is “open it”, the BERT semantic parsing model is used to identify the pronoun “it”. No omitted components or ambiguous expressions are identified. (2) The pronoun “it” is identified, the target device (left distribution box) determined in step S4 is obtained, and the pronoun “it” is replaced with the device identifier of the target device (the device identifier is “SB-001”, corresponding to the left distribution box). The instruction after replacement is “open SB-001”. (3) If the voice command is “Close” (with omitted components), then based on the task context (currently inspecting the distribution box) or historical operation record (the last operation was the left distribution box), complete the omitted information and generate the command “Close SB-001”. (4) If the voice command is “Open the distribution box” (which is ambiguous, as there are multiple distribution boxes in the equipment area), then a multi-round dialogue is initiated through the interactive display screen to display a list of candidate devices (distribution box SB-001 on the left and distribution box SB-002 on the right), and ask the user “Please select the distribution box to open: 1. SB-001 on the left 2. SB-002 on the right”. After the user selects via voice or touch screen, the disambiguation generates the complete command “Open SB-001”. (5) Convert the complete instruction into a structured preliminary control instruction in the format of “Operation type: Open; Target device: SB-001; Operation time: Current timestamp; Operation parameters: None” and transmit it to the permission verification module. The BERT semantic parsing model is trained on a construction site speech dataset and supports dialect adaptation (such as Mandarin, Sichuanese, and Cantonese), with a command parsing accuracy of ≥96%. The interactive display screen uses a 10.1-inch touchscreen and supports both voice and touch interaction. Initial control commands are stored in JSON format for easy subsequent permission verification and command execution. When omitting components, the current task context is used as the primary reference, and the three most recent historical operation records are referenced when no context is available. S6: The edge computing unit performs permission verification on the initial control command. If the target user has the right to operate the device and there is no security risk in the operation, the final control command is output and executed by the execution and feedback unit. If the verification fails (e.g., no permission or risk in the operation), the execution is refused and a prompt message is displayed on the interactive screen (e.g., "You do not have permission to operate this device, please contact the administrator").

[0015] The specific implementation process of permission verification in step S6 is as follows: (1) Obtain the identity of the target interactive personnel. The identity of the personnel is identified by facial recognition algorithm (compared with the construction site personnel database). The identity is “RY-005” (construction site electrician). The corresponding user permission configuration file is queried to find out that the personnel has the operation permission of the distribution box (but no operation permission of tower crane, frequency converter and other equipment). (2) Obtain the status information of the target device (SB-001, left-side distribution box) (collected through device sensors), determine the device sensitivity level as "medium" (sensitivity levels are divided into low, medium, and high, the distribution box is medium), and the current operating status is "off" (normal state, no fault). (3) Perform multi-dimensional permission verification: ① User identity verification: Verify the legitimacy of “RY-005”’s identity by comparing it with the personnel database and confirming that the person is an on-site electrician. The identity verification is successful. ② Device permission verification: Verify whether “RY-005” has the operation permission of SB-001. According to the permission configuration file, the person has the operation permission of the distribution box. The device permission verification is successful. ③ Operational risk verification: Verify whether there is any risk in the operation of "opening SB-001". The current device status is normal, and there is no risk of electric shock, short circuit or other risks in the opening operation. The operation risk verification is passed. ④ Contextual safety verification: Verify whether the current environment is suitable for performing the operation. The noise level collected by the environmental situation sensor is 65 dB, the light intensity is 500 lux, and the wind speed is 2 m / s, all of which meet the environmental requirements for the operation of the power distribution box. The contextual safety verification is passed. (4) Once all dimensions pass the verification, output the final control command (the format is the same as the initial control command, with an added verification pass indicator) and transmit it to the execution and feedback unit. (5) Before executing the final control command, collect and store a snapshot of the operation scenario. The snapshot content includes: the target interactive personnel identification "RY-005", the operation command "open SB-001", the execution timestamp "2026-03-15 14:30:25", environmental parameters (noise 65dB, light 500lux, wind speed 2m / s), equipment status (off, no fault), and audio and video clips (voice and facial video of the target interactive personnel). (6) The operation scenario snapshot is associated and stored in the local log database (capacity 1TB, supports cyclic overwrite). When the inspection robot is connected to the network, it is automatically synchronized to the remote audit server for subsequent auditing and accident investigation. The facial recognition algorithm uses ArcFace, with an accuracy rate of ≥99%, and supports recognition while wearing a safety helmet; user permission configuration files are stored in encrypted form and can be modified through the administrator backend; device sensitivity levels are set according to the security risk of the device (e.g., tower cranes are high sensitivity, distribution boxes are medium sensitivity, and lighting equipment is low sensitivity); operation scenario snapshots are stored in encrypted and compressed form, and video clips are encoded using H.265 to save storage space; the remote audit server supports querying operation records by time, personnel, equipment, and other dimensions.

[0016] Example 3: The image recognition-based smart construction site safety inspection device in this embodiment has the following specific structure: The inspection robot is a wheeled mobile robot (maximum speed 3km / h, endurance 8h, supports automatic charging). The robot is equipped with a multimodal acquisition unit, an edge computing unit, an execution and feedback unit, an eye tracking unit, and a head tracking unit. Each unit is electrically connected to the edge computing unit via a CAN bus to realize data transmission and command interaction. (1) Multimodal acquisition unit: including a ring microphone array (8-microphone ring structure, 5cm in diameter, set in the head of the robot, sampling rate 16kHz, frequency response 20Hz-20kHz), a wide-angle RGB visible light camera (resolution 1920×1080, frame rate 30 frames / second, field of view 120°, set in the front of the robot head), an infrared thermal imaging camera (resolution 640×480, temperature measurement range -20℃~150℃, field of view 90°, set in parallel with the RGB camera), a depth sensor (using a ToF depth camera, measurement range 0.5m~10m, accuracy ±1cm) and an environmental situation sensor (integrating a noise sensor, a light sensor, and a wind speed sensor, with measurement ranges of 30dB~130dB, 0lux~10000lux, and 0m / s~20m / s, respectively). (2) Edge computing unit: including memory (16GB DDR4 memory + 1TB SSD local storage) and processor (NVIDIA Jetson AGX Xavier chip, 8-core CPU + 512-core GPU, computing power 32TOPS). The memory stores computer programs (including modules for data acquisition, image recognition, voice parsing, and permission verification). When the processor executes the computer program, it implements the inspection system described in any one of claims 1-7, supporting real-time data processing and offline operation. (3) Execution and feedback unit: including robot motion control system, robotic arm and interactive display screen, used to receive and execute the final control command output by edge computing unit, and at the same time provide feedback on interactive status information. (4) Eye tracking unit: An infrared eye tracking module (sampling rate 100Hz, detection accuracy 0.1°, set on the robot head, facing the face of the target interactive person) is used to collect the eye rotation angle. (5) Head tracking unit: An IMU (inertial measurement unit) with a sampling rate of 100Hz and an attitude angle accuracy of 0.05°, integrated into the eye tracking module, is used to collect head attitude angles. The wheeled structure of the inspection robot adopts a shock-absorbing design to adapt to the rugged road surface of the construction site; the lenses of the multimodal acquisition unit are equipped with waterproof and dustproof shells (IP67 protection level) to adapt to the harsh environment of the construction site; the edge computing unit supports 5G / 4G networking and can synchronize data to the remote monitoring center in real time; the eye tracking unit and the head tracking unit adopt an integrated design to reduce space occupation and improve positioning accuracy; the power supply of each unit is provided by the robot's built-in lithium battery (24V, 100Ah), which supports fast charging (fully charged in 2 hours). The specific implementation of the execution and feedback unit is as follows: (1) Robot motion control system: including motor driver, hub motor (4, power 500W / each) and navigation module (GPS+IMU fusion navigation), receives instructions from edge computing unit, controls robot movement (such as going to the target equipment location, adjusting inspection route), turning, stopping, navigation accuracy ±0.1m, supports obstacle avoidance function (detects obstacles through depth sensor and automatically avoids them). (2) Robotic arm: A 4-degree-of-freedom robotic arm (maximum extension length 1.2m, load 1kg) is set at the front end of the robot to execute specific operation instructions, such as opening the power distribution box switch, pressing the equipment button, etc.; the end of the robotic arm is equipped with a gripper (which can be adapted to switches and buttons of different sizes), controlled by a servo motor, with an operation accuracy of ±0.5mm. (3) Interactive display screen: A 10.1-inch touch screen (resolution 1920×1080, brightness 500cd / m², supports sunlight visibility) is set on the side of the robot's head to display the command recognition status (such as "recognizing voice command" "command recognition successful"), permission verification results (such as "permission verification passed" "no operation permission") and prompt information (such as "please confirm operation" "equipment failure, please contact maintenance"); it also supports receiving manual confirmation from the user. For example, when there is ambiguity in the command, the user can select a candidate device by touching the screen or manually input the control command. Implementation process: When the edge computing unit outputs the final control command "Open SB-001", the execution and feedback unit responds: The robot motion control system controls the robot to move to the left power distribution box (SB-001) position, adjusts the posture so that the robotic arm is aligned with the power distribution box switch; the robotic arm extends, the gripper grabs the switch, and flips it upward to open the switch; the interactive display screen displays "Operation executed successfully", and at the same time feeds back the operation status to the edge computing unit to complete the command execution. The robot motion control system has an obstacle avoidance response time of ≤0.5s, avoiding collisions with construction site equipment and personnel; the operating speed of the robotic arm is adjustable (0.1m / s~0.5m / s) to adapt to different precision operation requirements; the interactive display screen supports voice feedback and can broadcast prompts (such as "permission verification failed") through the speaker; the execution and feedback unit has a fault self-checking function, and when the robotic arm, display screen and other components malfunction, it promptly feeds back the fault information to the edge computing unit and displays the fault prompt on the display screen.

[0017] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart construction site safety inspection system based on image recognition, characterized in that: The inspection system includes: S1: Real-time sensor data is acquired through a multimodal acquisition unit installed on the inspection robot; the multimodal acquisition unit includes: a microphone array for acquiring audio stream data, an RGB visible light camera and an infrared thermal imaging camera for acquiring video stream data, an environmental situation sensor for acquiring environmental parameter data, and a depth sensor for assisting positioning; S2: Calculate the sound source location based on the audio stream data, identify the personnel target based on the video stream data, and spatiotemporally correlate the sound source location with the personnel target location to determine the target interaction personnel; S3: Extract the visual features of lip movements of the target interactive person and the speech activity features in the audio stream data, calculate the visual-auditory temporal consistency, and filter valid speech commands based on the temporal consistency; S4: Extract the gaze focus of the target interactive person through the eye-tracking unit and head-tracking unit on the inspection robot, and determine the target device by combining it with the preset device physical-semantic map; S5: Perform semantic fusion between the effective voice command and the target device to generate preliminary control commands; S6: Perform permission verification on the preliminary control command. If the verification passes, output the final control command; if the verification fails, refuse to execute and provide a prompt message.

2. The intelligent construction site safety inspection system based on image recognition according to claim 1, characterized in that: Step S2 includes: Acquire audio stream data collected by a microphone array, and use a time difference of arrival algorithm to calculate at least one sound source azimuth angle and its sound source confidence. The system acquires video stream data from an RGB visible light camera and an infrared thermal imaging camera, uses a target detection network to identify at least one person target, and calculates the position and visual confidence of each person target in the image coordinate system. Based on the pre-calibrated microphone array-camera extrinsic parameters, the azimuth angles of each sound source are mapped to the image coordinate system to generate sound source azimuth lines; Calculate the pixel distance between the position of each person target and the azimuth line of each sound source, and calculate the matching score based on the pixel distance, sound source confidence, and visual confidence. Select the person with the highest matching score that is greater than the preset threshold as the candidate target interaction person; determine the person with the highest overall matching score among the candidate target interaction persons as the target interaction person.

3. The intelligent construction site safety inspection system based on image recognition according to claim 1, characterized in that: In step S3, the calculation of visual-auditory temporal consistency includes the following sub-steps: S31: Analyze the facial ROI region of the target interactive person using a 3D convolutional neural network to obtain the start and end time points of lip movements [t1, t2]. S32: Perform speech activity detection on the audio stream data, extract speech activity features, and obtain the speech start and end time points [t3, t4]; S33: Calculate the time overlap rate between [t1, t2] and [t3, t4]. If the time overlap rate is greater than a preset threshold, the audio stream data is determined to be a valid voice command; if the time overlap rate is less than the preset threshold, it is determined to be environmental noise and discarded.

4. The intelligent construction site safety inspection system based on image recognition according to claim 1, characterized in that: The specific process of extracting the gaze focus of the target interacting person in step S4 is as follows: Obtain the eye rotation angle output by the eye tracking unit and the head pose angle output by the head tracking unit; Synthesize the line-of-sight direction vector in absolute space; Kalman filtering is applied to the line-of-sight vector sequence to suppress high-frequency jitter; A sliding window mid-value filter is applied to the filtered line-of-sight point sequence to output a stable line-of-sight focus. A stable gaze focus is input into a temporal convolutional network for multiple consecutive frames, and the gaze type is classified and output. The gaze type includes saccade state, gaze state and tracking state. The current gaze focus is marked as a valid gaze point only when the gaze type is a gaze state and the duration exceeds a preset duration threshold; Spatiotemporal clustering is performed on continuous valid gaze points to generate gaze clusters, with each gaze cluster corresponding to a candidate target device.

5. The intelligent construction site safety inspection system based on image recognition according to claim 4, characterized in that: Step S4, which involves determining the target device by combining a preset device physical-semantic map, specifically includes: Construct a three-level device physical-semantic graph, the three levels being scenario-level, device-level, and component-level; Obtain the robot's real-time pose information, determine the current scene area, and load the device list corresponding to the scene area; Project the viewpoint into three-dimensional space, calculate the distance to each device's bounding box, and generate a device-level candidate list and corresponding distance weights; If there is a unique candidate in the device-level candidate list or the difference in candidate weights is greater than a preset threshold, then the candidate is locked as the target device. If there are multiple candidate devices and the weight difference is less than a preset threshold, then component-level matching is entered, the distance between the gaze focus and the key components of each device is calculated, and the target device is determined by combining the gaze duration. If multiple candidate devices still exist after component-level matching, obtain the semantic parsing result of the valid voice command; Extract device type keywords and location descriptors from the semantic parsing results; The device type keywords are matched with the semantic attributes of the candidate devices, and candidate devices that meet the preset matching conditions are filtered. The selected candidate devices are spatially matched with location descriptors to determine the target device; Record the historical target devices and their spatial locations from the most recent N interactions (the historical target devices are the target devices identified during the interaction process of the inspection system, which are automatically recorded and stored by the system). When the Euclidean distance between the spatial location of the current candidate device and the spatial location of the historical target device is less than a preset distance threshold, the matching weight of the candidate device is increased. When the duration of fixation at the current point exceeds a preset long-term threshold, the historical target device is overwritten and updated to the current candidate device.

6. The intelligent construction site safety inspection system based on image recognition according to claim 1, characterized in that: Step S5 includes: The valid voice command is parsed to identify pronouns, omitted components, and ambiguous expressions. When a pronoun is identified, the target device determined in step S4 is obtained, and the pronoun is replaced with the device identifier of the target device. When an omitted component is identified, the omitted information is completed based on the task context or historical operation records. When ambiguous expressions are identified, multiple rounds of dialogue are initiated through the interactive interface to disambiguate and generate complete and structured preliminary control instructions.

7. The intelligent construction site safety inspection system based on image recognition according to claim 1, characterized in that: The specific process of performing permission verification on the preliminary control command in step S6 is as follows: Obtain the identity of the target user and query the corresponding user permission configuration file; Obtain the status information of the target device to determine its sensitivity level and current operating status; Based on user permission configuration, device sensitivity level, and operation type of initial control command, multi-dimensional permission verification is performed. The multi-dimensional verification includes at least user identity verification (verifying the legitimacy of the user's identity), device permission verification (verifying whether the user has the permission to operate the device), operation risk verification (verifying whether the operation command has security risks), and contextual security verification (verifying whether the current environment and device status are suitable for executing the command). When all dimensions pass the validation, the final control command is output; Before executing the final control command, an operation scenario snapshot is collected and stored. The operation scenario snapshot includes at least: the identity of the target interactive personnel, the content of the operation command, the execution timestamp, environmental parameter data, device status data, and audio and video clips. The snapshot of the operation scenario is associated and stored in the local log database, and synchronized to the remote audit server when the robot is connected to the network.

8. A smart construction site safety inspection device based on image recognition, characterized in that: The inspection device includes an inspection robot, which is equipped with a multimodal acquisition unit, an edge computing unit, an execution and feedback unit, an eye-tracking unit, and a head-tracking unit. The multimodal acquisition unit, the execution and feedback unit, the eye-tracking unit, and the head-tracking unit are electrically connected to the edge computing unit. The multimodal acquisition unit includes a ring microphone array, a wide-angle RGB visible light camera, an infrared thermal imaging camera, a depth sensor, and an environmental situation sensor; the ring microphone array is a ring structure with ≥6 microphones and is set in the robot's head; the environmental situation sensor is used to collect noise decibel values, light intensity, and wind speed data; The edge computing unit includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the inspection system as described in any one of claims 1-7. The execution and feedback unit is used to receive and execute the final control command output by the edge computing unit, and at the same time provide feedback on the interaction status information.

9. A smart construction site safety inspection device based on image recognition according to claim 8, characterized in that: The execution and feedback unit includes a robot motion control system, a robotic arm, and an interactive display screen; the interactive display screen is used to display the command recognition status, permission verification results, and prompt information, and is also used to receive manual confirmation operations from the user.

Images