Work completion degree identification method and system based on multi-modal visual features of multifunctional safety helmet

Abstract

The invention discloses a work completion degree identification method and system based on multi-modal visual features of a multifunctional safety helmet, and the method comprises the steps: obtaining the multi-modal visual data, including visible light image data, infrared thermal imaging data and depth image data, of a work scene in real time through a multi-modal visual collection unit disposed on the multifunctional safety helmet; respectively optimizing the collected visual data of each modal, and extracting effective visual features; a cross-modal fusion algorithm driven by an attention mechanism is adopted to carry out weighted fusion on each single-modal visual feature, and a fusion feature vector containing multi-dimensional information is generated; and training a work completion degree evaluation model based on the multi-modal data set marked with the work stage label and the completion degree value, dynamically updating the task rule base, inputting the fusion feature vector into the pre-trained work completion degree evaluation model, and outputting a completion degree parameter of the current task. According to the invention, the technical problems of insufficient information, limited visual angle and poor adaptability of a traditional monitoring mode are solved.

Description

Technical Field

[0001] This invention relates to the field of visual recognition technology, specifically to a method and system for recognizing work completion based on the multimodal visual features of a multifunctional safety helmet. Background Technology

[0002] In pipeline welding operations within the construction industry, the welding process is a core component of pipeline system installation. Its progress directly impacts the overall project schedule, while its quality affects subsequent operational safety (such as potential leaks and pressure failures). This process typically consists of three key stages: "preheating stage → welding execution stage → cooling and finishing stage." Quantifying the progress and monitoring compliance of each stage are core requirements for project management. However, existing welding completion monitoring technologies have the following significant shortcomings: 1. Single-mode data cannot cover the key features of the entire welding process. Existing systems mostly rely on a single visible light camera for monitoring, which cannot comprehensively capture multi-dimensional information about the welding scene: Preheating stage: It is impossible to detect in real time whether the preheating temperature of the pipeline meets the specifications (such as carbon steel pipelines needing to be preheated to above 100℃). Relying solely on manual handheld thermometers for spot checks is inefficient and prone to omissions. During the welding stage: It is impossible to determine the spatial positional accuracy of the weld (such as whether the gap between pipes is ≤2mm), and it is also difficult to identify whether the relative angle between the welding torch and the pipe is compliant. Cooling stage: It is impossible to monitor whether the cooling rate meets the crack prevention requirements (e.g., high-temperature welds need to be slowly cooled to room temperature), and rapid cooling can easily lead to crack defects.

[0003] 2. Limited monitoring perspective and data misalignment Fixed-position surveillance cameras have blind spots and cannot cover key operating areas from the welder's first-person perspective (such as weld seams inside pipes and contact points between the welding torch and the pipe). If independent multimodal devices (such as infrared thermometers + cameras) are used, the "scene at the same moment" information is often misaligned due to timestamp discrepancies (such as when the infrared thermometer detects an abnormal temperature, the visible light image has already shown that the welding has ended), affecting the accuracy of progress judgment.

[0004] 3. Poor adaptability to dynamic scenes In welding scenarios, tool iterations (such as switching from a regular welding torch to a pulse welding torch) and process adjustments (such as changes in welding current parameters) are frequent, making traditional systems with fixed rule bases unable to adapt to new features. The appearance and texture of the new welding torch do not match the original model, causing the "tool recognition during preparation" to fail. After the welding process was adjusted, the rules for preheating temperature threshold and cooling time were not updated, resulting in a deviation in the completion assessment.

[0005] 4. Insufficient accuracy in completion assessment Existing methods are mostly qualitative (such as "whether the welding stage has been entered"), lacking quantitative progress percentage output. The preheating stage can only determine "whether heating has started", but cannot quantify "the degree of preheating completion (e.g., 80% of the target temperature has been reached)"; The progress during the welding stage relies on manually counting the number of welds, ignoring the impact of the welding quality of individual welds (such as whether there is incomplete fusion) on the overall completion. The lack of weighted calculations of the overall progress (e.g., preheating accounts for 20%, welding for 60%, and cooling for 20%) makes it impossible to meet the needs of refined management. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the present invention aims to provide a method and system for work completion recognition based on multimodal visual features of a multifunctional safety helmet. This method integrates a visible light camera (capturing welding torch operation actions), an infrared thermal imaging sensor (monitoring preheating / welding temperature), and a depth sensor (detecting the spatial position of the weld seam) onto the safety helmet, covering key operation areas from the welder's first-person perspective. It utilizes a high-precision clock to synchronize multimodal data and improves data quality through adaptive filtering, point cloud completion, and other optimization techniques. An attention-driven cross-modal fusion algorithm dynamically focuses on core features of each stage (such as temperature features in the preheating stage and spatial features in the welding stage). Combined with a dynamically updated task rule base and a multi-task evaluation model, it achieves accurate output of "stage identification, stage completion, and overall completion" of the welding process. This solves the technical problems of insufficient information, limited perspective, and poor adaptability inherent in traditional monitoring methods, providing technical support for the refined management of welding processes in construction.

[0007] To solve the above problems, the technical solution adopted by the present invention is as follows: A method for recognizing work completion based on multimodal visual features of a multifunctional safety helmet includes the following steps: By utilizing the multimodal vision acquisition unit deployed on the multifunctional safety helmet, multimodal vision data of the work scene can be acquired in real time, including visible light image data, infrared thermal imaging data, and depth image data; The collected visual data for each modality were optimized and effective visual features were extracted. A cross-modal fusion algorithm driven by an attention mechanism is used to perform weighted fusion of visual features from each single modality to generate a fusion feature vector containing multi-dimensional information. A task completion assessment model is trained based on a multimodal dataset labeled with task stage tags and completion values. The task rule base is dynamically updated, and the fused feature vector is input into the pre-trained task completion assessment model to output the completion parameters of the current task.

[0008] Preferably, when acquiring multimodal visual data of the work scene in real time, it includes: The high-precision clock module built into the multi-functional safety helmet synchronizes the acquisition timestamps of visual data for each modality. Infrared thermal imaging data is converted into a temperature value matrix, depth image data is converted into 3D point cloud data, and white balance correction and exposure compensation are performed on visible light image data. The multi-modal visual acquisition unit deployed on the multi-functional safety helmet includes a visible light camera, an infrared thermal imaging sensor, and a depth sensor. The installation position and viewing angle of each sensor ensure coverage of the key operating surfaces of the work area.

[0009] Preferably, when optimizing the collected visual data for each modality separately, the following steps are included: An adaptive Gaussian filter is used to denoise the temperature value matrix, and the size of the filter window is dynamically adjusted based on the temperature gradient of adjacent pixels. Outlier removal is performed on the 3D point cloud data. The average distance between each point and its 10 neighboring points is calculated using the K-nearest neighbor algorithm. Points whose distance exceeds 1.5 times the standard deviation of the average distance are marked as outliers and removed. Then, the missing depth values ​​are filled in using the triangulation interpolation method. After white balance correction and exposure compensation, the CLAHE algorithm was further used to enhance the local contrast of the image, with the CLAHE clipping limit set to 0.02 and the grid size set to 8×8.

[0010] Preferably, when extracting effective visual features, the process includes: Extract temperature distribution histogram features, temperature extreme value features of key operating regions, and temperature gradient direction features from the optimized temperature value matrix; Extract curvature features, normal vector features, and 3D bounding box features of key operational objects from the optimized 3D point cloud data. The edge orientation histogram features, local binary pattern texture features, and color histogram features of the target are extracted from the optimized visible light image.

[0011] Preferably, when generating a fused feature vector containing multi-dimensional information, the following steps are included: Linear transformation is performed on each single-modal visual feature to map features of different dimensions to a high-dimensional feature space of the same dimension; A cross-modal attention module is constructed. The unified feature vectors of each modality are input, and the relevance score between each modal feature and the task objective is calculated. The score is adjusted by combining the information entropy of the feature, where the feature with higher information entropy has a lower weight. The adjusted score is normalized using the softmax function to obtain the attention weight coefficient of each modality. Each modality feature vector is multiplied by its corresponding attention weight coefficient, and then summed to generate a preliminary fusion feature vector. A nonlinear transformation is performed on the initial fused feature vector to generate a fused feature vector containing multi-dimensional information.

[0012] Preferably, when training the work completion evaluation model, the following are included: Construct a training dataset containing multiple labeled samples, and associate and fuse feature vectors, work stage labels and completion ground truth values ​​for each sample; A weighted multi-task loss function is adopted, where the stage classification loss has a weight of 0.4 and the completion regression loss has a weight of 0.6. Training is stopped when the classification accuracy is ≥96% and the regression error is ≤2% on 5 consecutive validation sets. The trained work completion assessment model is then quantized and compressed.

[0013] Preferably, when dynamically updating the task rule base, the following is included: Real-time monitoring of changes in the task scenario; When the changes in the scene exceed a preset threshold, the rule base update process is automatically triggered; We receive manually annotated multimodal data of new scenes and use incremental learning algorithms to fine-tune the model; The updated task rule base will be synchronized to ensure real-time inference.

[0014] Preferably, when outputting the completion parameters of the current task, the parameters include: The feature semantic mapping layer utilizes a fully connected network to map the fused feature vectors to a 1024-dimensional high-dimensional semantic space; The correlation weights between features are calculated by a self-attention feature filtering layer to highlight key features that are strongly correlated with work completion. By combining the stage determination logic in the task rule base with the dual output layer, the current task stage label and the completion percentage of the current stage are output simultaneously. The pre-trained work completion evaluation model adopts a multi-task deep neural network architecture that integrates features, including a feature semantic mapping layer, a self-attention feature filtering layer, and a dual output layer connected in sequence. The output of the completion parameter is a structured result, including: Stage identifier: The current task stage is represented by a preset code; Phase Completion: Presents the progress of this phase as a percentage; Overall completion rate: The overall progress calculated based on the weights of each stage.

[0015] A work completion recognition system based on multimodal visual features of a multifunctional safety helmet, which, during runtime, executes the above-mentioned method, including: Multimodal vision acquisition unit: Deployed on a multi-functional safety helmet, it is used to acquire visible light image data, infrared thermal imaging data and depth image data of the work scene in real time; Single-modal feature processing unit: Communicatively connected to the multimodal vision acquisition unit, including a data optimization subunit and a feature extraction subunit, wherein: Data optimization subunit: preprocesses and optimizes visual data for each modality; Feature extraction subunit: Extracts effective visual features from the optimized visual data of each modality; Cross-modal feature fusion unit: It communicates with the single-modal feature processing unit and uses an attention mechanism-driven cross-modal fusion algorithm to perform weighted fusion of visual features from each single modality to generate a fusion feature vector containing multi-dimensional information; Completion Evaluation Unit: Communicates with the cross-modal feature fusion unit, and has a built-in pre-trained task completion evaluation model and a dynamically updatable task rule base. It is used to input the fused feature vector into the evaluation model and output the completion parameters of the current task in combination with the task rule base.

[0016] Preferably, the multi-functional safety helmet has a built-in high-precision clock module for synchronizing the acquisition timestamps of visual data from each modality.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Multi-dimensional perception, covering key features of the entire welding process By integrating a visible light camera, an infrared thermal imaging sensor, and a depth sensor, comprehensive multi-dimensional information about the welding scene can be captured: Infrared thermal imaging: Real-time monitoring of temperature characteristics such as preheating temperature (e.g., carbon steel pipes preheated to above 100°C) and cooling rate, solving the problem that traditional systems cannot quantify the heating / cooling progress; Depth sensor: accurately detects the spatial location of welds (such as pipe butt gap ≤2mm, compliance of the relative angle between the welding torch and the pipe), filling the gap in spatial structure monitoring; Visible light cameras: identify tool types and operating actions (such as the status of welding torches) to help determine the stage of the process.

[0018] The three modal data complement each other, completely solving the shortcomings of traditional single-modal systems that cannot cover the key features of the entire welding process, and providing a comprehensive and reliable basis for completion assessment.

[0019] (2) First-person perspective monitoring without blind spots ensures data consistency in time and space. Optimized viewing angle: The sensor is deployed on the multi-functional safety helmet to cover the front operating area (core welding area) and the side tool placement area (critical area in the preparation / finishing stage) from the welder's first-person perspective, eliminating the blind spots of the fixed camera. Spatiotemporal synchronization: The high-precision clock module synchronizes the timestamps of each modality's data, avoiding evaluation deviations caused by data misalignment (such as when the visible light image shows that the welding has ended when the infrared detection temperature is abnormal), ensuring the effective correlation of multimodal information in time and space, and significantly improving the accuracy of evaluation.

[0020] (3) Dynamic scene adaptation to adapt to changes in technology and tools Introducing a dynamically updated task rule base and an incremental learning mechanism: Scene change monitoring: Automatically identify tool iterations (such as changing the pulse welding gun) or process adjustments (such as changes in preheating temperature threshold) through key feature matching degree (such as below 80%). Incremental learning fine-tuning: Using labeled data from new scenarios, the model is fine-tuned with an initial training learning rate of 1 / 10, allowing for rapid adaptation to new features without reconstructing the entire system; Rule base synchronization: Real-time updates of stage division standards and completion thresholds to ensure that the evaluation logic matches new scenarios.

[0021] This addresses the problem of poor adaptability in traditional fixed rule base systems, reduces maintenance costs, and maintains evaluation accuracy.

[0022] (4) Accurately quantify the completion rate output to support refined management. Based on a pre-trained multi-task evaluation model, the output structured completion parameters are: Stage identifier: Clearly indicate the current stage of the process (e.g., preparation / S1, execution / S2, completion / S3); Phase completion rate: Quantitative output of single-phase progress (e.g., S2 welding phase 75% complete); Overall completion rate: The overall progress is calculated based on the stage weights (e.g., preheating 20%, welding 60%, cooling 20%).

[0023] This enables a shift from qualitative judgment (such as "whether to enter the welding stage") to quantitative and precise assessment, meeting the needs of building construction welding processes for refined progress management and quality control, helping to improve construction efficiency and welding quality, and reducing errors in manual assessment.

[0024] (5) Improve management efficiency and reduce labor costs Automated monitoring replaces traditional manual sampling (such as handheld thermometers and manual counting of welds), reducing human intervention; real-time output of completion parameters provides managers with decision-making basis, optimizes resource allocation, significantly reduces labor costs, and improves project management efficiency.

[0025] In summary, this invention effectively solves the four core defects of traditional welding completion monitoring technology, provides technical support for the refined management of welding procedures in building construction, and has significant practicality and economic benefits.

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the steps of the work completion recognition method based on multimodal visual features of a multifunctional safety helmet according to an embodiment of the present invention. Figure 2 This is a flowchart of the visual data optimization process for each modality according to an embodiment of the present invention; Figure 3 This is a flowchart of the effective visual feature extraction process for various modal visual data in this invention embodiment; Figure 4 This is a flowchart of the fusion feature vector generation process according to an embodiment of the present invention; Figure 5 This is a flowchart of the completion parameter output in an embodiment of the present invention.

[0028] Figure 6 This is an interactive diagram of the work completion recognition system module based on the multimodal visual features of a multifunctional safety helmet, according to an embodiment of the present invention.

[0029] The following are the reference numerals: 21. Multimodal visual acquisition unit; 211. High-precision clock module; 212. Visible light camera; 213. Infrared thermal imaging sensor; 214. Depth sensor; 22. Single-modal feature processing unit; 221. Data optimization subunit; 222. Feature extraction subunit; 23. Cross-modal feature fusion unit; 24. Completion assessment unit; 241. Task rule base; 242. Work completion assessment model. Detailed Implementation

[0030] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0031] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0032] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0033] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0034] Example 1, see Figure 1 The present invention provides a step-by-step diagram of a method for identifying work completion. Figure 1 The method for recognizing work completion based on multimodal visual features of a multifunctional safety helmet, as shown, includes the following steps: S1. Multimodal visual data acquisition: Using the multimodal visual acquisition unit 21 deployed on the multifunctional safety helmet, multimodal visual data of the working scene is acquired in real time. The multimodal visual data includes visible light image data, infrared thermal imaging data and depth image data. S2. Single-modal feature extraction: Optimize the collected visual data of each modality separately, and extract effective visual features from the optimized visual data of each modality. S3. Cross-modal feature fusion: An attention-driven cross-modal fusion algorithm is used to weight and fuse the visual features of each single modality to generate a fused feature vector containing multi-dimensional information. S4. Completion Assessment Output: Based on a multimodal dataset labeled with work stage tags and completion values, a work completion assessment model 242 is trained, the task rule base 241 is dynamically updated, and the fused feature vector is input into the pre-trained work completion assessment model 242 to output the completion parameters of the current task.

[0035] In this embodiment of the invention, further explanation is needed regarding the following: Step S1 integrates visual data from multiple sensors to acquire multi-dimensional information about the work scene and ensures the spatiotemporal consistency of the data, providing a comprehensive and reliable input foundation for subsequent analysis. Step S2 optimizes data quality (denoising, completion, and enhancement) based on the characteristics of different modalities, then extracts key features that reflect the work status, highlighting the effective information of each modality. Step S3 dynamically allocates the weights of each modality feature through an attention mechanism, fusing features of different dimensions into a unified vector, fully utilizing the complementarity of multimodal information, and improving feature representation capabilities. Step S4 uses a pre-trained multi-task model combined with a dynamically updated rule base to analyze the fused features, outputting a structured result of task completion, supporting adaptive adjustments to scene changes.

[0036] In step S1 above, acquiring multimodal visual data of the work scene in real time includes: The high-precision clock module 211 built into the multi-functional safety helmet is used to synchronize the acquisition timestamps of the visual data of each modality to ensure the spatiotemporal consistency of the multimodal data. Preprocessing of raw data: Infrared thermal imaging data is converted into a temperature value matrix, depth image data is converted into 3D point cloud data, and white balance correction and exposure compensation are performed on visible light image data to eliminate the impact of ambient light changes on data quality; The multi-modal visual acquisition unit 21 deployed on the multi-functional safety helmet includes a visible light camera 212, an infrared thermal imaging sensor 213, and a depth sensor 214. The installation position and viewing angle of each sensor ensure coverage of key operating surfaces in the work area (such as the front operating area and the side tool placement area).

[0037] In this embodiment of the invention, it is necessary to further explain that multimodal data (visible light, infrared, and depth) come from different sensors. If there is a deviation in the acquisition timestamp, it will cause information misalignment in the "scene at the same moment" (e.g., when the infrared sensor detects an abnormal device temperature, the visible light image shows that the operation has ended). Therefore, synchronizing the timestamps of each modal data through the high-precision clock module 211 is a prerequisite for ensuring the accuracy of subsequent cross-modal feature fusion and state analysis, and allows information from different dimensions to form an effective correlation in time and space.

[0038] Infrared thermal imaging data converted into a temperature matrix: The raw pixel values ​​output by the infrared sensor cannot directly reflect the actual temperature state. Temperature is a key indicator for judging the operating status of equipment (such as overheating) and the progress of operation (such as the welding heating stage). Converting it into a temperature matrix can intuitively extract temperature features.

[0039] Depth image to 3D point cloud: The raw data of the depth sensor 214 is a two-dimensional depth map, which lacks spatial structure information. Converting it into a 3D point cloud can accurately describe the position, shape and spatial relationship of the object being operated on (such as the relative distance between tools and equipment), supporting the judgment of the standardization of the operation.

[0040] White balance and exposure compensation for visible light images: Frequent changes in lighting conditions in work environments (such as indoor / outdoor transitions or nighttime operations) can easily lead to overexposure or color casts in images, affecting the feature extraction of tools, equipment, and other targets. White balance correction eliminates color deviations, and exposure compensation adjusts brightness, ensuring stable image quality and providing a reliable foundation for subsequent texture and edge feature extraction.

[0041] A single sensor cannot cover all the key information of a work scene: a visible light camera 212 captures scene texture and color (to identify tool type and operation actions), an infrared thermal imaging sensor 213 detects temperature anomalies (to warn of equipment malfunctions and determine the progress of heating processes), and a depth sensor 214 acquires spatial structure (to verify whether the operation position is in place). The three complement each other to form a multi-dimensional perception system; and by installing the sensor on the safety helmet, it can cover the front operation area (core work area) and the side tool placement area (key area in the preparation / finishing stage) from the operator's first-person perspective, ensuring that the collected data is highly correlated with the actual operation behavior and avoiding information omissions caused by blind spots in the monitoring perspective.

[0042] These designs collectively provide a comprehensive, consistent, and high-quality input foundation for subsequent feature extraction, cross-modal fusion, and completion assessment, solving the problems of insufficient information and low reliability of traditional single-modal data in complex operational scenarios.

[0043] See Figure 2 The flowchart for optimizing visual data of each modality is as follows: In step S2 above, when optimizing the collected visual data of each modality, the process includes: Optimization of infrared thermal imaging data: Adaptive Gaussian filtering is used to denoise the temperature value matrix. The size of the filtering window is dynamically adjusted based on the temperature gradient of adjacent pixels (e.g., the window shrinks to 3×3 when the temperature gradient is greater than 5℃, and expands to 7×7 when it is less than 2℃) in order to suppress noise while preserving details of key temperature anomaly areas. Optimization of depth image data: Outlier removal is performed on the 3D point cloud data. The average distance between each point and its 10 neighboring points is calculated using the K-nearest neighbor algorithm. Points whose distance exceeds 1.5 times the standard deviation of the average distance are marked as outliers and removed. Subsequently, the missing depth values ​​are filled in using the triangulation interpolation method to ensure the integrity of the 3D point cloud. Optimization of visible light image data: After white balance correction and exposure compensation, the CLAHE (Contrast Limit Adaptive Histogram Equalization) algorithm is further used to enhance the local contrast of the image. The clipping limit of CLAHE is set to 0.02 and the grid size is set to 8×8 to highlight the edge and texture features of objects such as tools and equipment in the work scene.

[0044] In this embodiment of the invention, it is necessary to further explain that the temperature matrix collected by the infrared sensor is easily affected by environmental noise (such as equipment electromagnetic interference and ambient temperature fluctuations), causing details in abnormal temperature areas (such as equipment overheating points and welding heating zones) to be obscured. A fixed filtering window struggles to balance noise reduction and detail preservation—a window that is too large blurs temperature gradient changes, while a window that is too small results in poor noise reduction. Adaptive Gaussian filtering effectively solves this contradiction by dynamically adjusting the window size (based on the temperature gradient between adjacent pixels: a 3×3 window is used to preserve details for values ​​>5℃, and a 7×7 window is used to enhance noise reduction for values ​​<2℃), thus suppressing noise while preserving details in key abnormal temperature areas.

[0045] 3D point clouds generated by the depth sensor 214 often contain outliers (such as anomalous points caused by sensor error or scene occlusion) and missing depth values. The former interferes with spatial structure judgment (such as the relative positions of tools and equipment), while the latter destroys the integrity of the point cloud. The combination of the K-nearest neighbor algorithm (calculating the average distance of 10 neighboring points and removing outliers exceeding 1.5 standard deviations) and triangulation interpolation to complete missing values ​​ensures the accuracy and integrity of the point cloud data, providing a reliable foundation for subsequent extraction of spatial features such as curvature and normal vectors.

[0046] Frequent changes in lighting conditions during work (such as switching between indoor and outdoor environments or working at night) result in insufficient contrast in visible light images, causing blurry edges and textures of tools, equipment, and other targets. Traditional histogram equalization can easily lead to local overexposure or loss of detail, while the CLAHE algorithm (with a clipping limit of 0.02 and a grid size of 8×8) enhances local contrast, highlighting the edges and texture features of the target, thus solving the problem of unstable image quality under fluctuating lighting conditions.

[0047] These optimization steps provide high-quality input for subsequent single-modal feature extraction: the optimized infrared data accurately reflects temperature distribution and anomalies; the deep point cloud supports precise spatial structure analysis; and the visible light image clearly presents target details. High-quality single-modal features are a prerequisite for the effectiveness of cross-modal fusion (attention-driven weighted fusion), ultimately ensuring the accuracy of parameters such as stage markers and progress percentages output by the work completion evaluation model 242. In the overall process, these features play a core supporting role in solidifying the data foundation and improving the reliability of subsequent stages.

[0048] See Figure 3 The flowchart for extracting effective visual features from each modality of visual data is as follows: In step S2 above, extracting effective visual features from the optimized visual data for each modality includes: Extract temperature distribution histogram features, temperature extreme value features of key operating regions, and temperature gradient direction features from the optimized temperature value matrix; Extract curvature features, normal vector features, and 3D bounding box features of key operational objects from the optimized 3D point cloud data. The edge orientation histogram (HOG), local binary pattern (LBP) texture, and color histogram features of the target are extracted from the optimized visible light image.

[0049] In this embodiment of the invention, it is necessary to further explain that single-modal feature extraction is the core bridge connecting optimized multimodal data with cross-modal fusion. Its goal is to remove redundant information from each modality of data and extract effective quantitative indicators that are strongly correlated with the degree of completion, providing an accurate and interpretable input foundation for subsequent fusion algorithms and evaluation models. The necessity of extracting corresponding features is explained below for each modality: (1) Infrared temperature feature extraction In a work scenario, temperature changes are a core indicator of process progress and equipment status (such as temperature thresholds during welding heating and equipment preheating, or overheating caused by equipment malfunction). The optimized temperature matrix needs to extract three key features: Temperature distribution histogram: reflects the proportion of the overall temperature range and determines whether the process is within the normal temperature range (such as whether the temperature distribution in the preheating stage meets the standard). Extreme temperatures in critical areas: detect abnormal temperature points (such as equipment overheating warnings) or critical thresholds in processes (such as the highest temperature that welding needs to reach). Temperature gradient direction: Captures temperature change trends (such as whether the direction of heat diffusion during heating conforms to process specifications) to help determine the compliance of operations.

[0050] These characteristics are directly related to process progress (such as "60% completion of the heating stage") and status safety, and are the core basis for evaluating the completion of the execution stage.

[0051] (2) 3D point cloud feature extraction The accuracy of spatial positioning and shape during operation is crucial for the completion of assembly and installation processes (e.g., whether parts are installed correctly and tools are in the designated operating positions). The optimized point cloud needs to have its spatial structural features extracted. Curvature characteristics: Identify the surface shape of an object (such as the curvature of a tool or the edge curvature of a part) to determine whether the object being operated on conforms to specifications; Normal vector characteristics: reflect surface orientation (such as whether the equipment mounting surface is flat and the fit of parts), and verify the accuracy of operation; 3D bounding box: Determines the spatial range of the object being manipulated (e.g., whether the tool has been moved to a specified area, whether the part is within the assembly position), and quantifies whether the spatial position meets the criteria.

[0052] These spatial features are the core basis for judging whether the operation is in place, and directly affect the stage identification (such as whether the assembly stage has entered the final stage) and the completion calculation.

[0053] (3) Visible light image feature extraction Visible light images contain intuitive visual information for tool recognition and action judgment (such as whether the correct tool was held during preparation and whether the correct actions were performed during execution). The optimized image needs to extract three types of visual features: HOG edge orientation histogram: captures object outlines and action postures (such as hand gestures for holding tools, and the shape of equipment) to identify operational behaviors; LBP Local Binary Mode: Extracts surface texture (such as texture differences between different tools) to distinguish tool type or device status; Color histogram: Using color information (such as the color markings of tools, the indicator colors of devices) to quickly identify key objects (such as whether a tool of a specific color is being used).

[0054] These characteristics form the basis for identifying "operational objects and behaviors" and are the core inputs for evaluating the completion of the preparation phase (such as "whether the tools are fully prepared") and the closing phase (such as "whether the tools are put back in place").

[0055] The aforementioned feature extraction transforms the raw data from each modality into structured metrics directly related to task completion. This retains the unique advantages of single-modal data (temperature perception in infrared, spatial perception in depth, and visual perception in visible light) while providing a comparable feature base for subsequent cross-modal fusion (weighted through an attention mechanism). Without these features, the redundancy of the raw data would prevent the model from effectively learning task status information, ultimately affecting the accuracy of stage judgment and completion calculation.

[0056] See Figure 4 The flowchart for generating the fused feature vector is as follows: In step S3 above, generating the fused feature vector containing multi-dimensional information includes: a) Feature Space Unification: Perform linear transformation on each single-modal visual feature (temperature feature, point cloud feature, visible light image feature) to map features of different dimensions to a high-dimensional feature space of the same dimension, ensuring that features between modalities are comparable; b) Attention Weight Calculation: Construct a cross-modal attention module, input the unified feature vectors of each modality, and generate weights using the following method: Calculate the relevance score of each modality feature to the task objective (such as the cosine similarity with predefined work stage labels). The score is adjusted based on the information entropy of the feature (which reflects the uncertainty of the feature); the higher the information entropy, the lower the weight of the feature. The adjusted scores are normalized using the softmax function to obtain the attention weight coefficients for each modality; c) Weighted fusion: Multiply each modality feature vector by its corresponding attention weight coefficient, then sum them up to generate a preliminary fusion feature vector; d) Feature enhancement: Perform a nonlinear transformation (such as the ReLU activation function) on the initial fused feature vector to generate a fused feature vector containing multi-dimensional information, further enhancing the expressive power of the features.

[0057] In this embodiment of the invention, it is necessary to further explain that step S3 is the core step of cross-modal feature fusion. Its goal is to integrate single-modal features (temperature, point cloud, visible light) into a unified, multi-dimensional fused feature vector, providing accurate input for subsequent completion evaluation. The following are the technical background and functions of each sub-step: (1) Unified feature space Different modalities inherently differ in feature dimensions and distribution (e.g., temperature features may be low-dimensional statistics, point cloud features are high-dimensional spatial vectors, and visible light features are texture / edge features). Direct fusion would result in ineffective information integration due to "dimensionality mismatch." By using linear transformations to map all single-modal features to a high-dimensional feature space of the same dimension, the dimensional barriers between modalities are eliminated, ensuring the comparability of features across modalities and laying the foundation for subsequent attention weight calculation and weighted fusion.

[0058] (2) Calculation of attention weights In multimodal data, the importance of different modalities varies significantly at different stages of a task (e.g., infrared temperature features are more critical than visible light features during the heating process; visible light features are more important for tool identification during the preparation stage). Traditional fixed-weight fusion cannot adapt to this dynamic change, while attention mechanisms can dynamically allocate weights according to task requirements. Specifically: Relevance score: The correlation between modal features and task objectives (such as predefined work stage labels) is calculated using cosine similarity to identify the most valuable modalities in the current stage; Information entropy adjustment: Reduce the weights of features with high uncertainty (high information entropy) to improve the reliability of fused features; softmax normalization: generates normalized weight coefficients that can be directly weighted.

[0059] Ultimately, dynamic focusing is achieved—allowing fusion features to prioritize retaining modal information most relevant to the current task completion level, thereby improving the targeting of the fusion effect.

[0060] (3) Weighted fusion Weighted fusion is a classic method for cross-modal integration, but traditional fixed weights (such as average weights) cannot reflect the dynamic importance of modalities. Combining dynamic weights with attention mechanisms can make the fused features more closely match the task requirements.

[0061] The feature vectors of each modality are multiplied by their corresponding attention weights and then summed to generate a preliminary fused feature vector. This vector integrates multi-dimensional information (temperature distribution, spatial structure, visual texture) and highlights the key modalities of the current stage, thus solving the problem of "one-sided information in a single modality".

[0062] (4) Feature enhancement Initial feature fusion is the result of linear combination, with limited expressive power and unable to capture complex nonlinear patterns. Nonlinear transformations (such as ReLU activation) can enhance the nonlinear expressive power of features and improve the learning efficiency of subsequent models.

[0063] By enhancing the initial fusion features through nonlinear transformations such as ReLU, the expressive power of the features is further improved, allowing the fusion features to better reflect the complex patterns of the task (such as the nonlinear correlation between process progress and multimodal information), thus providing higher quality input for the completion evaluation model.

[0064] The four sub-steps of step S3 form a dynamic and precise cross-modal fusion process: The process involves eliminating dimensional barriers through a "unified space," focusing on key information through "attention weighting," integrating multimodal data through "weighted fusion," and enhancing expressive capabilities through "feature enhancement." The resulting fused feature vector contains multidimensional information such as temperature (process progress), space (operation location), and vision (tool recognition), and can dynamically adapt to changes in task stages. This provides core support for the accurate stage identification and completion percentage output by the completion assessment model in step S4.

[0065] In step S4 above, training the work completion evaluation model 242 includes: Construct a training dataset containing multiple labeled samples, and associate and fuse feature vectors, work stage labels and completion ground truth values ​​for each sample; A weighted multi-task loss function is adopted: the stage classification loss (cross-entropy) has a weight of 0.4, and the completion regression loss (mean squared error) has a weight of 0.6; Training is stopped when the classification accuracy is ≥96% and the regression error is ≤2% on 5 consecutive validation sets. The trained work completion evaluation model 242 was quantized and compressed (INT8 precision).

[0066] In this embodiment of the invention, it should be further explained that the training method provided in this embodiment is designed for the multi-task requirements of work completion evaluation and the constraints of edge deployment scenarios. The core objective is to build a high-precision, lightweight model to support real-time completion recognition at the safety helmet end. (1) Construction of multi-dimensional labeled dataset Work completion assessment needs to address two related tasks simultaneously: stage classification (e.g., "preparation / execution / closing stage") and progress regression (e.g., "execution stage 75% complete"). Therefore, the dataset needs to contain three types of related information: fused feature vectors (a unified representation of multimodal information), stage labels (classification task objectives), and completion ground truth (regression task objectives). This provides the model with foundational samples for multi-task learning, ensuring the model can simultaneously learn the ability to discriminate stages and quantify progress.

[0067] (2) Weighted multi-task loss function The tasks of stage classification and completion regression have different priorities: accurate quantification of completion (e.g., percentage) is the core indicator of the final output, while stage classification is a prerequisite for assisting progress calculation. Therefore, a weighted loss function (regression loss weight 0.6 > classification loss 0.4) is used to balance the learning intensity of the two tasks, ensuring that the model maintains the accuracy of stage classification while prioritizing the accuracy of completion prediction, thus avoiding performance imbalance caused by the dominance of single-task loss.

[0068] (3) Conditions for stopping training To ensure the model's generalization ability and performance stability, relatively strict validation metrics need to be set: classification accuracy ≥ 96% (small stage discrimination error) and regression error ≤ 2% (small completion prediction deviation) on the validation set for 5 consecutive rounds. The requirement of 5 consecutive rounds avoids accidental achievement of the criteria and ensures that the model can still output reliable results stably in complex working scenarios, meeting the accuracy requirements of industrial applications.

[0069] (4) INT8 quantization compression Model deployment is constrained by the device's computing resources (such as computing power and memory) and real-time requirements (the need to process multimodal data quickly). INT8 quantization can convert model parameters from floating-point to integer, significantly reducing model size and computation (usually compressed to 1 / 4 of the original model), while maintaining controllable accuracy loss, ensuring efficient model inference and adapting to hardware constraints.

[0070] These training designs collectively support the model's high accuracy (meeting the stringent requirements for completion assessment in industrial scenarios) and lightweight characteristics (adapting to hardware constraints), which are key steps in achieving real-time work completion recognition based on safety helmets.

[0071] In step S4 above, dynamically updating the task rule base 241 includes: Real-time monitoring of changes in task scenarios (such as tool type updates and operation process adjustments); When the scene changes beyond a preset threshold (such as the key feature matching degree being less than 80%), the rule base update process is automatically triggered. We receive manually labeled multimodal data of new scenes and use an incremental learning algorithm to fine-tune the model (the learning rate is set to 1 / 10 of the initial training rate). The updated task rule base 241 (including the stage division criteria and completion threshold for new scenarios) will be synchronized to ensure real-time inference.

[0072] In this embodiment of the invention, it is necessary to further explain that the multimodal visual acquisition unit 21 deployed on the multifunctional safety helmet acquires visible light, infrared, and depth multimodal data. After optimization (denoising, completion, enhancement), feature extraction (temperature distribution, point cloud curvature, HOG texture, etc.), and cross-modal fusion (attention weighting), the data is input into the completion evaluation unit 24. When the scene changes (such as tool updates), the matching degree between the fused features and the original rule base will decrease (e.g., the matching degree of key features is lower than 80%). At this time, the dynamic update mechanism is triggered: receiving new scene labeled data, using incremental learning to fine-tune the model (the learning rate is 1 / 10 of the initial training to avoid destroying the original performance), and synchronously updating the stage division standard and completion threshold in the rule base to ensure that the new features can be correctly parsed.

[0073] This mechanism is a core component of the completion evaluation unit 24 (alongside the pre-trained model), and relies on: The real-time data acquisition capability of the multimodal vision acquisition unit 21 enables scene change monitoring; The feature output of the single-modal feature processing + cross-modal fusion unit realizes the calculation of key feature matching degree; The incremental learning algorithm and rule synchronization logic enable dynamic updates of the model and rule base, ultimately ensuring the real-time performance and accuracy of the completion parameters (stage identifier, stage completion, overall completion).

[0074] This mechanism achieved the following core functions: Solving the problem of dynamic scenario adaptation: Breaking through the limitations of traditional fixed rule bases, enabling the system to cope with scenario changes such as tool iteration and process adjustment; Maintaining assessment accuracy: Updates are triggered by a key feature matching threshold (80%) to ensure that the completion assessment error is controllable in new scenarios; Balancing model stability and adaptability: Incremental learning uses a small learning rate for fine-tuning, which preserves the original model's ability to adapt to historical scenarios while quickly learning the patterns of new scenarios; Ensuring real-time reasoning: The rule base synchronization mechanism ensures that updated logic can be immediately applied to the reasoning process.

[0075] This mechanism is the key to achieving "continuous and reliable work completion recognition" in the entire technical solution, enabling the system to shift from static adaptation to dynamic self-adaptation and meet the complex needs of industrial operation scenarios.

[0076] See Figure 5 The flowchart for outputting the completion parameter is as follows: In step S4 above, when outputting the completion parameter of the current task, it includes: The feature semantic mapping layer utilizes a fully connected network to map the fused feature vectors to a 1024-dimensional high-dimensional semantic space; The correlation weights between features are calculated by using a self-attention feature filtering layer to highlight key features that are strongly correlated with work completion (such as tool usage status and changes in the operation area). By combining the phase determination logic in the task rule base 241 with the dual output layer (such as "device power on → execution phase begins"), the current task's current work phase label (such as "preparation phase", "execution phase", "completion phase") and the completion percentage value of the work phase (accurate to 1%) are output simultaneously. Among them, the pre-trained work completion evaluation model 242 adopts a multi-task deep neural network architecture with fused features, including a feature semantic mapping layer, a self-attention feature filtering layer and a dual output layer connected in sequence. The completion parameter is output in structured form, including: Stage identifier: The current task stage is represented by a preset code (such as "S1" or "S2"). Phase completion: Present the progress of this phase as a percentage (e.g., "S2: 75%"). Overall Completion: The overall progress calculated based on the weight of each stage (e.g., preparation stage accounts for 20%, execution stage accounts for 60%, and closing stage accounts for 20%).

[0077] In this embodiment of the invention, the following needs further explanation: 1. The overall architecture principle of the model The pre-trained work completion assessment model 242 employs a multi-task deep neural network architecture with fused features. Its core is a three-layered, sequentially connected module that transforms cross-modal fused features into structured completion parameters. This architecture simultaneously handles two tasks: "stage classification" (discrete labels) and "progress regression" (continuous percentages), leveraging inter-task correlations (e.g., stage division is a prerequisite for progress calculation) to improve overall inference accuracy.

[0078] II. Principles of Each Layer and Module 1. Feature semantic mapping layer Principle: The fused feature vector is a linear combination of multimodal information (such as a weighted sum of temperature, point cloud, and visible light features), which remains at a low-level feature level and is difficult to directly associate with abstract semantics such as "work stage" or "progress". By using a fully connected network to map the fused features to a 1024-dimensional high-dimensional semantic space, and by using linear transformation (weight matrix multiplication) + nonlinear activation (such as ReLU), the low-level features are transformed into high-dimensional semantic features containing abstract associations, providing a more expressive foundation for subsequent key feature selection and task output.

[0079] Function: Eliminate the dimensional barriers of multimodal features, integrate scattered modal information into a unified semantic representation, and enhance the correlation between features and work completion.

[0080] 2. Self-attention feature filtering layer Principle: High-dimensional semantic features contain a large amount of redundant information that is weakly correlated with completion (such as background noise and non-critical region features). A self-attention mechanism is used to calculate the correlation weights between features. For each feature vector, calculate the similarity (e.g., dot product) with all other feature vectors to obtain the importance score of that feature. Normalize the scores (e.g., using softmax) to generate weight coefficients; We use weighted summation of feature vectors to highlight key features strongly correlated with completion (such as tool usage status and changes in the operation area) and filter out redundant information.

[0081] Function: Focus on core features to improve the efficiency and accuracy of model inference and avoid judgment bias caused by information overload.

[0082] 3. Dual output layer Principle: Combining task rule base 241 (including stage judgment logic, such as "device powered on → execution stage begins"), two parallel tasks are implemented simultaneously: Stage classification sublayer: Fully connected + softmax activation is used to output the discrete stage label of the current task (such as preparation / execution / finalization), and the loss function is cross-entropy; Progress regression sublayer: Employs fully connected + linear activation to output the percentage of continuous completion of this stage (accurate to 1%), with the loss function being mean squared error.

[0083] Function: Utilize the business logic constraint model output of Task Rule Base 241 to ensure that the phase division conforms to the actual operation specifications, while meeting the dual requirements of "qualitative (phase) + quantitative (schedule)".

[0084] III. Structured Output Principle The output consists of three parts, and the principle is as follows: Stage identifier: Map the stage classification results of the dual output layer to a preset code (such as "S1" representing the preparation stage) to facilitate quick identification and storage by the system; Phase Completion: The output value (in percentage form) of the progress regression sub-layer is used directly to reflect the specific progress of the current phase; Overall Completion: Based on the preset weights of each stage (e.g., preparation 20%, execution 60%, and completion 20%), the weights of completed stages (if the stage completion is 100%, add all weights; if not completed, add weights proportionally to the progress) are added to the progress of the current stage multiplied by the weight to obtain the overall progress (e.g., if the preparation stage is 100% complete, add 20%; if the execution stage is 75% complete, add 60% × 75% = 45%, and the overall completion is 65%).

[0085] Purpose: Structured output combines intuitiveness and interpretability, meeting the needs of managers for rapid decision-making and system integration.

[0086] This model achieves precise transformation from multimodal fusion features to practical completion parameters through the process of "semantic enhancement → feature focusing → multi-task output → structured integration".

[0087] Example 2, see Figure 6 The interaction diagram of the work completion recognition system module is provided in this invention. Figure 6 The work completion recognition system based on multimodal visual features of a multifunctional safety helmet shown includes: a multimodal visual acquisition unit 21, a single-modal feature processing unit 22, a cross-modal feature fusion unit 23, and a completion evaluation unit 24; The multimodal vision acquisition unit 21 is deployed on the multifunctional safety helmet to acquire visible light image data, infrared thermal imaging data and depth image data of the work scene in real time.

[0088] The single-modal feature processing unit 22 is communicatively connected to the multimodal vision acquisition unit 21, and includes a data optimization subunit 221 and a feature extraction subunit 222, wherein: The data optimization subunit 221 performs preprocessing and optimization on visual data of each modality (such as infrared temperature matrix conversion, depth point cloud denoising and completion, visible light image enhancement, etc.). The feature extraction subunit 222 extracts effective visual features (such as temperature distribution features, point cloud curvature features, visible light target texture features, etc.) from the optimized visual data of each modality. The cross-modal feature fusion unit 23 is communicatively connected to the single-modal feature processing unit 22. It adopts an attention mechanism-driven cross-modal fusion algorithm to perform weighted fusion of each single-modal visual feature and generate a fusion feature vector containing multi-dimensional information.

[0089] The completion evaluation unit 24 is communicatively connected to the cross-modal feature fusion unit 23, and has the following built-in features: Pre-trained job completion evaluation model 242 (trained on a multimodal dataset labeled with job stage labels and completion values); The task rule library 241 can be dynamically updated (supporting incremental learning and rule synchronization triggered by scene changes). This is used to input the fused feature vector into the evaluation model and, combined with the task rule base 241, output the completion parameters of the current task (stage identifier, stage completion, and overall completion).

[0090] In one possible embodiment, the multi-functional safety helmet has a built-in high-precision clock module 211 for synchronizing the acquisition timestamps of the visual data for each modality.

[0091] In one possible embodiment, the multimodal vision acquisition unit 21 includes a visible light camera 212, an infrared thermal imaging sensor 213, and a depth sensor 214. The installation position and viewing angle of each sensor ensure coverage of the key operating surfaces of the work area.

[0092] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A work completion degree recognition method based on multi-functional safety helmet multi-modal visual features, characterized in that, The method comprises the following steps: Real-time acquisition of multi-modal visual data of the work scene by a multi-modal visual acquisition unit deployed on the multifunctional safety helmet, including visible light image data, infrared thermal imaging data and depth image data; Optimization of each modality of visual data collected, and extraction of effective visual features; Weighted fusion of each single modality visual feature by using a cross-modality fusion algorithm driven by an attention mechanism to generate a fusion feature vector containing multi-dimensional information; Training of a work completion degree evaluation model based on a multi-modal data set labeled with work stage labels and completion degree values, dynamic updating of a task rule library, and input of the fusion feature vector into the pre-trained work completion degree evaluation model to output the completion degree parameter of the current task.

2. The method of claim 1, wherein, When real-time acquisition of multi-modal visual data of the work scene is performed, the following steps are included: Synchronization of the collection time stamps of each modality of visual data by a high-precision clock module built in the multifunctional safety helmet; Conversion of the infrared thermal imaging data into a temperature value matrix, conversion of the depth image data into three-dimensional point cloud data, and execution of white balance correction and exposure compensation on the visible light image data; The multi-modal visual acquisition unit deployed on the multifunctional safety helmet includes a visible light camera, an infrared thermal imaging sensor and a depth sensor, and the installation positions and viewing angles of the sensors are set to ensure coverage of the key operation surface of the work area.

3. The method of claim 2, wherein, When each modality of visual data collected is optimized, the following steps are included: Adaptive Gaussian filtering is used to denoise the temperature value matrix, and the filter window size is dynamically adjusted based on the temperature gradient of adjacent pixels; Outlier removal is performed on the three-dimensional point cloud data, the average distance of each point and its 10 neighboring points is calculated by using the K nearest neighbor algorithm, points with a distance exceeding 1.5 times the standard deviation of the average distance are marked as outliers and removed, and then the triangular subdivision interpolation method is used to complete the missing depth values; After white balance correction and exposure compensation, the CLAHE algorithm is further used to enhance the local contrast of the image, wherein the clipping limit of CLAHE is set to 0.02, and the grid size is set to 8x8.

4. The method of claim 3, wherein, When the effective visual features are extracted, the following steps are included: Temperature distribution histogram features, temperature extreme value features and temperature gradient direction features of the key operation area are extracted from the optimized temperature value matrix; Curvature features, normal vector features and three-dimensional bounding box features of the key operation object are extracted from the optimized three-dimensional point cloud data; Edge direction histogram features, local binary pattern texture features and color histogram features of the target are extracted from the optimized visible light image.

5. The method of claim 1, wherein, When the fusion feature vector containing multi-dimensional information is generated, the following steps are included: Linear transformation is performed on each single modality visual feature to map features of different dimensions to a high-dimensional feature space of the same dimension; A cross-modality attention module is constructed, the unified modality feature vectors are input, the correlation scores of each modality feature and the task target are calculated, and the scores are adjusted combined with the information entropy of the features, wherein the higher the information entropy, the lower the weight of the feature; the adjusted scores are normalized using the softmax function to obtain the attention weight coefficients of each modality. The modal feature vectors are multiplied by the corresponding attention weight coefficients, and then accumulated and summed to generate a preliminary fusion feature vector; A nonlinear transformation is performed on the preliminary fusion feature vector to generate a fusion feature vector containing multi-dimensional information.

6. The method of claim 1, wherein, When training the work completion degree evaluation model, the following steps are included: A training data set containing multiple labeled samples is constructed, and each sample is associated with a fusion feature vector, a work stage label, and a true value of completion degree; A weighted multi-task loss function is used, with a stage classification loss weight of 0.4 and a completion degree regression loss weight of 0.6; When the classification accuracy on the validation set is ≥96% and the regression error is ≤2% for 5 consecutive rounds, the training is stopped; The trained work completion degree evaluation model is quantized and compressed.

7. The method of claim 6, wherein, When dynamically updating the task rule base, the following steps are included: Real-time monitoring of changes in the task scene; When the scene changes exceed the preset threshold, automatically trigger the rule base update process; Receive manually labeled new scene multi-modal data, and fine-tune the model using incremental learning algorithm; The updated task rule base is synchronized to ensure real-time inference.

8. The method of claim 7, wherein, When outputting the completion degree parameters of the current task, the following steps are included: Through the feature semantic mapping layer, the fusion feature vector is mapped to a 1024-dimensional high-dimensional semantic space using a fully connected network; Through the self-attention feature screening layer, the correlation weight between features is calculated to highlight the key features strongly associated with the work completion degree; Through the dual output layer combined with the stage determination logic in the task rule base, the work stage label and the completion degree percentage value of the current task are output simultaneously; The pre-trained work completion degree evaluation model uses a multi-task deep neural network architecture with fusion features, including a feature semantic mapping layer, a self-attention feature screening layer, and a dual output layer connected in sequence; The output form of the completion degree parameters is a structured result, including: Stage identification: using a preset encoding to represent the current task stage; Stage completion degree: presenting the progress of the stage in percentage form; Overall completion degree: the comprehensive progress calculated based on the weights of each stage.

9. A work completion degree recognition system based on multi-modal visual features of a multi-functional safety helmet, characterized in that, When running, the method of claim 1 is executed, including: Multimodal visual acquisition unit: deployed in a multifunctional safety helmet, used to acquire visible light image data, infrared thermal imaging data, and depth image data of the work scene in real time; Single-modal feature processing unit: in communication connection with the multimodal visual acquisition unit, including a data optimization subunit and a feature extraction subunit, wherein: Data optimization subunit: pre-processing and optimizing each modal visual data; Feature extraction subunit: extracting effective visual features from the optimized each modal visual data; Cross-modal feature fusion unit: in communication connection with the single-modal feature processing unit, using an attention mechanism driven cross-modal fusion algorithm to weight and fuse each single-modal visual feature to generate a fusion feature vector containing multi-dimensional information; Completion degree evaluation unit: in communication connection with the cross-modal feature fusion unit, built-in pre-trained work completion degree evaluation model and dynamically updateable task rule base, used to input the fusion feature vector into the evaluation model and output the completion degree parameters of the current task in combination with the task rule base.

10. The system of claim 9, wherein, The multifunctional safety helmet is internally provided with a high-precision clock module for synchronizing the collection time stamps of the visual data of various modes.

Images