Self-rescuer blind wearing action intelligent examination method and system based on end-side AI
Patent Information
- Application Number
- CN202611322073.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-03
AI Technical Summary
Traditional self-rescue device blind wearing assessment relies on manual supervision, which is highly subjective, inefficient, lacks a precise traceability mechanism, and is limited by network dependence, making it difficult to achieve efficient and accurate action recognition and assessment in coal mines.
Standard actions are converted into text feature vectors and embedded on the device. Using cross-modal multimodal models and temporal sliding window algorithms, millisecond-level frame extraction and real-time inference are achieved on mobile devices. Through image-text feature similarity retrieval, the start and end timestamps of micro-actions are accurately recorded, and an evaluation report with a chain of evidence is generated.
It achieves millisecond-level accurate recognition of blind self-rescue device wearing actions on ordinary mobile devices, reduces system deployment costs, eliminates subjective bias and visual omissions caused by manual proctoring, supports parallel automated assessment on multiple terminals, and generates fine-grained assessment reports.
Smart Images
Figure CN123033285A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent assessment technology, and in particular to an intelligent assessment method and system for blind wearing of self-rescue devices based on edge AI. Background Technology
[0002] Underground coal mining is a high-risk industry. In the event of disasters such as gas explosions, coal and gas outbursts, roof falls, or fires, the ability of workers to correctly don self-rescue devices within a very short time directly impacts their safety. According to national coal mine safety regulations, underground workers must master the practical skill of donning a self-rescue device blindly within 30 seconds. However, traditional blind-donning assessments primarily rely on on-site supervision and scoring by examiners. This model has several problems: High subjectivity and inconsistent standards: Human proctoring is easily affected by the subjective factors of the examiners, and different examiners may have different definitions of the standardization of actions and the completion time. Inefficiency and high labor costs: Coal mining enterprises have a large number of employees, and manual assessment requires a lot of invigilation manpower and time, making it difficult to cope with the frequent and regular assessments of a large number of employees. Lack of a precise traceability mechanism: Blindly wearing the device involves multiple rapid micro-movements such as "moving it to the front, opening the cover, deploying the airbag, biting the mouthpiece, and clipping the nose clip". These movements are easily missed by the naked eye, and there is no objective data record for post-event auditing or targeted review and teaching. Network dependency limitations: In coal mine assessment scenarios (such as underground entrances or remote training bases), network signals are often unstable or unable to connect to cloud servers, making it impossible to guarantee real-time performance and continuity when relying on large cloud-based models for recognition. To address the aforementioned technical deficiencies, a solution is proposed. Summary of the Invention
[0003] The purpose of this invention is to transform complex action recognition into lightweight image-text feature similarity retrieval by pre-converting standard actions into text feature vectors and embedding them on the device side. This eliminates the dependence on cloud servers and network environments, enabling millisecond-level frame extraction and real-time inference on ordinary mobile devices. This reduces system deployment costs and assessment computing power overhead. Furthermore, by introducing a cross-modal multimodal model and a temporal sliding window anti-shake clustering algorithm, it can accurately record the start and end timestamps of each micro-action at the millisecond level, accurately locate fine-grained violations such as missing actions, reversed order, and transition stuttering, and form an assessment report with a chain of evidence. It also supports parallel automated assessment on multiple terminals.
[0004] To achieve the above objectives, the present invention adopts the following technical solution: a method for intelligent assessment of blind wearing actions of self-rescue devices based on edge AI, comprising the following steps: S1. Convert the N standard action steps of the blind wearing test of coal mine self-rescue device into corresponding Chinese behavioral description text, and generate standard text feature vectors that correspond one-to-one with the N standard action steps through a text encoder. Store the generated standard text feature vectors in the terminal database of the mobile device. S2. Use the camera of the mobile device to capture a video of the blind wearing process of the self-rescue device by the person to be assessed, and extract a continuous image frame sequence by uniformly extracting the process video at equal intervals on the mobile device. S3. Using the image encoder of the multimodal model deployed on the mobile device, each frame of the image frame sequence is mapped to a unified semantic space in real time to generate a corresponding image feature vector. The cosine similarity between the image feature vector and each standard text feature vector in the terminal database is calculated, and the standard action step corresponding to the highest cosine similarity is used as the predicted action label of the frame image. S4. Establish a time-series sliding window, perform anti-shake filtering and continuous action segment clustering on the frame sequence with predicted action labels, extract the execution order list of the actual action steps, and the start and end timestamps of each action step. S5. Compare the extracted action step execution sequence list with the preset standard action step sequence for sequential consistency. At the same time, calculate the total blind wearing time based on the timestamp of each action step, comprehensively determine whether the candidate passes the assessment, and output fine-grained violation deduction items.
[0005] Furthermore, the specific process for generating standard text feature vectors is as follows: According to the coal mine safety regulations and the standard operating procedures for self-rescue devices, the complete blind-wearing process is deconstructed into N logically independent standard action steps; For each standard action step n, where n ∈ [1, N], construct a step text set containing the main description text and K synonymous extended description texts: ; A text encoder using a pre-trained multimodal model processes each text in the text set Tn at each step. Feature extraction is performed, where k ∈ [1, K], to generate the corresponding D-dimensional text feature vector. ; Mean pooling is used to fuse all D-dimensional text feature vectors of the nth standard action step to calculate the coarse feature vector of that standard action step. : ; For the fused coarse feature vector L2 norm normalization is performed to obtain the final standard text feature vector. : ; In the formula, Represents coarse feature vectors The L2 norm.
[0006] Furthermore, the specific process for obtaining a continuous sequence of image frames is as follows: At the start of the assessment, the assessment application software on the mobile device calls the underlying camera hardware interface of the system to turn on the rear camera and uses the built-in fixed-frequency sampling timer to perform uniform frame sampling at equal intervals. During sampling, the hardware acquisition callback function extracts a frame from the video stream buffer at fixed time intervals. At the moment of extraction, it reads the system monotonic hardware clock and directly binds the current millisecond-level absolute timestamp as metadata to the image frame, generating a sampling frame with time attributes. The sampling frame is quickly converted into an RGB three-channel matrix image in memory using the graphics processing chip of the mobile device. The image is then cropped with the candidate's main body area as the center. The RGB three-channel matrix image is normalized and scaled to the standard input size required by the multimodal model image encoder. Mean elimination and variance scaling are performed on the scaled RGB three-channel matrix image to map the pixel values to the specified standard value range, thus obtaining the preprocessed image frame. The pre-processed image frames are pushed into a pre-allocated circular dynamic memory buffer in the mobile device's memory in chronological order, forming a continuous sequence of image frames strictly arranged according to the timeline.
[0007] Furthermore, the specific process for obtaining the predicted action label of the image is as follows: The continuous image frame sequence obtained in S2 is input into an image encoder deployed on the GPU of a mobile device. The image encoder performs forward inference on the m-th frame and outputs the corresponding original image feature vector. ; For the feature vector of the original image L2 norm normalization is performed to obtain the image feature vector. : ,in, Represents the L2 norm; Read N pre-stored, normalized standard text feature vectors from the client-side database, and convert the image feature vectors... With the text feature vector matrix stored in a fixed location Multiplication allows for the parallel generation of the similarity score set of the m-th frame image relative to all N standard action steps in a single computation. : ; After obtaining the set of similarity scores for all standard action steps for the m-th frame image, search for the maximum similarity score in the set of similarity scores and obtain its corresponding standard action step index. Preset reliability threshold The maximum similarity score Compared with the preset confidence threshold By comparison, the final predicted action label for the m-th frame image is assigned according to the following formula. : ; In the formula, = This indicates that the image frame was successfully identified as the first... A standard action procedure; = This indicates that the image frame has insufficient action confidence and is classified as a background, transition, or invalid action frame.
[0008] Furthermore, the specific process of obtaining frame sequences with predicted action labels and constructing a temporal sliding window for anti-shake filtering is as follows: Set a time-series sliding window that covers a fixed number of frames, move it frame by frame along the time axis, use the mode filtering logic within the time-series sliding window, count the action tags that appear most frequently within the time-series sliding window as high-frequency tags, and replace the original tags of the center frame of the time-series sliding window with high-frequency tags to obtain the filtered frame sequence. Temporally continuous clustering is performed on the filtered frame sequence. The frame sequence is traversed, and the frame sequences that are temporally continuous and have the same predicted action label are merged into independent action event segments. For each action event segment generated by clustering, the absolute timestamp bound to its first frame is extracted and recorded as the start timestamp of the action event segment, and the absolute timestamp bound to its last frame is extracted as the end timestamp of the action event segment. The standard action numbers corresponding to each action event segment are extracted in chronological order and arranged sequentially to form a linked list of the execution order of the actual action steps performed by the examinee.
[0009] Furthermore, the specific process for determining whether a candidate has passed the assessment is as follows: A linked list of standard blind-wearing action steps is pre-stored in the terminal database. Using the longest common subsequence matching logic and finite state machine verification rules, the execution order linked list of the actual action steps performed by the examinee is compared with the standard linked list. Each necessary action step in the standard order linked list is retrieved one by one. If a certain standard step is not found in the actual action step linked list, the violation of missing key action step is triggered and recorded. Based on the unidirectional irreversible transition rule of the finite state machine, the order of each action step in the linked list is determined. If a subsequent action step is detected to occur before a previous action step, the action order reversal violation is triggered and recorded. If the same action step in the actual action chain is repeatedly and discontinuously triggered, then the action duplication violation will be triggered and recorded. Extract the start timestamp of the first valid non-zero action segment that actually occurs as the start time of the assessment, and extract the end timestamp of the last valid standard action segment as the end time of the assessment. Calculate the time difference between the end time and the start time, which is the total blind wearing time of the person to be assessed. Calculate the interval between two adjacent action segments. If the transition interval between two adjacent actions exceeds the preset pause threshold, it is determined that the candidate hesitated during the action transition, triggering and recording the violation of excessively slow action transition. A maximum score is preset, and a multi-level deduction rule base and a veto mechanism are established: If the calculated total time for blind wearing exceeds the preset limit, regardless of whether the action is standardized, the assessment will be deemed as failing due to timeout. If a critically vital life-saving action is found to be missing, the assessment will be deemed a major safety hazard and will be deemed unqualified. If the veto item is not triggered, points will be deducted based on the rule base matched with the identified violation type: a preset action missing score will be deducted for each missing non-core action step. Each time the order of actions is reversed, a pre-set score for sequence error will be deducted; Each time a transition between actions is delayed and timed out, a preset delay score will be deducted. Subtract the sum of the points deducted for each violation from the full score to obtain the final practical score; If the final practical score reaches or exceeds the preset passing score and does not trigger a veto, the assessment is considered passed; otherwise, the assessment is considered failed.
[0010] This invention also provides an intelligent assessment system for blind wearing actions of self-rescue devices based on edge AI, including an assessment standard pre-storage unit, an image extraction unit, an image comparison unit, an action assessment unit, and an assessment evaluation unit, wherein: The assessment standard pre-storage unit is used to convert the N standard action steps of the blind wearing assessment of coal mine self-rescue device into corresponding Chinese behavioral description text, and generate standard text feature vectors that correspond one-to-one with the N standard action steps through the text encoder. The generated standard text feature vectors are then stored in the terminal database of the mobile device. The image extraction unit is used to capture a video of the blind wearing process of the self-rescue device by the camera of the mobile device, and to extract a continuous image frame sequence by uniformly extracting the process video at equal intervals on the mobile device. The image comparison unit is used to map each frame of the image frame sequence to a unified semantic space in real time using the image encoder of the multimodal model deployed on the mobile device, generate the corresponding image feature vector, calculate the cosine similarity between the image feature vector and each standard text feature vector in the terminal database, and use the standard action step corresponding to the highest cosine similarity as the predicted action label of the frame image. The action assessment unit is used to establish a time-series sliding window, perform anti-shake filtering and continuous action segment clustering on frame sequences with predicted action labels, extract the execution order list of the actual action steps, and the start and end timestamps of each action step. The assessment unit compares the extracted action step execution sequence list with the preset standard action step sequence for sequential consistency. It also calculates the total blind wearing time based on the timestamp of each action step, comprehensively determines whether the candidate passes the assessment, and outputs fine-grained violation deduction items.
[0011] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: This edge AI-based intelligent assessment method and system for blind-wearing self-rescue devices pre-converts standard actions into text feature vectors and embeds them on the edge, transforming complex action recognition into lightweight image-text feature similarity retrieval. It eliminates reliance on cloud servers and network environments, enabling millisecond-level frame extraction and real-time inference on ordinary mobile devices, reducing system deployment costs and assessment computing power. Furthermore, it introduces cross-modal multimodal models and temporal sliding window anti-shake clustering algorithms, eliminating subjective bias and human error inherent in traditional manual proctoring. It can accurately record the start and end timestamps of each micro-action at the millisecond level, precisely locating fine-grained violations such as missing actions, reversed sequences, and transition stutters, generating assessment reports with a chain of evidence. It also supports parallel automated assessment across multiple terminals, solving the problem of insufficient proctoring manpower when conducting frequent assessments of large numbers of personnel in coal mining enterprises. Attached Figure Description
[0012] Figure 1 A schematic diagram of the overall method flow of the present invention is shown; Figure 2 A schematic diagram of the overall structure of the present invention is shown. Detailed Implementation
[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] Example 1: like Figure 1 As shown, a method for intelligent assessment of blind wearing actions of self-rescue devices based on edge AI includes the following steps: S1. Convert the N standard action steps of the blind wearing test of coal mine self-rescue device into corresponding Chinese behavioral description text, and generate standard text feature vectors that correspond one-to-one with the N standard action steps through a text encoder. Store the generated standard text feature vectors in the terminal database of the mobile device. The specific process for generating standard text feature vectors is as follows: Based on the coal mine safety regulations and the standard operating procedures for self-rescue devices (such as chemical oxygen / compressed oxygen self-rescue devices), the complete blind-wearing process is deconstructed into N logically independent standard action steps; Specifically: Let N=5, and the steps of each action are defined as follows: Step 1: Move the self-rescue device from the right side to the front with both hands; Step 2: Open the top cover with your right hand and discard the top cover; Step 3: Deploy the airbag with both hands and pull the detonation fuse; Step 4: Put the mouthpiece into your mouth and bite down with your teeth; Step 5: Use a nose clip to hold your nose in place and continue breathing.
[0015] For each standard action step n, where n ∈ [1, N], construct a step text set containing the main description text and K synonymous extended description texts: ; For example, for step 2 "open the top cover with the right hand and discard the top cover" (n=2), the constructed text set T2 includes: t{2,1}: "Open the top cover with the right hand and discard the top cover" (main description); t{2,2}: "Use the right hand to unlock the metal buckle and lift the top cover of the self-rescue device"; t{2,3}: "Open the self-rescue device cover and throw it away".
[0016] A text encoder using a pre-trained multimodal model processes each text in the text set Tn at each step. Feature extraction is performed, where k ∈ [1, K], to generate the corresponding D-dimensional text feature vector. ; Mean pooling is used to fuse all D-dimensional text feature vectors of the nth standard action step to calculate the coarse feature vector of that standard action step. : ; For the fused coarse feature vector L2 norm normalization is performed to obtain the final standard text feature vector. : ; In the formula, Represents coarse feature vectors The L2 norm.
[0017] The standard text feature vector has the following technical effects: No need for online text feature calculation: During the evaluation process, the edge device only needs to perform unidirectional feature extraction on the image frame, which completely eliminates the computational overhead and resource consumption of running a heavy text encoder on the edge. Improve cross-modal matching robustness: By using synonym expansion and mean pooling, semantic deviations between different text expressions are smoothed, enabling the generated standard vectors to more accurately represent action features in visual images.
[0018] S2. Use the camera of the mobile device to capture a video of the blind wearing process of the self-rescue device by the person to be assessed, and extract a continuous image frame sequence by uniformly extracting the process video at equal intervals on the mobile device. The specific process for obtaining a continuous sequence of image frames is as follows: At the start of the assessment, the assessment application software on the mobile device calls the underlying camera hardware interface of the system (such as the Camera2 API of the Android system or the AVFoundation framework of the iOS system) to turn on the rear camera and perform uniform frame sampling at equal intervals through the built-in fixed frequency sampling timer. During sampling, the hardware acquisition callback function extracts a frame from the video stream buffer at fixed time intervals. At the moment of extraction, it reads the system monotonic hardware clock and directly binds the current millisecond-level absolute timestamp as metadata to the image frame, generating a sampling frame with time attributes. The sampling frame is quickly converted into an RGB three-channel matrix image in memory using the graphics processing chip of the mobile device. The image is then cropped with the candidate's main body area as the center. The RGB three-channel matrix image is normalized and scaled to the standard input size required by the multimodal model image encoder. Mean elimination and variance scaling are performed on the scaled RGB three-channel matrix image to map the pixel values to the specified standard value range, thus obtaining the preprocessed image frame. The pre-processed image frames are pushed into a pre-allocated circular dynamic memory buffer in the mobile device's memory in chronological order, forming a continuous sequence of image frames strictly arranged according to the timeline.
[0019] The circular buffer adopts a first-in-first-out memory scheduling mechanism. The maximum capacity of the circular buffer is set to accommodate dozens of image frames. When the multimodal inference module on the edge reads and completes the feature extraction of a certain image frame, it immediately destroys the RGB matrix data of that frame in memory and releases its memory space. At the same time, newly sampled image frames are sequentially written to the end of the buffer, thereby ensuring that the memory usage of the mobile device remains stable during long-term recording, and completely avoiding memory overflow and interface lag.
[0020] S3. Using the image encoder of the multimodal model deployed on the mobile device, each frame of the image frame sequence is mapped to a unified semantic space in real time to generate a corresponding image feature vector. The cosine similarity between the image feature vector and each standard text feature vector in the terminal database is calculated, and the standard action step corresponding to the highest cosine similarity is used as the predicted action label of the frame image. The specific process for obtaining the predicted action label of an image is as follows: The continuous image frame sequence obtained in S2 is input into an image encoder deployed on the GPU of a mobile device. The image encoder performs forward inference on the m-th frame and outputs the corresponding original image feature vector. ; For the feature vector of the original image L2 norm normalization is performed to obtain the image feature vector. : ,in, Represents the L2 norm; Read N pre-stored, normalized standard text feature vectors from the client-side database, and convert the image feature vectors... With the text feature vector matrix stored in a fixed location Multiplication allows for the parallel generation of the similarity score set of the m-th frame image relative to all N standard action steps in a single computation. : ; After obtaining the set of similarity scores for all standard action steps for the m-th frame image, search for the maximum similarity score in the set of similarity scores and obtain its corresponding standard action step index. Preset reliability threshold The maximum similarity score Compared with the preset confidence threshold By comparison, the final predicted action label for the m-th frame image is assigned according to the following formula. : ; In the formula, = This indicates that the image frame was successfully identified as the first... A standard action procedure; = This indicates that the image frame has insufficient action confidence and is classified as a background, transition, or invalid action frame.
[0021] S4. Establish a time-series sliding window, perform anti-shake filtering and continuous action segment clustering on the frame sequence with predicted action labels, extract the execution order list of the actual action steps, and the start and end timestamps of each action step. The specific process of obtaining frame sequences with predicted action labels and constructing a temporal sliding window for anti-shake filtering is as follows: Set a time-series sliding window that covers a fixed number of frames, move it frame by frame along the time axis, use the mode filtering logic within the time-series sliding window, count the action tags that appear most frequently within the time-series sliding window as high-frequency tags, and replace the original tags of the center frame of the time-series sliding window with high-frequency tags to obtain the filtered frame sequence. Temporally continuous clustering is performed on the filtered frame sequence. The frame sequence is traversed, and the frame sequences that are temporally continuous and have the same predicted action label are merged into independent action event segments. For each action event segment generated by clustering, the absolute timestamp bound to its first frame is extracted and recorded as the start timestamp of the action event segment, and the absolute timestamp bound to its last frame is extracted as the end timestamp of the action event segment. The standard action numbers corresponding to each action event segment are extracted in chronological order and arranged sequentially to form a linked list of the execution order of the actual action steps performed by the examinee.
[0022] S5. Compare the extracted action step execution sequence list with the preset standard action step sequence for sequential consistency. At the same time, calculate the total blind wearing time based on the timestamp of each action step, comprehensively determine whether the candidate passes the assessment, and output fine-grained violation deduction items.
[0023] A linked list of standard blind-wearing action steps is pre-stored in the terminal database. Using the longest common subsequence matching logic and finite state machine verification rules, the execution order linked list of the actual action steps performed by the examinee is compared with the standard linked list. Each necessary action step in the standard order chain is searched one by one. If a certain standard step is not found in the actual action step chain (e.g., the "nose clip" action is not found), then the violation of missing key action step is triggered and recorded. Based on the unidirectional irreversible transition rule of the finite state machine, the order of each action step in the linked list is determined. If a subsequent action step is detected to occur before a previous action step, the action order reversal violation is triggered and recorded. If the same action step in the actual action chain is repeatedly and discontinuously triggered, then the action duplication violation will be triggered and recorded. Extract the start timestamp of the first valid non-zero action segment that actually occurs as the start time of the assessment, and extract the end timestamp of the last valid standard action segment as the end time of the assessment. Calculate the time difference between the end time and the start time, which is the total blind wearing time of the person to be assessed. Calculate the interval between two adjacent action segments. If the transition interval between two adjacent actions exceeds the preset pause threshold, it is determined that the candidate hesitated during the action transition, triggering and recording the violation of excessively slow action transition. A maximum score is preset, and a multi-level deduction rule base and a veto mechanism are established: If the calculated total time for blind wearing exceeds the preset limit, regardless of whether the action is standardized, the assessment will be deemed as failing due to timeout. If a critically vital life-saving action is found to be missing, the assessment will be deemed a major safety hazard and will be deemed unqualified. If the veto item is not triggered, points will be deducted based on the rule base matched with the identified violation type: a preset action missing score will be deducted for each missing non-core action step. Each time the order of actions is reversed, a pre-set score for sequence error will be deducted; Each time a transition between actions is delayed and timed out, a preset delay score will be deducted. Subtract the sum of the points deducted for each violation from the full score to obtain the final practical score; If the final practical score reaches or exceeds the preset passing score and does not trigger a veto, the assessment is considered passed; otherwise, the assessment is considered failed.
[0024] Example 2: like Figure 2 As shown, the present invention also provides an intelligent assessment system for blind wearing actions of self-rescue devices based on edge AI, including an assessment standard pre-storage unit, an image extraction unit, an image comparison unit, an action assessment unit, and an assessment evaluation unit, wherein: The assessment standard pre-storage unit is used to convert the N standard action steps of the blind wearing assessment of coal mine self-rescue device into corresponding Chinese behavioral description text, and generate standard text feature vectors that correspond one-to-one with the N standard action steps through the text encoder. The generated standard text feature vectors are then stored in the terminal database of the mobile device. The image extraction unit is used to capture a video of the blind wearing process of the self-rescue device by the camera of the mobile device, and to extract a continuous image frame sequence by uniformly extracting the process video at equal intervals on the mobile device. The image comparison unit is used to map each frame of the image frame sequence to a unified semantic space in real time using the image encoder of the multimodal model deployed on the mobile device, generate the corresponding image feature vector, calculate the cosine similarity between the image feature vector and each standard text feature vector in the terminal database, and use the standard action step corresponding to the highest cosine similarity as the predicted action label of the frame image. The action assessment unit is used to establish a time-series sliding window, perform anti-shake filtering and continuous action segment clustering on frame sequences with predicted action labels, extract the execution order list of the actual action steps, and the start and end timestamps of each action step. The assessment unit compares the extracted action step execution sequence list with the preset standard action step sequence for sequential consistency. It also calculates the total blind wearing time based on the timestamp of each action step, comprehensively determines whether the candidate passes the assessment, and outputs fine-grained violation deduction items.
[0025] The size of the interval and threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by those skilled in the art for each set of sample data; as long as it does not affect the ratio between the parameter and the quantized value.
[0026] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. In the two embodiments provided in this application, it should be understood that the disclosed apparatus and system can be implemented in other ways; for example, the apparatus embodiments described above are merely illustrative, and the division of modules is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed; another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the apparatus or module can be electrical, mechanical or other forms. The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for intelligent assessment of blind wearing actions of self-rescue devices based on edge AI, characterized in that, Includes the following steps: S1. Convert the N standard action steps of the blind wearing test of coal mine self-rescue device into corresponding Chinese behavioral description text, and generate standard text feature vectors that correspond one-to-one with the N standard action steps through a text encoder. Store the generated standard text feature vectors in the terminal database of the mobile device. S2. Use the camera of the mobile device to capture a video of the blind wearing process of the self-rescue device by the person to be assessed, and extract a continuous image frame sequence by uniformly extracting the process video at equal intervals on the mobile device. S3. Using the image encoder of the multimodal model deployed on the mobile device, each frame of the image frame sequence is mapped to a unified semantic space in real time to generate a corresponding image feature vector. The cosine similarity between the image feature vector and each standard text feature vector in the terminal database is calculated, and the standard action step corresponding to the highest cosine similarity is used as the predicted action label of the frame image. S4. Establish a time-series sliding window, perform anti-shake filtering and continuous action segment clustering on the frame sequence with predicted action labels, extract the execution order list of the actual action steps, and the start and end timestamps of each action step. S5. Compare the extracted action step execution sequence list with the preset standard action step sequence for sequential consistency. At the same time, calculate the total blind wearing time based on the timestamp of each action step, comprehensively determine whether the candidate passes the assessment, and output fine-grained violation deduction items.
2. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 1, characterized in that, The specific process for generating standard text feature vectors is as follows: According to the coal mine safety regulations and the standard operating procedures for self-rescue devices, the complete blind-wearing process is deconstructed into N logically independent standard action steps; For each standard action step n, where n ∈ [1, N], construct a step text set containing the main description text and K synonymous extended description texts: ; A text encoder using a pre-trained multimodal model processes each text in the text set Tn at each step. Feature extraction is performed, where k ∈ [1, K], to generate the corresponding D-dimensional text feature vector. ; Mean pooling is used to fuse all D-dimensional text feature vectors of the nth standard action step to calculate the coarse feature vector of that standard action step. : ； For the fused coarse feature vector L2 norm normalization is performed to obtain the final standard text feature vector. : ； In the formula, Represents coarse feature vectors The L2 norm.
3. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 1, characterized in that, The specific process of uniformly extracting frames at equal intervals from the process video is as follows: At the start of the assessment, the assessment application software on the mobile device calls the underlying camera hardware interface of the system to turn on the rear camera and uses the built-in fixed-frequency sampling timer to perform uniform frame sampling at equal intervals. During sampling, the hardware acquisition callback function extracts a frame from the video stream buffer at fixed time intervals. At the moment of extraction, it reads the system monotonic hardware clock and directly binds the current millisecond-level absolute timestamp as metadata to the image frame, generating a sampling frame with time attributes.
4. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 3, characterized in that, The specific process for obtaining a continuous sequence of image frames is as follows: The sampling frame is quickly converted into an RGB three-channel matrix image in memory using the graphics processing chip of the mobile device. The image is then cropped with the candidate's main body area as the center. The RGB three-channel matrix image is normalized and scaled to the standard input size required by the multimodal model image encoder. Mean elimination and variance scaling are performed on the scaled RGB three-channel matrix image to map the pixel values to the specified standard value range, thus obtaining the preprocessed image frame. The pre-processed image frames are pushed into a pre-allocated circular dynamic memory buffer in the mobile device's memory in chronological order, forming a continuous sequence of image frames strictly arranged according to the timeline.
5. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 1, characterized in that, The specific process for calculating the cosine similarity between the image feature vector and each standard text feature vector in the edge database is as follows: The continuous image frame sequence obtained in S2 is input into an image encoder deployed on the GPU of a mobile device. The image encoder performs forward inference on the m-th frame and outputs the corresponding original image feature vector. ; For the original image feature vector L2 norm normalization is performed to obtain the image feature vector. : ,in, Represents the L2 norm; Read N pre-stored, normalized standard text feature vectors from the client-side database, and convert the image feature vectors... With the text feature vector matrix stored in a fixed location By multiplying, a single computation can yield the set of similarity scores for the m-th frame relative to all N standard action steps in parallel: 。 6. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 5, characterized in that, The specific process for obtaining the predicted action label of an image is as follows: After obtaining the set of similarity scores for all standard action steps for the m-th frame image, search for the maximum similarity score in the set of similarity scores and obtain its corresponding standard action step index. Preset reliability threshold The maximum similarity score Compared with the preset confidence threshold By comparison, the final predicted action label for the m-th frame image is assigned according to the following formula. : ； In the formula, = This indicates that the image frame was successfully identified as the first one. A standard action procedure; = This indicates that the image frame has insufficient action confidence and is classified as a background, transition, or invalid action frame.
7. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 1, characterized in that, The specific process of obtaining frame sequences with predicted action labels and constructing a temporal sliding window for anti-shake filtering is as follows: Set a time-series sliding window that covers a fixed number of frames, move it frame by frame along the time axis, use the mode filtering logic within the time-series sliding window, count the action tags that appear most frequently within the time-series sliding window as high-frequency tags, and replace the original tags of the center frame of the time-series sliding window with high-frequency tags to obtain the filtered frame sequence. Temporally continuous clustering is performed on the filtered frame sequence. The frame sequence is traversed, and the frame sequences that are temporally continuous and have the same predicted action label are merged into independent action event segments. For each action event segment generated by clustering, the absolute timestamp bound to its first frame is extracted and recorded as the start timestamp of the action event segment, and the absolute timestamp bound to its last frame is extracted as the end timestamp of the action event segment. The standard action numbers corresponding to each action event segment are extracted in chronological order and arranged sequentially to form a linked list of the execution order of the action steps actually performed by the examinee.
8. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 1, characterized in that, The specific process for performing sequence consistency comparison is as follows: A linked list of standard blind-wearing action steps is pre-stored in the terminal database. Using the longest common subsequence matching logic and finite state machine verification rules, the execution order linked list of the actual action steps performed by the examinee is compared with the standard linked list. Each necessary action step in the standard order linked list is retrieved one by one. If a certain standard step is not found in the actual action step linked list, the violation of missing key action step is triggered and recorded. Based on the unidirectional irreversible transition rule of the finite state machine, the order of each action step in the linked list is determined. If a subsequent action step is detected to occur before a previous action step, the action order reversal violation is triggered and recorded. If the same action step in the actual action chain is repeatedly and discontinuously triggered, then the action duplication violation will be triggered and recorded. Extract the start timestamp of the first valid non-zero action segment that actually occurs as the start time of the assessment, and extract the end timestamp of the last valid standard action segment as the end time of the assessment. Calculate the time difference between the end time and the start time, which is the total blind wearing time of the person to be assessed. Calculate the interval between two adjacent action segments. If the transition interval between two adjacent actions exceeds the preset pause threshold, it is determined that the examinee hesitated during the action transition, triggering and recording the violation of excessively slow action transition.
9. The intelligent assessment method for blind wearing actions of self-rescue devices based on edge AI according to claim 1, characterized in that, The specific process for determining whether a candidate has passed the assessment is as follows: A maximum score is preset, and a multi-level deduction rule base and a veto mechanism are established: If the calculated total time for blind application exceeds the preset limit, regardless of whether the action is standardized, the assessment will be deemed as failing due to timeout. If a critically vital life-saving action is found to be missing, the assessment will be deemed a major safety hazard and will be deemed unqualified. If the veto item is not triggered, points will be deducted based on the rule base matched with the identified violation type: a preset action missing score will be deducted for each missing non-core action step. Each time the order of actions is reversed, a pre-set score for sequence error will be deducted; Each time a transition between actions is delayed and timed out, a preset delay score will be deducted. Subtract the sum of the points deducted for each violation from the full score to obtain the final practical score; If the final practical score reaches or exceeds the preset passing score and does not trigger a veto, the assessment is considered passed; otherwise, the assessment is considered failed.
10. A self-rescue device blind-wearing action intelligent assessment system based on edge AI, characterized in that, The self-rescue device blind-wearing action intelligent assessment method based on edge AI, as described in claims 1-9, includes an assessment standard pre-storage unit, an image extraction unit, an image comparison unit, an action assessment unit, and an assessment evaluation unit, wherein: The assessment standard pre-storage unit is used to convert the N standard action steps of the blind wearing assessment of coal mine self-rescue device into corresponding Chinese behavioral description text, and generate standard text feature vectors that correspond one-to-one with the N standard action steps through the text encoder. The generated standard text feature vectors are then stored in the terminal database of the mobile device. The image extraction unit is used to capture a video of the blind wearing process of the self-rescue device by the camera of the mobile device, and to extract a continuous image frame sequence by uniformly extracting the process video at equal intervals on the mobile device. The image comparison unit is used to map each frame of the image frame sequence to a unified semantic space in real time using the image encoder of the multimodal model deployed on the mobile device, generate the corresponding image feature vector, calculate the cosine similarity between the image feature vector and each standard text feature vector in the terminal database, and use the standard action step corresponding to the highest cosine similarity as the predicted action label of the frame image. The action assessment unit is used to establish a time-series sliding window, perform anti-shake filtering and continuous action segment clustering on frame sequences with predicted action labels, extract the execution order list of the actual action steps, and the start and end timestamps of each action step. The assessment unit compares the extracted action step execution sequence list with the preset standard action step sequence for sequential consistency. It also calculates the total blind wearing time based on the timestamp of each action step, comprehensively determines whether the candidate passes the assessment, and outputs fine-grained violation deduction items.