A method, system, and medium for guiding the donning of protective clothing based on a dual-branch frame.

Abstract

This invention discloses a method, system, and medium for guiding the donning of protective clothing based on a dual-branch frame. The method includes the following steps: constructing a dual-branch frame; connecting a camera to the dual-branch frame to capture real-time video of workers' protective clothing donning actions; the dual-branch frame obtaining the visual sampling interval based on the camera's frame rate and sampling video frame segments; simultaneously inputting the video frame segments into coarse-grained branch one and fine-grained branch two for judgment; if the judgment result of fine-grained branch two is correct, it is transmitted together with the action category output from coarse-grained branch one to the topK refinement module for refined judgment; if the judgment result of fine-grained branch two is incorrect, it is transmitted together with the action category output from coarse-grained branch one to the process guidance module for prompting guidance. This invention provides workers with comprehensive guidance and prompts for donning protective clothing, thereby reducing their risk of being harmed by infectious sources and ensuring their safety and health, and has broad application prospects.

Description

Technical Field

[0001] This invention belongs to the technical field of deep learning and protective clothing donning guidance, specifically relating to a protective clothing donning guidance method, system and medium based on a dual-branch framework. Background Technology

[0002] Wearing protective clothing is necessary in many scenarios, such as when working in environments with infectious disease risks, such as customs, hospitals, and laboratories. In these scenarios, protective clothing is an important protective tool that can effectively reduce the risk of physical contact with hazardous substances and protect the health and safety of workers. Wearing protective clothing can effectively isolate medical personnel from contact with the virus, reducing the risk of infection. The process of wearing protective clothing typically includes protective equipment such as a hat, mask, goggles, gloves, and a protective suit, which can cover most areas of the body and effectively avoid transmission routes such as droplets, blood, and bodily fluids. However, the process of wearing protective clothing is quite complex, mainly in the following aspects:

[0003] 1) Complex donning procedure: Donning protective clothing requires following a specific procedure, including hand hygiene, donning a cap, mask, face shield, and gloves. Each step must be strictly performed according to standard operating procedures, otherwise the protective effect will be affected.

[0004] 2) High requirements for wearing: Wearing protective clothing requires professional training and guidance, and places high demands on the wearer, including physical coordination, operational skills, and psychological qualities.

[0005] 3) The process of wearing protective clothing is susceptible to contamination: the process of wearing protective clothing is susceptible to contamination from the surrounding environment, such as dust and microorganisms. Strict environmental hygiene control is required to avoid contaminating the protective clothing.

[0006] Existing protective clothing guidance systems can guide the wearing of protective clothing through models, but they can only determine whether the wearing action is correct. When the model malfunctions, it is difficult to correct such malfunctions, and it is difficult to provide further guidance in complex environments, which reduces the usability of the model. Summary of the Invention

[0007] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method, system and medium for guiding the wearing of protective clothing based on a dual-branch frame, which provides comprehensive operational guidance and prompts for workers wearing protective clothing, thereby reducing the risk of being harmed by sources of infection and ensuring their safety and health.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] On the one hand, a method for guiding the wearing of protective clothing based on a dual-branch frame is provided, the method comprising the following steps:

[0010] A dual-branch framework is constructed; the dual-branch framework includes a coarse-grained branch one, a fine-grained branch two, a topK refinement module, and a process guidance module;

[0011] The camera is connected to the dual-branch frame to collect real-time video of the staff's actions of putting on protective clothing;

[0012] The dual-branch framework obtains the visual sampling interval based on the camera frame rate and samples video frame segments;

[0013] The video frame segments are simultaneously input into both coarse-grained branch one and fine-grained branch two for judgment. Specifically, coarse-grained branch one judges the type of protective clothing donning action in the video frame segments; fine-grained branch two judges the correctness of the protective clothing donning action in the video frame segments based on the preset actions to be performed.

[0014] If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is considered correct; if it does not exist, the result of the fine-grained branch two is considered incorrect, and an error message is fed back; the fine-grained branch two uses the output of the coarse-grained branch one to perform self-correction based on the error message;

[0015] If the result of the fine-grained branch two is incorrect, it is transmitted together with the action category output by the coarse-grained branch one to the process guidance module to obtain the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and to provide prompts and guidance to the staff.

[0016] As a preferred technical solution, in the dual-branch framework, coarse-grained branch one is connected to the topK refinement module and the process guidance module respectively; fine-grained branch two is connected to the topK refinement module and the process guidance module respectively.

[0017] The coarse-grained branch one and fine-grained branch two are constructed based on a two-dimensional residual neural network with time-series modeling.

[0018] As a preferred technical solution, the coarse-grained branch determines the protective clothing donning action category for a pair of video frame segments, specifically as follows:

[0019] Multiple video frame segments are input into coarse-grained branch one, and residual blocks are used to extract features and output feature maps through convolution, residual connection and nonlinear transformation operations.

[0020] Temporal offset and channel shuffling are performed on the output feature map of each residual block to model the temporal information of multiple video frame segments during feature extraction and extract deep features.

[0021] The extracted deep features are processed through pooling and fully connected layers to output a class code. The protective clothing wearing action corresponding to the largest class code value is taken as the current protective clothing wearing action class.

[0022] The dimension of the category encoding is the same as the number of all actions involved in the protective clothing donning process.

[0023] As a preferred technical solution, the fine-grained branch two determines the correctness of the protective clothing donning action in the video frame segment based on the preset actions to be performed, specifically as follows:

[0024] Retrieve the preset protective clothing donning action from the predefined list of protective clothing donning actions, and set its category code value to dimension two;

[0025] Multiple video frame segments are input into the fine-grained branch 2, and residual blocks are used to extract features and output feature maps through convolution, residual connection and nonlinear transformation operations.

[0026] Temporal offset and channel shuffling are performed on the output feature map of each residual block to model the temporal information of multiple video frame segments during feature extraction and extract deep features.

[0027] The extracted deep features are processed through pooling and fully connected layers to output a maximum class encoding value.

[0028] If the maximum category code value is dimension one, then the currently correctly executed protective clothing donning action is determined to be incorrect;

[0029] If the maximum category code value is dimension two, then the currently performed action of putting on protective clothing is considered correct.

[0030] As a preferred technical solution, the process by which the topK refinement module uses the output of a coarse-grained branch pair to determine the fine-grained branch pair is as follows:

[0031] If the fine-grained branch 2 determines that the currently performed protective clothing donning action is correct, the topK refinement module takes the first K categories of the category codes output by the coarse-grained branch 1 as the correct category range;

[0032] Determine whether the protective clothing donning action currently being performed in fine-grained branch two falls within the correct category range;

[0033] If it exists, the judgment result of fine-grained branch two is confirmed to be correct; if it does not exist, the judgment result of fine-grained branch two is confirmed to be incorrect, and the error information is fed back to fine-grained branch two.

[0034] After receiving the error message, the fine-grained branch two retrains using the output of the coarse-grained branch one to perform self-correction.

[0035] As a preferred technical solution, the process guidance module specifically comprises:

[0036] If the fine-grained branch 2 determines that the currently executed protective clothing donning action is incorrect, the process guidance module extracts the action category with the highest confidence output from the coarse-grained branch 1 and uses this action category as the currently executed incorrect protective clothing donning action.

[0037] Based on a predefined list of protective clothing donning actions, the order of the incorrect protective clothing donning actions and the order of the correct protective clothing donning actions to be performed are determined to obtain the specific protective clothing donning action to be performed.

[0038] The system uses voice prompts and / or video demonstrations to alert staff to the incorrect protective clothing donning actions they are currently performing and the correct protective clothing donning actions they should be taking.

[0039] On the other hand, a protective clothing donning guidance system based on a dual-branch framework is provided, which is applied to the protective clothing donning guidance method based on a dual-branch framework, including a model building module, a video acquisition module, a segment sampling module and an action judgment module.

[0040] The model building module is used to build a dual-branch framework; the dual-branch framework includes a coarse-grained branch one, a fine-grained branch two, a topK refinement module, and a process guidance module;

[0041] The video acquisition module is used to connect the camera to the dual-branch frame to capture real-time video of the staff's actions of putting on protective clothing.

[0042] The segment sampling module is used by the dual-branch frame to obtain the visual sampling interval based on the camera frame rate and to sample video frame segments;

[0043] The action determination module is used to simultaneously input video frame segments into coarse-grained branch one and fine-grained branch two for determination, wherein:

[0044] The coarse-grained branch determines the type of protective clothing donning action for a pair of video frame segments;

[0045] Fine-grained branch two determines the correctness of the protective clothing donning action in the video frame segment based on the preset actions to be performed;

[0046] If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is considered correct; if it does not exist, the result of the fine-grained branch two is considered incorrect, and an error message is fed back; the fine-grained branch two uses the output of the coarse-grained branch one to perform self-correction based on the error message;

[0047] If the result of the fine-grained branch two is incorrect, it is transmitted together with the action category output by the coarse-grained branch one to the process guidance module to obtain the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and to provide prompts and guidance to the staff.

[0048] In another aspect, an electronic device is provided, the electronic device comprising:

[0049] At least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0050] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the described method for guiding the wearing of protective clothing based on a dual-branch frame.

[0051] On another front, a computer-readable storage medium is provided, storing a program that, when executed by a processor, implements the aforementioned method for guiding the wearing of protective clothing based on a dual-branch framework.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] 1. This invention provides comprehensive operational guidance and prompts for workers wearing protective clothing by real-time monitoring and analysis of motion information, combined with a dual-branch framework. This helps them better master the correct way to put on and take off protective clothing, thereby reducing the risk of exposure to infectious agents and other hazards. Simultaneously, through self-correction and process guidance, correct action instructions can be quickly conveyed to workers, reducing their exposure time due to incorrect actions and ensuring their safety and health. In summary, this invention has advantages such as high real-time performance, comprehensive guidance, and accurate operation, effectively improving the safety and efficiency of protective clothing donning and doffing operations, and has broad application prospects.

[0054] 2. The dual-branch framework proposed in this invention significantly improves accuracy compared to existing models. In protective clothing donning guidance, false positives are a more serious problem than false negatives. A false negative means the person donning the protective clothing is performing the correct action, but the model / network failed to capture it correctly; conversely, a false positive means the person is performing the wrong action, but the model incorrectly identifies it as correct. Therefore, false positives are often accompanied by donning errors and inadequate protection, leading to serious consequences. The dual-branch framework proposed in this invention significantly improves upon the false positive phenomenon of previous models, greatly enhancing the usability of the protective clothing donning guidance system.

[0055] 3. The dual-branch framework proposed in this invention has more detailed and efficient action guidance capabilities than existing models. Previous models / networks can only determine whether the action that should be performed is correct, but do not know the specific category of the erroneous action. In contrast, the dual-branch framework of this invention can provide corrective guidance information to personnel in protective clothing when they make mistakes, making the judgment process for protective clothing more efficient.

[0056] 4. In the dual-branch framework proposed in this invention, both the coarse-grained branch one and the fine-grained branch two use lightweight two-dimensional neural networks for video feature extraction and action judgment. Although the computational load is slightly increased compared to existing models, the accuracy and performance are greatly improved, and it has strong usability. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a protective clothing wearing guidance method based on a dual-branch frame, as described in an embodiment of the present invention.

[0059] Figure 2 This is a structural diagram of a protective clothing donning guidance system based on a dual-branch frame, according to an embodiment of the present invention.

[0060] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0062] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0063] like Figure 1 As shown in the figure, this embodiment of a protective clothing donning guidance method based on a dual-branch frame includes the following steps:

[0064] S1. Construct a dual-branch framework, including coarse-grained branch one, fine-grained branch two, topK refinement module, and process guidance module;

[0065] S2. Connect the camera to the dual-branch frame to collect real-time video of the staff's actions of putting on protective clothing;

[0066] S3, the dual-branch frame obtains the visual sampling interval based on the camera frame rate and samples video frame segments;

[0067] S4. Input the video frame segment into both coarse-grained branch one and fine-grained branch two for judgment. Coarse-grained branch one judges the protective clothing wearing action category of the video frame segment; fine-grained branch two judges the correctness of the protective clothing wearing action in the video frame segment according to the preset action to be performed.

[0068] S5. If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is determined to be correct; if it does not exist, the result of the fine-grained branch two is determined to be incorrect, and an error message is fed back; the fine-grained branch two performs self-correction based on the error message using the output of the coarse-grained branch one.

[0069] S6. If the result of the fine-grained branch two is incorrect, it is transmitted to the process guidance module together with the action category output by the coarse-grained branch one. The module obtains the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and provides prompts and guidance to the staff.

[0070] Specifically, the dual-branch framework in this embodiment of the invention includes a coarse-grained branch one, a fine-grained branch two, a topK refinement module, and a process guidance module. The coarse-grained branch one is connected to both the topK refinement module and the process guidance module; the fine-grained branch two is also connected to both the topK refinement module and the process guidance module. Both the coarse-grained branch one and the fine-grained branch two are constructed based on a two-dimensional residual neural network using time-series modeling.

[0071] Coarse-grained branch one performs coarse-grained motion judgment on the input video frame segments; fine-grained branch two performs fine-grained motion judgment on the input video frame segments. The output of fine-grained branch two, along with the output of coarse-grained branch one, is sent to the topK refinement module. The purpose of this module is to use the output of coarse-grained branch one to refine the output of fine-grained branch two, further improving the accuracy of fine-grained branch two. Simultaneously, the outputs of both branches are also input into the process guidance module to achieve more accurate guidance on wearing protective clothing.

[0072] Specifically, the dual-branch framework in this embodiment of the invention is used to determine whether the worker's wearing action is accurate in complex scenarios, and to make detailed judgments on the worker's incorrect actions; through the rapid judgment capability of the coarse-grained branch one and the high accuracy of the fine-grained branch two, the efficiency of protective clothing wearing guidance is further improved.

[0073] Furthermore, the coarse-grained branch determines the type of protective clothing donning action for a pair of video frame segments, specifically as follows:

[0074] Multiple video frame segments are input into coarse-grained branch one, and residual blocks are used to extract features and output feature maps through convolution, residual connection and nonlinear transformation operations.

[0075] Then, temporal offset and channel shuffling are performed on the output feature map of each residual block to model the temporal information of multiple video frame segments during feature extraction and extract deep features.

[0076] The extracted deep features are processed through pooling and fully connected layers to output a class code. The protective clothing wearing action corresponding to the largest class code value is taken as the current protective clothing wearing action class.

[0077] For coarse-grained branch one, the dimension of the category encoding is the same as the number of all actions in the protective clothing donning process.

[0078] The primary purpose of coarse-grained branch one is to identify the current actions of workers in complex scenarios. It utilizes a time-aware neural network to learn temporal change patterns in video sequences, enabling efficient classification of different actions. Its efficient computational method allows coarse-grained branch one to quickly determine the actions. This rapid judgment capability allows for a quick initial assessment of the donning method. The results of this rapid assessment can then be used for error correction in fine-grained branch two. Simultaneously, it can determine the specific action category currently being performed by the worker. If the action is incorrect, more precise guidance can be provided, thereby improving the efficiency of protective clothing donning instructions.

[0079] Furthermore, the fine-grained branch two determines the correctness of the protective clothing donning action in the video frame segment based on the preset actions to be performed, specifically as follows:

[0080] Retrieve the preset protective clothing donning action from the predefined list of protective clothing donning actions, and set its category code value to dimension two;

[0081] Multiple video frame segments are input into the fine-grained branch 2, and residual blocks are used to extract features and output feature maps through convolution, residual connection and nonlinear transformation operations.

[0082] Then, temporal offset and channel shuffling are performed on the output feature map of each residual block to model the temporal information of multiple video frame segments during feature extraction and extract deep features.

[0083] The extracted deep features are processed through pooling and fully connected layers to output a maximum class encoding value.

[0084] If the maximum category code value is dimension one, and it is different from the preset category code value of the protective clothing donning action to be performed at present, then the protective clothing donning action to be performed at present is judged to be incorrect.

[0085] If the maximum category code value is dimension two, and it is the same as the preset category code value of the protective clothing donning action to be performed, then the protective clothing donning action being performed is determined to be correct.

[0086] Unlike coarse-grained branch one, fine-grained branch two knows the specific action category that the worker should perform. Its main task is to perform binary classification judgment on the currently performed action to determine whether the current action is correct. Fine-grained branch two also uses a neural network with time-series awareness to learn the temporal change patterns in the video sequence and can accurately judge the current action. Furthermore, the rapid judgment results of coarse-grained branch one can be used to verify the accurate judgment results of fine-grained branch two, further improving the accuracy of protective clothing donning action judgment. Once fine-grained branch two determines that the currently performed action is incorrect, it can provide more precise guidance to the person wearing the protective clothing based on the precise description of the current action output by coarse-grained branch one, helping them to perform the correct operation.

[0087] The dual-branch framework combines the coarse-grained branch one and fine-grained branch two mentioned above, using the result of coarse-grained branch one to adjust the judgment of fine-grained branch two. First, if fine-grained branch two predicts that the current worker's protective clothing donning action is correct, it will use the action category judgment from coarse-grained branch one and use the top-K refinement module to determine whether the action is truly correct. If the action is incorrect, it will use the action category judgment from coarse-grained branch one and obtain refined guidance information through the process guidance module.

[0088] Specifically, the process by which the topK refinement module uses the output of a coarse-grained branch pair to determine the result of a fine-grained branch pair is as follows:

[0089] If the fine-grained branch 2 determines that the currently performed protective clothing donning action is correct, the topK refinement module takes the first K categories of the category codes output by the coarse-grained branch 1 as the correct category range;

[0090] Determine whether the protective clothing donning action currently being performed in fine-grained branch two falls within the correct category range;

[0091] If it exists, the judgment result of fine-grained branch two is confirmed to be correct; if it does not exist, the judgment result of fine-grained branch two is confirmed to be incorrect, and the error information is fed back to fine-grained branch two.

[0092] After receiving error information, fine-grained branch two retrains itself using the output of coarse-grained branch one to mitigate the false positive problem and perform self-correction. Through the refinement process of the Top-K refinement module, the false positive problem in protective clothing judgment is effectively mitigated. When a false positive error occurs in the judgment of fine-grained branch two, it can be corrected based on the output of coarse-grained branch one, thereby making the model's predictions more accurate.

[0093] Specifically, if the currently performed action is incorrect, fine-grained branch two lacks the ability to predict the specific error type, which could lead to confusion and misunderstanding among personnel wearing protective clothing. Therefore, the process guidance module needs to provide detailed guidance information, specifically:

[0094] If the fine-grained branch 2 determines that the currently executed protective clothing donning action is incorrect, the process guidance module extracts the action category with the highest confidence output from the coarse-grained branch 1 and uses this action category as the currently executed incorrect protective clothing donning action.

[0095] Based on a predefined list of protective clothing donning actions, the order of the incorrect protective clothing donning actions and the order of the correct protective clothing donning actions to be performed are determined to obtain the specific protective clothing donning action to be performed.

[0096] The system uses voice prompts and / or video demonstrations to alert staff to the incorrect protective clothing donning actions they are currently performing and the correct protective clothing donning actions they should be taking.

[0097] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0098] Based on the same idea as the protective clothing donning guidance method based on a dual-branch frame in the above embodiments, the present invention also provides a protective clothing donning guidance system based on a dual-branch frame, which can be used to execute the aforementioned protective clothing donning guidance method based on a dual-branch frame. For ease of explanation, the structural schematic diagram of an embodiment of a protective clothing donning guidance system based on a dual-branch frame only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0099] like Figure 3 As shown, another embodiment of the present invention provides a protective clothing wearing guidance system based on a dual-branch framework, including a model building module, a video acquisition module, a segment sampling module, and an action judgment module;

[0100] The model building module is used to build a dual-branch framework, including coarse-grained branch one, fine-grained branch two, topK refinement module, and process guidance module.

[0101] The video capture module is used to connect the camera to the dual-branch frame to capture real-time video of the staff's actions of putting on protective clothing;

[0102] The segment sampling module is used by the dual-branch frame to obtain the visual sampling interval based on the camera frame rate and to sample video frame segments;

[0103] The action judgment module is used to input video frame segments into both coarse-grained branch one and fine-grained branch two for judgment. Specifically, coarse-grained branch one judges the type of protective clothing wearing action in the video frame segment; fine-grained branch two judges the correctness of the protective clothing wearing action in the video frame segment based on the preset actions to be performed.

[0104] If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is considered correct; if it does not exist, the result of the fine-grained branch two is considered incorrect, and an error message is fed back; the fine-grained branch two uses the output of the coarse-grained branch one to perform self-correction based on the error message;

[0105] If the result of the fine-grained branch two is incorrect, it is transmitted together with the action category output by the coarse-grained branch one to the process guidance module to obtain the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and to provide prompts and guidance to the staff.

[0106] It should be noted that the protective clothing wearing guidance system based on a dual-branch frame of the present invention corresponds one-to-one with the protective clothing wearing guidance method based on a dual-branch frame of the present invention. The technical features and beneficial effects described in the above embodiment of the protective clothing wearing guidance method based on a dual-branch frame are applicable to the embodiment of the protective clothing wearing guidance system based on a dual-branch frame. For details, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here.

[0107] Please see Figure 3 In one embodiment, an electronic device is provided for implementing a protective clothing wearing guidance method based on a dual-branch frame. The electronic device may include a first processor, a first memory, and a bus, and may also include a computer program, such as a protective clothing wearing guidance program, stored in the first memory and executable on the first processor.

[0108] The first memory includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the first memory can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the first memory can include both internal and external storage units of the electronic device. The first memory can be used not only to store application software and various types of data installed on the electronic device, such as code for protective clothing donning instructions, but also to temporarily store data that has been output or will be output.

[0109] In some embodiments, the first processor may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory (e.g., protective clothing donning instructions) and calls data stored in the first memory to perform various functions of the electronic device and process data.

[0110] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0111] The protective clothing donning instructions stored in the first memory of the electronic device are a combination of multiple instructions, which, when run in the first processor, can achieve the following:

[0112] Construct a dual-branch framework, including coarse-grained branch one, fine-grained branch two, a topK refinement module, and a process guidance module;

[0113] The camera is connected to the dual-branch frame to collect real-time video of the staff's actions of putting on protective clothing;

[0114] The dual-branch framework obtains the visual sampling interval based on the camera frame rate and samples video frame segments;

[0115] The video frame segment is simultaneously input into both the coarse-grained branch one and the fine-grained branch two for judgment, where:

[0116] The coarse-grained branch determines the type of protective clothing donning action for a pair of video frame segments;

[0117] Fine-grained branch two determines the correctness of the protective clothing donning action in the video frame segment based on the preset actions to be performed;

[0118] If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is considered correct; if it does not exist, the result of the fine-grained branch two is considered incorrect, and an error message is fed back; the fine-grained branch two uses the output of the coarse-grained branch one to perform self-correction based on the error message;

[0119] If the result of the fine-grained branch two is incorrect, it is transmitted together with the action category output by the coarse-grained branch one to the process guidance module to obtain the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and to provide prompts and guidance to the staff.

[0120] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for guiding the donning of protective clothing based on a dual-branch frame, characterized in that, The method includes the following steps: A dual-branch framework is constructed; the dual-branch framework includes a coarse-grained branch one, a fine-grained branch two, a topK refinement module, and a process guidance module; The camera is connected to the dual-branch frame to collect real-time video of the staff's actions of putting on protective clothing; The dual-branch framework obtains the visual sampling interval based on the camera frame rate and samples video frame segments; The video frame segments are simultaneously input into both coarse-grained branch one and fine-grained branch two for judgment. Specifically, coarse-grained branch one judges the type of protective clothing donning action in the video frame segments; fine-grained branch two judges the correctness of the protective clothing donning action in the video frame segments based on the preset actions to be performed. If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is considered correct; if it does not exist, the result of the fine-grained branch two is considered incorrect, and an error message is fed back; the fine-grained branch two uses the output of the coarse-grained branch one to perform self-correction based on the error message; If the result of the fine-grained branch two is incorrect, it is transmitted together with the action category output by the coarse-grained branch one to the process guidance module to obtain the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and to provide prompts and guidance to the staff.

2. The protective clothing donning guidance method based on a dual-branch frame according to claim 1, characterized in that, In the dual-branch framework, coarse-grained branch one is connected to the topK refinement module and the process guidance module respectively; fine-grained branch two is connected to the topK refinement module and the process guidance module respectively. The coarse-grained branch one and fine-grained branch two are constructed based on a two-dimensional residual neural network with time-series modeling.

3. The protective clothing donning guidance method based on a dual-branch frame according to claim 2, characterized in that, The coarse-grained branch determines the type of protective clothing donning action for a pair of video frame segments, specifically as follows: Multiple video frame segments are input into coarse-grained branch one, and residual blocks are used to extract features and output feature maps through convolution, residual connection and nonlinear transformation operations. Temporal offset and channel shuffling are performed on the output feature map of each residual block to model the temporal information of multiple video frame segments during feature extraction and extract deep features. The extracted deep features are processed through pooling and fully connected layers to output a class code. The protective clothing wearing action corresponding to the largest class code value is taken as the current protective clothing wearing action class. The dimension of the category encoding is the same as the number of all actions involved in the protective clothing donning process.

4. The protective clothing donning guidance method based on a dual-branch frame according to claim 3, characterized in that, The second fine-grained branch determines the correctness of the protective clothing donning action in the video frame segment based on the preset actions to be performed, specifically: Retrieve the preset protective clothing donning action from the predefined list of protective clothing donning actions, and set its category code value to dimension two; Multiple video frame segments are input into the fine-grained branch 2, and residual blocks are used to extract features and output feature maps through convolution, residual connection and nonlinear transformation operations. Temporal offset and channel shuffling are performed on the output feature map of each residual block to model the temporal information of multiple input video frame segments during feature extraction and extract deep features. The extracted deep features are processed through pooling and fully connected layers to output a maximum class encoding value. If the maximum category code value is dimension one, then the currently correctly executed protective clothing donning action is determined to be incorrect; If the maximum category code value is dimension two, then the currently performed action of putting on protective clothing is considered correct.

5. The protective clothing donning guidance method based on a dual-branch frame according to claim 4, characterized in that, The process by which the topK refinement module uses the output of a coarse-grained branch pair to determine the fine-grained branch pair is as follows: If the fine-grained branch 2 determines that the currently performed protective clothing donning action is correct, the topK refinement module takes the first K categories of the category codes output by the coarse-grained branch 1 as the correct category range; Determine whether the protective clothing donning action currently being performed in fine-grained branch two falls within the correct category range; If it exists, the judgment result of fine-grained branch two is confirmed to be correct; if it does not exist, the judgment result of fine-grained branch two is confirmed to be incorrect, and the error information is fed back to fine-grained branch two. After receiving the error message, the fine-grained branch two retrains using the output of the coarse-grained branch one to perform self-correction.

6. The protective clothing donning guidance method based on a dual-branch frame according to claim 4, characterized in that, The process guidance module specifically includes: If the fine-grained branch 2 determines that the currently executed protective clothing donning action is incorrect, the process guidance module extracts the action category with the highest confidence output from the coarse-grained branch 1 and uses this action category as the currently executed incorrect protective clothing donning action. Based on a predefined list of protective clothing donning actions, the order of the incorrect protective clothing donning actions and the order of the correct protective clothing donning actions to be performed are determined to obtain the specific protective clothing donning action to be performed. The system uses voice prompts and / or video demonstrations to alert staff to the incorrect protective clothing donning actions they are currently performing and the specific protective clothing donning actions they should be taking.

7. A protective clothing donning guidance system based on a dual-branch frame, characterized in that, The protective clothing wearing guidance method based on a dual-branch frame, applicable to any one of claims 1-6, includes a model building module, a video acquisition module, a segment sampling module, and an action judgment module; The model building module is used to build a dual-branch framework; the dual-branch framework includes a coarse-grained branch one, a fine-grained branch two, a topK refinement module, and a process guidance module; The video acquisition module is used to connect the camera to the dual-branch frame to capture real-time video of the staff's actions of putting on protective clothing. The segment sampling module is used by the dual-branch frame to obtain the visual sampling interval based on the camera frame rate and to sample video frame segments; The action determination module is used to simultaneously input video frame segments into coarse-grained branch one and fine-grained branch two for determination, wherein: The coarse-grained branch determines the type of protective clothing donning action for a pair of video frame segments; Fine-grained branch two determines the correctness of the protective clothing donning action in the video frame segment based on the preset actions to be performed; If the result of the fine-grained branch two is correct, it is transmitted together with the action category output by the coarse-grained branch one to the topK refinement module to determine whether the protective clothing donning action being performed in the current video frame segment exists in the action category output by the coarse-grained branch one; if it exists, the result of the fine-grained branch two is considered correct; if it does not exist, the result of the fine-grained branch two is considered incorrect, and an error message is fed back; the fine-grained branch two uses the output of the coarse-grained branch one to perform self-correction based on the error message; If the result of the fine-grained branch two is incorrect, it is transmitted together with the action category output by the coarse-grained branch one to the process guidance module to obtain the sequence of incorrect protective clothing wearing actions and the sequence of correct protective clothing wearing actions that should be performed, and to provide prompts and guidance to the staff.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform a protective clothing donning guidance method based on a dual-branch frame as described in any one of claims 1-6.

9. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the protective clothing wearing guidance method based on a dual-branch frame as described in any one of claims 1-6.

Images