A fatigue self-adaptive control method and device based on human posture recognition

Abstract

The application discloses a fatigue self-adaptive control method and device based on human posture recognition, and belongs to the technical field of human-computer interaction and automatic control. The method comprises the following steps: collecting a work posture image and inputting the posture image into a lightweight posture recognition model; the recognition model takes MobileNet V3 large as a backbone network, replaces deep separable convolution in part of the residual block with a hollow convolution with different hollow rates, and connects two levels of multi-scale gate feature enhancement and multi-scale normalization fusion modules and a dynamic sampling and shallow-middle layer feature fusion module; joint point coordinates are output, RULA fatigue scores are calculated frame by frame, and fatigue cumulative original values in a task cycle are calculated by using time weighted average. After normalization, the value is converted into an expected assist torque by an exponential nonlinear mapping function, and adaptive assist is output by driving an exoskeleton device based on fuzzy PID closed-loop control. The application realizes the cooperative control of high-precision lightweight posture recognition, fine time cumulative fatigue evaluation and adaptive exoskeleton assist.

Description

Technical Field

[0001] This application relates to the fields of automatic control, human-computer interaction and fatigue assessment, and in particular to a fatigue adaptive control method and device based on human posture recognition. Background Technology

[0002] For complex system operations, operators use external exoskeleton assistive devices. Wearable exoskeletons, in particular, can improve operator efficiency and reduce fatigue risks. However, currently, the activation and deactivation of these devices, as well as the level of assistance (adjustment), require manual adjustment. This not only interferes with the task but also often results in delayed or forgotten adjustments, further hindering the original task. Furthermore, fixed assistance levels do not meet human comfort requirements and increase the power consumption of the exoskeleton. Therefore, automatic adjustment based on fatigue monitoring, fatigue perception, and fatigue feedback is crucial for the true integration of assistive devices with operators. Some studies use muscle fatigue monitoring (primarily sEMG), heart rate monitoring, and other methods to interactively control assistive devices, estimating fatigue and adjusting assistance through various regression models. However, this adjustment requires attaching electrodes or wearing wristbands, which can interfere with operation and are difficult to apply to various real-world task environments. Relying solely on heart rate monitoring also has low accuracy.

[0003] Common assembly and maintenance operations are highly susceptible to fatigue, leading to increased safety and health risks and decreased operational efficiency. Exoskeleton assistive devices have become an important tool for mitigating fatigue risks. However, the level of assistance provided is not rapidly and in real-time linked to human fatigue levels, and there is a lack of non-contact, real-time, and reliable assessment methods suitable for engineering applications. Therefore, to prevent occupational diseases, protect worker health, and improve work efficiency, it is necessary to conduct rapid, real-time, and accurate ergonomic fatigue assessments. This assessment should be used as input to enable adaptive and proactive adjustment of the exoskeleton assistive device, thereby achieving its true effectiveness and practicality.

[0004] 202110600585.0 A Real-Time RULA Evaluation Method in Virtual Reality. Firstly, this invention provides real-time RULA assessment by acquiring motion frames and obtaining limb vectors and principal angles in real time, using these as inputs for RULA scoring. It allows for real-time observation of high-risk postures, but it cannot assess fatigue accumulation, thus offering limited value for risk identification, as short-term high-risk postures can be corrected through timely adjustments. Secondly, this method lacks practicality. While identification and evaluation of a single frame within a virtual environment is feasible in the patent's implementation, it becomes extremely time-consuming and labor-intensive for real-world work tasks involving numerous postures.

[0005] CN104173052A describes a human comfort measurement method based on RULA (Relaxed Postural Analysis). This method incorporates muscle MVC (Muscle Muscle Capacity) and postural duration factors to calculate a body posture comfort level and provide suggestions. While this method considers fatigue factors more comprehensively, it has the following limitations. First, this method is suitable for jobs with minimal postural changes; operators should maintain a relatively fixed posture as much as possible during work to ensure stable and accurate measurements of both RULA and MVC. Second, the scoring method of this method, which calculates scores to classify levels, is relatively coarse. It can be used for risk identification but is not suitable for precise fatigue assessment.

[0006] CN112101802A describes a method, apparatus, electronic device, and storage medium for evaluating posture conformity data. This invention acquires personnel work posture data using a depth sensor, calculates the posture duration using a rapid on-site assessment method, matches it against a pre-set scoring table, and obtains a score using a posture load evaluation formula. This method is designed for jobs requiring sustained posture maintenance, but its scoring rules are highly subjective and lack data validation.

[0007] CN113633281A discloses a method and system for assessing human posture during assembly and maintenance. This invention identifies joint points by acquiring posture depth images, overlays and corrects them with corresponding color images, and uses the RULA method to determine the risk score for each joint. When the score exceeds a set threshold, the posture and joint point position are recorded, and a risk warning is issued. While the correction method for identifying human joint points is simple, it has a large margin of error. Furthermore, this method outputs the current risk level of each joint point for assessment without considering the accumulation of time, making it prone to fatigue false alarms. That is, even if a limb momentarily exhibits a high-risk movement and then immediately recovers without fatigue, the device still alarms. Summary of the Invention

[0008] This application provides a fatigue adaptive control method and device based on human posture recognition, which is used to assess the cumulative fatigue risk level of a certain non-fixed posture operation task (such as complex system assembly and maintenance) over a period of time, and adjust the exoskeleton assistive device accordingly to provide adaptive and smooth assistance to the operator.

[0009] This application provides a fatigue adaptive control method based on human posture recognition, including: Collect images of the workers' postures during operation; The work posture image is input into a lightweight posture recognition model to identify the coordinates of human joints and limb connection information; the lightweight posture recognition model includes: a backbone network, a multi-scale gating encoder and a dynamic sampling fusion module; The backbone network adopts the MobileNet V3 large network architecture, and some of the depthwise separable convolutions of the inverted residual blocks in the backbone network are replaced with dilated convolutions to expand the receptive field and maintain the feature map resolution. The multi-scale gated encoder is connected to the output of the backbone network and includes at least two dilated convolutional branches with different dilation rates arranged in parallel. It is used to extract multi-scale features and perform gated weighted fusion on the deep features output by the backbone network using a multi-scale gated feature enhancement module, and output an initial feature map. The initial feature map is input into the first stage of OpenPose. The initial feature map is then gated and weighted fused with the predicted feature map output by the first stage of OpenPose. After weighted fusion, the feature map is refined by the second stage of OpenPose and output as a refined feature map. The dynamic sampling fusion module is used to upsample the refined feature map output by OpenPose and perform gated attention fusion with at least one intermediate layer feature map extracted from the backbone network to output joint heatmap and joint coordinates. Based on the identified joint coordinates, a frame-level fatigue score is calculated, and the raw value of cumulative fatigue within the task cycle is calculated. Based on the original fatigue accumulation value, the desired assist torque is calculated according to a preset exponential nonlinear mapping function, and control commands are generated through a closed-loop control algorithm. The exoskeleton assist device is driven by the control command to output a corresponding assist torque to assist the worker.

[0010] This application provides a fatigue adaptive control device based on human posture recognition, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the fatigue adaptive control method based on human posture recognition as described above.

[0011] The method described in this application can assess the cumulative fatigue risk level of a non-fixed posture operation task (such as the assembly and maintenance of complex systems) over a period of time, and based on this assessment, adjust the exoskeleton assistive device to provide adaptive and smooth assistance to the operator. This application is applicable to a wide range of tasks, especially for tasks with long operation times, complex and precise systems, confined environments, and constantly changing postures. The method of this application shows advantages in these tasks, such as the maintenance of complex systems like aircraft, vehicles, and power systems.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0013] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating the fatigue adaptive control method based on human posture recognition, as described in an embodiment of this application. Figure 2 This is a schematic diagram of the workflow of the multi-scale gating feature enhancement module of the fatigue adaptive control method based on human posture recognition in this application embodiment; Figure 3 This is a schematic diagram of the workflow of the improved lightweight posture recognition model of the fatigue adaptive control method based on human posture recognition, which is an embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0015] This application provides a fatigue adaptive control method based on human posture recognition, such as... Figure 1 As shown, it includes: In step S101, images of the worker's posture are acquired. For example, for a specific task, video of the worker performing the task can be acquired. Specifically, cameras can be set up, ensuring that the viewfinder of each acquisition device can capture all limb parts of the subject, minimizing obstruction of the subject's limbs by factors other than their own body parts. Then, video recording begins when the worker starts the task and continues until the task is completed. Generally, the recording equipment is set to 30 frames per second, with each frame outputting an RGB image.

[0016] In step S102, the work posture image is input into a lightweight posture recognition model to identify the coordinates of human joints and limb connection information. An RGB image is input into the lightweight posture recognition model, which outputs a diagram of joints and limb connections. In a specific embodiment, this application proposes an improved lightweight posture recognition model with a backbone network of MobileNetV3 large. The MobileNet V3 large network, due to the addition of inverted residual blocks and SE modules, as well as the removal of gaps and simultaneous extraction of shallow, mid, and deep network features, can retain more feature information. However, the continuous downsampling across multiple layers of the network reduces resolution, leading to blurred feature details, which significantly impacts joint coordinate point recognition tasks.

[0017] In this embodiment, the lightweight pose recognition model includes: a backbone network, a multi-scale gating encoder, and a dynamic sampling fusion module, wherein, The backbone network adopts the MobileNet V3 large network architecture, and replaces the depthwise separable convolutions of some inverted residual blocks in the backbone network with dilated convolutions to expand the receptive field and maintain the feature map resolution. The main features of this application include three points. This part is the first improvement, which replaces the ordinary depthwise separable convolutions (DW) of some inverted residual blocks in the network with dilated convolutions with different dilation rates (DR) to expand the receptive field (including the last downsampling).

[0018] The multi-scale gated encoder is connected to the output of the backbone network and includes at least two dilated convolutional branches with different dilation rates arranged in parallel. The dilated convolutional branches are used to extract multi-scale features and perform gated weighted fusion on the deep features output by the backbone network using the multi-scale gated feature enhancement module to output an initial feature map.

[0019] The initial feature map is input into the first stage of OpenPose, and the initial feature map is gated and weightedly fused with the predicted feature map output from the first stage of OpenPose. After weighted fusion, the feature map is refined by the second stage of OpenPose and output as a refined feature map. The second improvement of this application is that after the feature map is output from the backbone network, the previously blurred pose and limb heatmap features are refined through two-level multi-scale feature extraction and fusion. The initial enhanced feature map output from the multi-scale feature extraction and fusion stage is refined by the OpenPose refinement stage to output a refined feature map, that is, a refined pose and limb heatmap.

[0020] The dynamic sampling fusion module is used to upsample the refined feature map output by OpenPose and perform gated attention fusion with at least one intermediate layer feature map extracted from the backbone network to output a joint heatmap and joint coordinates. The third improvement is the multi-scale feature extraction and fusion stage. After the initial enhanced feature map output by the OpenPose refinement stage outputs a refined pose limb heatmap, dynamic sampling (Dysample) is used to fused the dynamically sampled feature map with shallow and intermediate network feature maps extracted from the MobileNet V3 large backbone network, finally outputting a high-resolution joint heatmap and joint coordinates.

[0021] In step S103, a frame-level fatigue score is calculated based on the identified joint coordinates, and the original value of cumulative fatigue within the task cycle is calculated.

[0022] In step S104, the desired assist torque is calculated according to the original fatigue accumulation value and a preset exponential nonlinear mapping function, and a control command is generated through a closed-loop control algorithm.

[0023] In step S105, the exoskeleton assist device is driven to output a corresponding assist torque according to the control command, so as to assist the operator.

[0024] The method in this application embodiment can assess the cumulative fatigue risk level of a non-fixed posture operation task (such as complex system assembly and maintenance) over a period of time, and adjust the exoskeleton assistive device accordingly to provide adaptive and smooth assistance to the operator.

[0025] In some embodiments, replacing the depthwise separable convolutions of some inverted residual blocks with dilated convolutions in the backbone network includes: Replace the depthwise separable convolution in the 8th layer inverted residual block with a dilated convolution with a dilation rate of 2; Replace the depthwise separable convolutions in the inverted residual blocks from layer 14 to layer 16 with dilated convolutions with a dilation rate of 4.

[0026] As shown in Table 1. The advantage of this improvement is that by replacing the dilated convolution, the receptive field can be expanded while maintaining the current resolution, thus preserving sufficient feature information. The deep features output are directly input into the subsequent multi-scale gated encoder, while the shallow and mid-level features are used to concatenate with the dynamically sampled output.

[0027] Table 1. Architecture of MobileNet V3 large backbone network after replacing dilated convolutions

[0028] Note: The downsampling stage refers to the stage where the resolution of the output feature map size decreases. The table shows that after the fully connected layer is calculated, the resolution decreases, but the image depth increases, meaning the amount of information is preserved; that is, the receptive field is expanded while the resolution is reduced. In the table, HS is the H-Swish activation function, and RE is the ReLU activation function.

[0029] In some embodiments, the multi-scale gated feature enhancement module specifically includes: The first-level enhancement unit consists of at least three dilated convolutional branches with different dilation rates connected in parallel at the end of the backbone network. These branches extract feature maps of different scales, including shallow, mid-level, and deep feature maps. The feature maps of each branch are adjusted to the same number of channels using a 1×1 convolution and then concatenated along the channel dimensions. The softmax function is used to generate adaptive weights for each feature and then weighted. The weighted feature maps are then summed pixel by pixel and restored to the target number of channels using a 1×1 convolution, outputting the initial feature map.

[0030] In specific examples, such as Figure 2 As shown, the first-level enhancement of the multi-scale gated feature enhancement module in this embodiment is a feature enhancement module consisting of three dilated convolutional branches connected in parallel at the end of the backbone network. It simultaneously extracts features from the three branches (each branch has a different information focus), and then performs gated weighted fusion on these parallel features. The main steps of the gated weighted fusion include: first, using 1*1 convolution to increase / decrease the dimensionality of several features until the number of channels is the same, and then concatenating the channel dimensions; then, using the softmax function to generate adaptive weights and assigning weights to the several features; finally, rearranging and weighting the features, at which point the size and depth of the features are exactly the same, performing pixel-by-pixel addition, using 1*1 convolution to restore the number of channels, and outputting the initial feature map.

[0031] The second-level enhancement unit inputs the initial feature map into the first prediction stage of OpenPose and outputs an initial prediction feature map; after scale-aware normalization processing of the initial feature map and the initial prediction feature map, it performs gated weighted fusion and outputs an initial enhanced feature map. The initial enhanced feature map is used as input, and the refined feature map is output after the second prediction stage of OpenPose.

[0032] In a specific embodiment, the second-level enhancement of the multi-scale gated feature enhancement module involves normalizing and gating the initial feature map output by the aforementioned feature enhancement module with the initial predicted feature map from OpenPose (the first round of coarse prediction results). This normalization process unifies the different scales of features from different layers, preventing large deviations in the joint coordinates after fusion due to scale inconsistencies. Then, weighted fusion is performed to output the initial enhanced feature map.

[0033] The advantage of this improvement lies in its approach of extracting features through multi-branch parallel processing followed by gated weighted fusion. Firstly, this filters out redundancy and noise in the original backbone network's output features, purifying and retaining useful information. Secondly, it captures features with different information priorities at multiple levels, enhancing the receptive field, adjusting scaling imbalances and joint drift, retaining more effective information, and improving the refinement of the original network's output feature map. Finally, the gated weighting computation is extremely low, almost identical to the original network model's output time. This method solves the problem of feature information being retained but location information becoming increasingly blurred due to the concatenated convolutional kernels and continuous downsampling in the original MobileNet V3 large model. This is crucial for providing the joint coordinates and limb connection information required in this application.

[0034] Specific steps are as follows: Figure 3 As shown: First, the original image is input into the MobileNet V3 backbone network, where dilated convolutions replace some stages of the DW convolutions. The input image size is 256*256*3, and the final output deep feature image size is 16*16*160. Simultaneously, a shallow feature map of 64*64*32 and a mid-level feature map of 32*32*64 are cropped.

[0035] Secondly, a multi-scale gated feature enhancement module is connected after the backbone network. The deep feature map (16*16*160) output from the backbone network is used as the module input. After passing through the parallel output of the three branches within the module, it is gated and weighted, resulting in a feature map of size 16*16*128, which becomes the initial feature map. This step filters out redundancy and noise in the original backbone network output features, and performs multi-receptive field multi-scale enhancement on the same deep feature map. This not only solves the problem of inconsistent human proportions in the image due to perspective and distance (addressing the issue of different human body sizes in multiple poses), but also reduces the dimensionality of the feature map, purifying it and retaining useful information.

[0036] Then, the initial feature map (16*16*128) is input into the first stage of OpenPose's initial prediction stage, and the initial prediction feature map (16*16*C, where C is the number of channels in the stage, usually 256 or 128) is output. This initial prediction feature map is then scaled and weighted and fused with the initial feature map to output the initial enhanced feature map (16*16*128).

[0037] The fourth step involves taking the initial enhanced feature map as input and passing it through the second stage of OpenPose's refinement prediction phase to output a refined feature map (16*16*128).

[0038] In some embodiments, the scale-aware normalization process includes: Calculate the global mean and standard deviation of the initial feature map and the initial predicted feature map, respectively; Based on the global mean and standard deviation, each feature map is normalized to a standard normal distribution; The normalized feature map is rescaled using affine parameters.

[0039] Based on the aforementioned example, the specific method of scale-aware normalized SAN includes: firstly, calculating the global mean of the initial feature map and the initial predicted feature map. and standard deviation :

[0040] in, Here, B, C, and H represent the image's length, width, and height, respectively, and W represents the number of channels.

[0041]

[0042] in, This is to avoid fixed constants with zero variance.

[0043] Then, each feature is normalized to a standard normal distribution to obtain the normalized features. , .

[0044] Finally, Rescaling yields the final features , . and Each channel has independent, learnable affine parameters (general mathematical definition), with initial values ​​of 1 and 0 respectively. Multi-stage features are normalized and fused to eliminate conflicts and scale inconsistencies, reducing issues such as joint drift and limb disproportion in the feature map.

[0045] In some embodiments, the dynamic sampling fusion module includes: The first-level upsampling fusion unit dynamically upsamples the refined feature map and performs gated weighted fusion with the extracted mid-layer feature map of the backbone network. The second-level upsampling fusion unit dynamically upsamples the first-level fusion result and performs gated weighted fusion with the extracted shallow feature map of the backbone network to output a joint heatmap.

[0046] Based on the aforementioned example, in a specific example, the refined feature map is input to the dynamic sampling (Dysample) and upsampled to produce an output (32*32*128). This output feature map is then fused with the mid-layer network output feature map (32*32*64) extracted from the MobileNet V3 large backbone network using gated attention, resulting in an output feature map (32*32*64).

[0047] The upsampled output (64*64*64) is then used again with dynamic sampling and gated weighted fusion with the shallow network output feature map (64*64*32) extracted from the MobileNet V3 large backbone network to output a feature map (64*64*32).

[0048] Finally, a 1*1 convolution is used to reduce the dimension to 64*64*14, that is, the resolution is 64*64, and the heatmap of 14 key points is retained.

[0049] In some embodiments, calculating the initial value of accumulated fatigue within a task cycle includes: The fatigue scores corresponding to each frame within the task cycle are weighted by using the time of each frame as a weighting factor to obtain the original value of cumulative fatigue. The fatigue score of a single frame is calculated based on the RULA rapid upper limb assessment method.

[0050] Based on the aforementioned example, using the identified coordinates of 14 joint points in each frame, the relevant posture data for fatigue assessment is calculated. The corresponding data, such as joint angles and distances, are calculated according to the selected posture fatigue assessment method. This application uses the RULA assessment method as an example to calculate limb joint angles, or outputs joint angles using OpenSIM software. Then, fatigue assessment is performed on each frame of the image according to a standard scoring table and grading rules to obtain a fatigue level score for each image. Finally, the cumulative fatigue score within a unit of time is calculated. .

[0051] The specific steps are as follows: For each frame of image, obtain a fatigue score, represented as... This embodiment uses the fatigue accumulation during a single task operation. The fatigue characteristic value within this time period, also known as the cumulative fatigue value, is calculated using a time-weighted average method. . , yes The corresponding RULA rating for the time period. This is the total time, expressed in seconds (s).

[0052] In this specific implementation, the RULA method is used to evaluate fatigue in each frame of the image. The camera samples 30 frames per second. For a specific maintenance operation task, after removing the beginning and end non-operation times, a total sampling time of 15 minutes and 12 seconds is obtained, resulting in 27,360 images and 27,360 corresponding scores. … , The value is 0.033s. This implementation case calculates... It is 4.331.

[0053] The assist device in this application embodiment is mainly a wearable active exoskeleton equipment. The operator wears the equipment to perform the task, and the equipment automatically adjusts the assist based on the fatigue level fed back in real time. The exoskeleton equipment consists of a drive controller, a servo motor reducer module, a torque sensor, and a Bowden wire drive system. The process of automatically adjusting the assist size based on fatigue data is as follows: (1) The drive controller receives the fatigue assessment signal, calculates the assist torque by executing the built-in fatigue and assist mapping control algorithm, and converts the fatigue level into a control command using fuzzy PID closed-loop control; (2) The servo motor and reducer serve as the core power source, and accurately output the target torque upon receiving the command; (3) The torque sensor collects the actual output torque and the actual load of the joint in real time, and transmits the data back to the controller for dynamic closed-loop correction to prevent the assist from being too large or too small; (4) The Bowden wire drive system flexibly transmits the motor power to the joint to enhance the assist.

[0054] This application's embodiments design an exponential nonlinear mapping function between real-time fatigue assessment results and exoskeleton assistive device control parameters, which serves as the input to the drive controller's control algorithm. This ensures that during mild fatigue, the assist torque increases slowly, providing almost no assistance and not interfering with natural movements; while during high-risk fatigue, the assist torque increases rapidly, quickly assisting in offsetting the body's load. Firstly, Normalization processing makes Then, the desired torque is calculated based on an exponential nonlinear mapping function. In some embodiments, the exponential nonlinear mapping function takes the form of:

[0055] in, To provide the desired assist torque, , These are the maximum safe torque values ​​for basic zero-assistance and exoskeleton assistive devices, respectively. For normalized fatigue accumulation score, To enhance the adjustment coefficient, the higher the k value, the greater the degree of enhancement of high-risk factors. For example... In the specific example, k can be 3.

[0056] In some embodiments, generating control commands through a closed-loop control algorithm includes: The actual output torque is collected, and the deviation between the expected torque and the actual torque, as well as the rate of change of the deviation, are calculated. For example, the torque deviation at time g is calculated. deviation change rate , It is the actual output torque collected in real time by the torque sensor.

[0057] The deviation and its rate of change are used as inputs to a fuzzy PID controller. Fuzzy inference is performed according to a preset fuzzy rule base, and the correction values ​​for the PID control parameters are output. In a specific example, [the following is a more detailed explanation of the process]. , As fuzzy control input variables, both are fuzzily divided into 7-level fuzzy language subsets according to the general quantization interval. Based on a pre-written engineering general fuzzy rule library, the fuzzy partitioned... , As input, the PID control parameter reference proportional correction amount Integral correction Differential correction quantity For the output, perform fuzzy logic reasoning. Then, perform defuzzification processing to... , , Convert to numerical correction amount .

[0058] The initial PID reference parameters are added to the correction amount to obtain the current PID control parameters. In other words, the initial PID reference parameters are added to the correction amount to obtain the currently sampled real-time PID control parameters. (Reference ratio) Integral coefficient Differential coefficients .

[0059] The incremental PID algorithm is used to calculate the control output at the current moment, generating the target torque command. In this specific example, a discrete incremental PID algorithm is used to calculate the controller output at the current moment. It is converted into a target torque command according to the communication protocol. Finally, the motor's power is flexibly transmitted to the joints for enhanced assistance.

[0060] This application addresses the problem that feature maps become blurred after dimensionality reduction due to the concatenation of convolutional blocks and inverted residual blocks in lightweight networks followed by downsampling, and that it is difficult to improve positional accuracy after dimensionality upsampling. It employs a two-stage, multi-scale feature parallel extraction and gated fusion approach, followed by inverse upsampling. After the MobileNetV3large backbone network outputs multi-layer features, a two-stage multi-scale feature extraction and fusion module refines the previously blurred pose and limb heatmap features. Dynamic sampling (Dysample) is then used to enhance the heatmap resolution, resulting in a high-resolution joint heatmap and joint coordinates. This method is adaptable to all dynamic working conditions and different user groups, meets the requirements for lightweight deployment, and is more compatible with the integrated control of assistive devices, thus exhibiting good engineering feasibility.

[0061] The method in this application, based on accurate identification of joint posture, first calculates the cumulative fatigue risk value (rather than the instantaneous state); secondly, it incorporates the influence of accumulated muscle strength under load and sustained high-risk posture on fatigue accumulation, coupling these factors to form a fatigue deterioration index, thus improving the fatigue level assessment results. Using this value as the input for graded control of assistive devices better reflects the gradual accumulation of fatigue in industrial operations, resulting in stronger control reliability.

[0062] This application also proposes a fatigue adaptive control device based on human posture recognition, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned fatigue adaptive control method based on human posture recognition.

[0063] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0064] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0066] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A fatigue adaptive control method based on human posture recognition, characterized in that, include: Collect images of the workers' postures during operation; The work posture image is input into a lightweight posture recognition model to identify the coordinates of human joints and limb connection information. The lightweight attitude recognition model includes: a backbone network, a multi-scale gated encoder, and a dynamic sampling fusion module; The backbone network adopts the MobileNet V3 large network architecture, and some of the depthwise separable convolutions of the inverted residual blocks in the backbone network are replaced with dilated convolutions to expand the receptive field and maintain the feature map resolution. The multi-scale gated encoder is connected to the output of the backbone network and includes at least two dilated convolutional branches with different dilation rates arranged in parallel. It is used to extract multi-scale features and perform gated weighted fusion on the deep features output by the backbone network using a multi-scale gated feature enhancement module, and output an initial feature map. The initial feature map is input into the first stage of OpenPose. The initial feature map is then gated and weighted fused with the predicted feature map output by the first stage of OpenPose. After weighted fusion, the feature map is refined by the second stage of OpenPose and output as a refined feature map. The dynamic sampling fusion module is used to upsample the refined feature map output by OpenPose and perform gated attention fusion with at least one intermediate layer feature map extracted from the backbone network to output joint heatmap and joint coordinates. Based on the identified joint coordinates, a frame-level fatigue score is calculated, and the raw value of cumulative fatigue within the task cycle is calculated. Based on the original fatigue accumulation value, the desired assist torque is calculated according to a preset exponential nonlinear mapping function, and control commands are generated through a closed-loop control algorithm. The exoskeleton assist device is driven by the control command to output a corresponding assist torque to assist the worker.

2. The fatigue adaptive control method based on human posture recognition as described in claim 1, characterized in that, The replacement of some depthwise separable convolutions in the backbone network with dilated convolutions includes: Replace the depthwise separable convolution in the 8th layer inverted residual block with a dilated convolution with a dilation rate of 2; Replace the depthwise separable convolutions in the inverted residual blocks from layer 14 to layer 16 with dilated convolutions with a dilation rate of 4.

3. The fatigue adaptive control method based on human posture recognition as described in claim 1, characterized in that, The multi-scale gated feature enhancement module specifically includes: The first-level enhancement unit consists of at least three dilated convolutional branches with different dilation rates connected in parallel at the end of the backbone network. These branches extract feature maps at different scales, including shallow, mid-level, and deep feature maps. The feature maps from each branch are then adjusted to the same number of channels using a 1×1 convolution and concatenated along the channel dimensions. Adaptive weights for each feature are generated using the softmax function and weighted accordingly. The weighted feature maps are then summed pixel-by-pixel and restored to the target number of channels using a 1×1 convolution, outputting the initial feature map. The second-level enhancement unit inputs the initial feature map into the first prediction stage of OpenPose and outputs an initial prediction feature map; after scale-aware normalization processing of the initial feature map and the initial prediction feature map, it performs gated weighted fusion and outputs an initial enhanced feature map. The initial enhanced feature map is used as input, and the refined feature map is output after the second prediction stage of OpenPose.

4. The fatigue adaptive control method based on human posture recognition as described in claim 3, characterized in that, The scale-aware normalization process includes: Calculate the global mean and standard deviation of the initial feature map and the initial predicted feature map, respectively; Based on the global mean and standard deviation, each feature map is normalized to a standard normal distribution; The normalized feature map is rescaled using affine parameters.

5. The fatigue adaptive control method based on human posture recognition as described in claim 3, characterized in that, The dynamic sampling fusion module includes: The first-level upsampling fusion unit dynamically upsamples the refined feature map and performs gated weighted fusion with the extracted mid-layer feature map of the backbone network. The second-level upsampling fusion unit dynamically upsamples the first-level fusion result and performs gated weighted fusion with the extracted shallow feature map of the backbone network to output a joint heatmap.

6. The fatigue adaptive control method based on human posture recognition as described in claim 3, characterized in that, The calculation of the initial cumulative fatigue value within the task cycle includes: The fatigue scores corresponding to each frame within the task cycle are weighted by using the time of each frame as a weighting factor to obtain the original value of cumulative fatigue. The fatigue score of a single frame is calculated based on the RULA rapid upper limb assessment method.

7. The fatigue adaptive control method based on human posture recognition as described in claim 6, characterized in that, The form of the exponential nonlinear mapping function is: in, To provide the desired assist torque, , These are the maximum safe torque values ​​for basic zero-assistance and exoskeleton assistive devices, respectively. To help adjust the coefficient, The normalized fatigue accumulation score.

8. The fatigue adaptive control method based on human posture recognition as described in claim 7, characterized in that, Control commands generated through closed-loop control algorithms include: Collect the actual output torque, and calculate the deviation between the expected torque and the actual torque, as well as the rate of change of the deviation. The deviation and the rate of change of deviation are used as inputs to the fuzzy PID controller. Fuzzy inference is performed according to the preset fuzzy rule base, and the correction amount of the PID control parameters is output. The initial PID baseline parameters are added to the correction amount to obtain the current PID control parameters; The control output is calculated based on the incremental PID algorithm at the current moment, and the target torque command is generated.

9. A fatigue adaptive control device based on human posture recognition, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the fatigue adaptive control method based on human posture recognition as described in any one of claims 1 to 8.

Images