Motion recognition method, device and equipment based on improved ResNet network

Abstract

The invention provides an action recognition method, device and equipment based on an improved ResNet network, and relates to the technical field of action recognition, and the method comprises the steps: obtaining multi-modal biological image data of a target object; through a pre-trained improved ResNet network, a basic action recognition result corresponding to the target object is recognized based on the multi-modal biological image data, and the improved ResNet network comprises a backbone network and a ResNet3D enhancement structure based on residual attention fusion; identifying a pain level identification result corresponding to the target object based on the multi-modal biological image data through a pre-trained pain identification network; and determining a target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result. According to the invention, the accuracy of action recognition can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action recognition, in particular to an action recognition method, device and equipment based on an improved ResNet network. BACKGROUND

[0002] Existing human action recognition models are difficult to accurately extract human action features and relevance and relevance to surrounding environment objects. Taking the detection of myocardial infarction Levine action features as an example, action feature recognition models based on skeleton key points such as ST-GCN are prone to misidentification when facing some similar actions (such as crossing hands to protect the chest and hands covering the chest due to chest pain). The patent application with publication number "CN114863561" proposes a real-time human action recognition method based on key point sequences, which aligns depth and color pictures, extracts key points through the open source Mediapipe model, and uses the PP-TinyPose model to train to achieve the effect of action recognition, which cannot solve the problem of similar actions. The patent application with publication number "CN113343786" proposes a lightweight video action recognition method and system based on deep learning, which proposes to extract features of different spatial and temporal dimensions using three different dimensional convolution kernels and then perform feature fusion. Although this method can alleviate the computational load of 3D Resnet to some extent, it is difficult to guarantee the spatiotemporal consistency of video data, thereby reducing the accuracy of video understanding and leading to low accuracy of action recognition. SUMMARY

[0003] Therefore, the purpose of the present application is to provide an action recognition method, device and equipment based on an improved ResNet network, which can significantly improve the accuracy of action recognition.

[0004] In a first aspect, the present application provides an action recognition method based on an improved ResNet network, comprising: obtaining multi-modal biological image data of a target object; recognizing a basic action recognition result corresponding to the target object based on the multi-modal biological image data through a pre-trained improved ResNet network, the improved ResNet network comprising a backbone network and a ResNet3D enhancement structure based on residual attention fusion; recognizing a pain level recognition result corresponding to the target object based on the multi-modal biological image data through a pre-trained pain recognition network; determining a target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result, the target action recognition result being used to judge whether the target object makes an action conforming to a specified feature.

[0005] In an implementation, the specified feature includes a myocardial infarction Levine action feature.

[0006] In an implementation, the multi-modal biological image data includes wide-view pose image data; an initial action recognition result corresponding to a target object is recognized based on the multi-modal biological image data by a pre-trained improved ResNet network, including: a first feature map of the wide-view pose image data is extracted by a backbone network; a second feature map is obtained by performing multi-stage residual attention fusion on the first feature map based on a ResNet3D enhancement structure based on residual attention fusion, and the initial action recognition result corresponding to the target object is recognized based on the second feature map.

[0007] In an implementation, the ResNet3D enhancement structure based on residual attention fusion includes a plurality of residual networks, and at least one residual unit in the residual network is inserted into a ResNet attention fusion unit; the second feature map is obtained by performing multi-stage residual attention fusion on the first feature map, including: by a current residual network in the ResNet3D enhancement structure, the following operations are performed on an input to-be-processed feature map: the to-be-processed feature map is the first feature map output by the backbone network, or the second feature map output by a previous residual network corresponding to the current residual network: a residual feature map is obtained by performing residual processing on the to-be-processed feature map by a residual unit; in the case where the residual unit is inserted into the ResNet attention fusion unit, a target attention fusion feature map is obtained by performing ResNet attention fusion on the residual feature map by a ResNet attention fusion unit; the next residual unit is continued to be used to perform residual processing on the target attention fusion feature map until a high-dimensional feature map of the target is obtained; wherein the high-dimensional feature map is a feature map output by a terminal residual unit or a ResNet attention fusion unit inserted therein in the current residual network; the to-be-processed feature map and the high-dimensional feature map are fused to obtain the second feature map output by the current residual network.

[0008] In an implementation, the ResNet attention fusion unit includes a convolution branch and an attention branch in parallel, and the attention branch includes a channel attention submodule, a spatial attention submodule, and a temporal attention submodule; the target attention fusion feature map is obtained by performing ResNet attention fusion on the residual feature map, including: a convolution feature map is obtained by performing convolution processing on the residual feature map by the convolution branch; The channel attention submodule, the spatial attention submodule and the time attention submodule in the attention branch are used to respectively perform channel attention processing, spatial attention processing and time attention processing on the residual feature map, and the channel attention feature map, the spatial attention feature map and the time attention feature map obtained are fused to obtain an initial attention fusion feature map; Based on the gating mechanism, the convolution feature map and the initial attention fusion feature map are fused to obtain a target attention fusion feature map.

[0009] In an embodiment, based on the gating mechanism, the convolution feature map and the initial attention fusion feature map are fused to obtain a target attention fusion feature map, comprising: Based on the channel attention feature map, the spatial attention feature map and the time attention feature map, the gating fusion weight corresponding to the convolution branch and the gating fusion weight corresponding to the attention branch are determined; The initial attention fusion feature map is subjected to shape alignment processing; According to the gating fusion weight corresponding to the convolution branch and the gating fusion weight corresponding to the attention branch, the convolution feature map and the initial attention fusion feature map after shape alignment are fused to obtain a target attention fusion feature map.

[0010] In an embodiment, the multi-modal biological image data further includes facial image data, and the pain level recognition result is obtained by a pain recognition network based on the facial image data; based on the basic action recognition result and the pain level recognition result, a target action recognition result corresponding to the target object is determined, comprising: The basic action and its probability value contained in the basic action recognition result are fused with the pain level and its probability value contained in the pain level recognition result to obtain an intermediate action recognition result, and the intermediate action recognition result is used to judge the probability value of the target object making an action conforming to the specified feature; The continuity of the multi-frame intermediate action recognition result is detected to determine the target action recognition result corresponding to the target object based on the result of the continuity detection.

[0011] In a second aspect, the present application further provides an action recognition device based on an improved ResNet network, comprising: An image acquisition module is configured to acquire multi-modal biological image data of a target object; A basic action recognition module is configured to recognize a basic action recognition result corresponding to the target object based on the multi-modal biological image data by using a pre-trained improved ResNet network, wherein the improved ResNet network comprises a backbone network and a ResNet3D enhancement structure based on residual attention fusion. a pain recognition module configured to recognize a pain level recognition result corresponding to the target object based on the multi-modal biological image data by using a pre-trained pain recognition network; a target action recognition module configured to determine a target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result, the target action recognition result being used to determine whether the target object makes an action meeting a specified feature.

[0012] In a third aspect, the present application further provides an electronic device comprising a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of the first aspect.

[0013] In a fourth aspect, the present application further provides a computer readable storage medium storing computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method of any one of the first aspect.

[0014] The method, device and equipment for action recognition based on the improved ResNet network provided by the present application first acquire multi-modal biological image data of a target object; then recognize a basic action recognition result corresponding to the target object based on the multi-modal biological image data by using a pre-trained improved ResNet network, the improved ResNet network comprising a backbone network and a ResNet3D enhancement structure based on residual attention fusion; and recognize a pain level recognition result corresponding to the target object based on the multi-modal biological image data by using a pre-trained pain recognition network; and finally determine a target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result, the target action recognition result being used to determine whether the target object makes an action meeting a specified feature. The above method uses the improved ResNet network and the pain recognition network to respectively process the multi-modal biological image data, to obtain the basic action recognition result and the pain level recognition result, and to comprehensively determine whether the target object makes an action meeting a specified feature, thereby significantly improving the accuracy of action recognition.

[0015] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description, claims and drawings.

[0016] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art of the present application, the drawings required to be used in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0018] Figure 1 A flowchart of a motion recognition method based on an improved ResNet network is provided for an embodiment of the present application. Figure 2 A structure diagram of an improved ResNet network is provided for an embodiment of the present application. Figure 3 A residual unit structure diagram in which a ResNet attention fusion unit is inserted is provided for an embodiment of the present application. Figure 4 A structure diagram of a motion recognition device based on an improved ResNet network is provided for an embodiment of the present application. Figure 5 A structure diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with embodiments. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0020] At present, the recognition accuracy of the existing motion recognition algorithm needs to be improved. Therefore, the present application provides a motion recognition method, device and equipment based on an improved ResNet network, which can significantly improve the accuracy of motion recognition. Specifically, the embodiments of the present application maintain the spatio-temporal consistency of video content, and add a self-attention mechanism to the local key area of the video understanding part to improve the accuracy of video understanding, especially in the rigorous field requiring "gold standard" in the medical field.

[0021] In order to facilitate the understanding of the present embodiment, first, a motion recognition method based on an improved ResNet network disclosed by the present embodiment will be described in detail, referring to Figure 1 A flowchart of a motion recognition method based on an improved ResNet network is provided for an embodiment of the present application. Step S102, acquiring multi-modal biological image data of a target object.

[0022] The multi-modal biological image data includes wide-view pose image data and facial image data. In an example, two cameras can be used for data acquisition, one is a wide-view pose camera located at a higher height, which captures the action and pose information of the target object to obtain wide-view pose image data, and the other is a high-definition camera which is level with the face, which captures the facial image data of the target object.

[0023] In step S104, the pre-trained improved ResNet network is used to identify the basic action recognition result corresponding to the target object based on the multi-modal biological image data.

[0024] The improved ResNet network includes a backbone network and a ResNet3D enhancement structure based on residual attention fusion. The ResNet3D enhancement structure based on residual attention fusion includes multiple stages of residual networks, and at least one residual unit in the residual network is inserted into a ResNet attention fusion unit. In an example, the improved ResNet network can be used to identify the basic action recognition result corresponding to the target object based on the wide-view pose image data. The basic action recognition result can be understood as the specific action made by the target object and the corresponding probability value, for example, the probability value of the target object making a "double-hand chest-hugging" action.

[0025] In step S106, the pre-trained pain recognition network is used to identify the pain level recognition result corresponding to the target object based on the multi-modal biological image data.

[0026] The pain recognition network can use a YOLOv11 network. In an example, the YOLOv11 network can be used to identify the pain level recognition result corresponding to the target object based on the facial image data, which is the degree of facial expression pain of the target object.

[0027] In step S108, the basic action recognition result and the pain level recognition result are used to determine the target action recognition result corresponding to the target object.

[0028] The target action recognition result is used to determine whether the target object makes an action conforming to a specified feature, for example, the specified feature includes a myocardial infarction Levine action feature, and is used to determine whether the target object makes a double-hand chest-hugging action due to pain caused by myocardial infarction. In addition, it can also be extended to other action recognition scenarios, such as falling, asthma, standing, sitting, lying, etc. In an example, the target object makes an action conforming to a specified feature can be determined by a multi-modal recognition result and continuity detection, for example, the basic action recognition result and the pain level recognition result at the same time are fused to preliminarily determine whether the target object makes an action conforming to a specified feature, which is recorded as an intermediate action recognition result, and it is further determined whether the intermediate action recognition result continuously appears multiple frames of results, all of which are “the target object makes an action conforming to a specified feature (for example, the probability value of making the action is greater than a preset threshold)”. If so, it can be finally determined that the target action recognition result is that the target object makes an action conforming to a specified feature.

[0029] The action recognition method based on the improved ResNet network provided by the embodiment of the application uses the improved ResNet network and the pain recognition network to process multi-modal biological image data respectively, to obtain a basic action recognition result and a pain level recognition result, and the two results are used to comprehensively determine whether the target object makes an action conforming to a specified feature, thereby significantly improving the accuracy of action recognition.

[0030] Considering that the traditional ResNet3D relies on a 3D convolution with a fixed receptive field to capture short-term spatiotemporal local information, it is difficult to model long-time dependence, cross-frame relationship and semantic invariance. The fine local dynamic capture and deployment efficiency of the traditional Transformer when applied on the terminal side are easily limited. Therefore, the embodiment of the application optimizes the overfitting problem that is prone to occur in a large amount of video data of ResNet3D, and optimizes the problem that the output accuracy is not high when a small amount of data is trained under the condition of insufficient medical public data, and proposes a residual attention fusion ResNet3D enhancement structure, adds a self-attention mechanism, extracts features of key regions of the video, and fuses double attention, which considers the importance of channels and also preserves the key frames in time sequence, and is more robust for action recognition. Meanwhile, compared with the Transformer, the embodiment of the application is simpler, uses channel-space-time multi-branch attention to gate the 3D convolution branch and the lightweight spatiotemporal attention branch, aligns and normalizes fusion, and improves the action / time recognition accuracy under the same or lower computational complexity, robustness and real-time deployment.

[0031] For ease of understanding, this embodiment of the invention takes the Levine action of myocardial infarction as an example and provides an action recognition method based on an improved ResNet network. This method is applied to an action recognition system and consists of four parts: a data acquisition end, a front-end access module, a deep model inference engine, and a result post-processing output layer. (1) Data acquisition end: composed of industrial / consumer camera. In order to ensure the accuracy of action recognition, this system uses two cameras for data acquisition. One is a wide-angle pose camera located at a higher height to capture the patient's action and pose information, and the other is a high-definition camera at the same height as the face to capture the patient's face. Both cameras control the frame rate at 15-30 FPS and the resolution at 2K. (2) Front-end access module: video stream pulling, decoding, buffer sampling, exposure setting, and other preprocessing. (3) Deep model inference engine (including improved ResNet network and pain recognition network), wherein the improved ResNet network includes the backbone network: ResNet3D-18, supporting (THW) = (8-64, 112-320, 112-320), where T is the number of frames of consecutive images, and H and W are the height and width of the image resolution, respectively; ResNet3D enhancement structure based on residual attention fusion: the ResNet attention fusion module (ResNet-Attention-Fusion for short, RAF module) is inserted into at least one residual unit (basic / bottleneck) in each stage to complete multi-branch attention fusion. (4) Post-processing output of results: the results output by the improved ResNet network and the pain recognition network are fused to reduce the probability of misidentification.

[0032] Based on the above system architecture, this embodiment of the invention provides a specific implementation of an action recognition method based on an improved ResNet network.

[0033] (a) Acquire wide-view pose image data and facial image data of the target object.

[0034] (ii) Using a pre-trained improved ResNet network, the initial action recognition result of the target object is identified based on wide-view pose image data.

[0035] First, the structure of the improved ResNet network will be explained. See [link to relevant documentation]. Figure 2 The diagram shows a schematic of an improved ResNet network, which includes a backbone network and a ResNet3D enhancement structure based on residual attention fusion. The ResNet3D enhancement structure based on residual attention fusion includes a residual network with multiple stages, and at least one residual unit in the residual network is inserted into the ResNet attention fusion unit. Figure 2 (Not shown).

[0036] The residual network includes a plurality of residual units, and the residual units can adopt a Basic module or a Bottleneck module. Specifically, the Basic module includes: a 3x3x3 three-dimensional convolution layer (Conv), a batch normalization layer (Batch Normalization, BN), a ReLU activation function, another 3x3x3 three-dimensional convolution layer, and a batch normalization layer connected in sequence; the output of the module and the feature map of the input are added element by element through a residual connection (shortcut connection) to realize an identity mapping; the Bottleneck module: contains three convolution layers operated in sequence, first through a 1x1x1 convolution for channel dimension reduction, then connected with a 3x3x3 spatial-temporal convolution to extract spatial-temporal features, and finally through another 1x1x1 convolution to realize channel dimension increase; the structure also introduces a residual connection to add the processed feature map and the input feature map; the downsampling mode: the spatial or temporal dimension can be down-sampled by setting the convolution stride to 2, wherein the temporal and spatial dimension down-sampling can be controlled independently (i.e. decoupled down-sampling); or a 3D pooling layer (such as 3D MaxPooling) can be used for overall down-sampling.

[0037] Further, at least one residual unit in the residual network is inserted with a ResNet attention fusion unit, such as Figure 3 A residual unit structure diagram inserted with a ResNet attention fusion unit is shown. Specifically, the ResNet attention fusion unit can be inserted into at least one residual unit in the residual network according to a preset insertion rule, for example, the ResNet attention fusion unit is inserted into each residual unit of stage2 (stage 2) to stage3 (stage 3) in the residual network, and the residual unit of the subsequent stage is inserted with the ResNet attention fusion unit with a probability of 50%.

[0038] The ResNet attention fusion unit includes a convolution branch and an attention branch in parallel, and the attention branch includes a channel attention submodule, a spatial attention submodule, and a temporal attention submodule. Specifically: (1) Convolution branch: The 3D convolution output from the residual unit is denoted as a convolution feature map ; (2) Attention branch: A lightweight spatial-temporal attention module, The 3D convolution output from the residual unit is denoted as an initial attention fusion feature map . The attention branch specifically includes: (2.1) The expression of channel attention (using Squeeze-and-Excitation / Efficient Channel Attention) is as follows: , wherein, is the result of channel attention processing, that is, a channel attention feature map, is the 3D convolution output of the residual unit, that is, a residual feature map, represents a multilayer perceptron, represents a global average pooling layer, is a sigmoid activation function; (2.2) The expression of spatial attention (Spatial) is as follows: ; wherein, is the result of spatial attention processing, that is, a spatial attention feature map, represents 2D convolution processing, represents average pooling, represents maximum pooling, is a sigmoid activation function; (2.3) The expression of temporal attention (Temporal) is as follows: , wherein, is the result of temporal attention processing, that is, a temporal attention feature map, represents a normalized exponential function, is a feature map obtained by performing frame mean, patch or convolution downsampling processing on the residual feature map , and is a weight matrix.

[0039] On the basis of the above network structure, the basic action of the target object can be identified according to the following steps: Step 1: extracting a first feature map of wide-view pose image data through a backbone network; Step 2: performing multi-stage residual attention fusion on the first feature map through a ResNet3D enhanced structure based on residual attention fusion to obtain a second feature map, and identifying an initial action recognition result corresponding to the target object based on the second feature map.

[0040] In an implementation, the following operations are performed by a current residual network in the ResNet3D enhanced structure for an input to-be-processed feature map, which is the first feature map output by the backbone network or the second feature map output by a previous residual network corresponding to the current residual network: Step 2.1, residual processing is performed on the to-be-processed feature map by using the residual unit to obtain a residual feature map. In an example, the residual processing can be performed on the to-be-processed feature map by using the Basic module or the Bottleneck module to obtain the residual feature map. .

[0041] Step 2.2, in the case where the ResNet attention fusion unit is inserted in the residual unit, ResNet attention fusion is performed on the residual feature map by using the ResNet attention fusion unit to obtain a target attention fusion feature map.

[0042] In actual application, if the ResNet attention fusion unit is not inserted in the residual unit, the residual feature map is continuously input to the next residual unit for processing; if the ResNet attention fusion unit is inserted in the residual unit, the residual feature map is processed by using the ResNet attention fusion unit. Step 2.3, the target attention fusion feature map is output. The following operations (1) to (3) are performed: (1) the residual feature map is processed by convolution through the convolution branch to obtain a convolution feature map; (2) the residual feature map is processed by channel attention, spatial attention and temporal attention through the channel attention sub-module, the spatial attention sub-module and the temporal attention sub-module in the attention branch, respectively, and the obtained channel attention feature map, the spatial attention feature map and the temporal attention feature map are fused to obtain an initial attention fusion feature map. The attention processing process can be referred to the introduction of the attention branch in the foregoing, and details are not described herein again.

[0043] (3) the convolution feature map and the initial attention fusion feature map are fused based on the gating mechanism to obtain the target attention fusion feature map. Specifically, the gating fusion weight corresponding to the convolution branch and the gating fusion weight corresponding to the attention branch are determined based on the channel attention feature map, the spatial attention feature map and the temporal attention feature map; the initial attention fusion feature map is processed by shape alignment; the convolution feature map and the initial attention fusion feature map after shape alignment are fused based on the gating fusion weight corresponding to the convolution branch and the gating fusion weight corresponding to the attention branch to obtain the target attention fusion feature map.

[0044] In specific implementation, the gating fusion weights are first determined according to the following formula: ; wherein, represents element-wise multiplication, , the gating fusion weights corresponding to the convolution branch and the attention branch, respectively, and the constraint , ensure energy conservation and numerical stability, sigmoid activation function, , is the weight matrix corresponding to the convolution branch and the attention branch, , , is the channel attention feature map, the spatial attention feature map and the time attention feature map.

[0045] Then use the alignment calibration to the result of the attention fusion processing Shape alignment, ensure the consistency of the output dimension, finally, the convolution feature map and the initial attention fusion feature map after shape alignment are gate fusion, the expression of gate fusion is as follows: , wherein, is the target attention fusion feature map, is the input residual feature map, , respectively the gate fusion weight corresponding to the convolution branch and the attention branch, is the convolution feature map, is the initial attention fusion feature map after alignment.

[0046] The embodiment of the application is inside / bypass of ResNet3D residual unit, The learnable and constrained gate realizes the energy conservation fusion of the convolution branch and the attention branch, improves the expression stability and gradient controllability. In addition, the multi-branch attention of channels, space and time is used as the gate input, which shows the distinction between short-term local and long-term dependence.

[0047] Step 2.3, continue to use the next residual unit to perform residual processing on the target attention fusion feature map until the target high-dimensional feature map is obtained; wherein the high-dimensional feature map is the feature map output by the last residual unit in the current residual network or the inserted ResNet attention fusion unit.

[0048] In the specific implementation, the target attention fusion feature is continuously input to the next residual unit, and the foregoing steps 2.1 to 2.2 are repeated until it is input to the last residual unit. If the last residual unit is not inserted with the ResNet attention fusion unit, the residual feature map output by the last residual unit is taken as the high-dimensional feature map. If the last residual unit is inserted with the ResNet attention fusion unit, the target attention fusion feature map output by the inserted ResNet attention fusion unit of the last residual unit is taken as the high-dimensional feature map.

[0049] Step 2.4, fusing the to-be-processed feature map and the high-dimensional feature map to obtain a second feature map output by the current residual network.

[0050] Further, the embodiment of the present application also provides a training process of the improved ResNet network, and the actual ResNet3D spatio-temporal attention fusion model inputs a picture with a size of 320x240, continuously inputs 12 frames of images for action prediction, adopts a ResNet3D-18 layer model, inserts a ResNet attention fusion unit into each residual unit of stage2 to stage3 in the residual network, and inserts a ResNet attention fusion unit into the residual unit of the subsequent stage with a probability of 50%. The loss function adopts cross entropy and RAF regularization, so that stable and sparse. The optimizer selects AdamW, and the predicted user state is the Levine feature of whether there is myocardial infarction. The test hardware environment is a 12th-i7-12700KF processor and an RTX3080 graphics card, the time required for inference of 12 pictures is 250ms, the camera frame rate is 20fps, and the speed requirement of real-time inference is met. In the embodiment of the present application, through the RAF regularization and consistency constraint, the gating and multi-view input regularization / consistency loss can inhibit unstable fusion and overfitting.

[0051] (Three) identifying the pain level recognition result corresponding to the target object based on the facial image data through the pre-trained pain recognition network. Specifically, the facial image data can be processed by using a YOLOv11 network to infer and identify the pain level recognition result corresponding to the target object.

[0052] (Four) determining the target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result. In specific implementation, the following steps are included: Step 1, fusing the basic action and its probability value included in the basic action recognition result, and the pain level and its probability value included in the pain level recognition result to obtain an intermediate action recognition result, the intermediate action recognition result being used to judge the probability value of the target object making an action conforming to a specified feature.

[0053] In one embodiment, for the basic action recognition result and the pain level recognition result corresponding to the wide-view pose image data and the facial image data collected at the same time, in the case that the probability of the basic action being double-hand chest-hugging is greater than a preset threshold, and the probability of the pain level being pain is greater than a preset threshold, the intermediate action recognition result corresponding to the moment is determined as the target object making an action conforming to a specified feature.

[0054] Step 2, continuity detection is performed on the multi-frame intermediate action recognition result, to determine the target action recognition result corresponding to the target object based on the result of continuity detection. For example, in the case that the results of continuous multiple frames are all that the target object makes the action conforming to the specified feature, it is finally determined that the target object makes the action conforming to the specified feature, otherwise it is determined that the target object does not make the action conforming to the specified feature.

[0055] The embodiment of the application can effectively solve the problem of low resolution of the input picture of the action model by the dual-camera fusion scheme, and improve the high accuracy required in the medical scene.

[0056] In summary, the method provided by the embodiment of the application is realized by two cameras, and due to the real-time nature of the inference action model, the picture input of the pose data is down-sampled to 320x240, which results in that the recognition of the facial features of the user does not reach the expected accuracy, so the recognition of the facial symptoms is separately collected by the facial camera, and finally the results of the two parts are comprehensively evaluated and output. The method has at least the following characteristics: the RAF model optimized by the embodiment of the application has an action recognition accuracy of more than 90% based on 2GB of picture data cut from 120 video data collected by experiments, while the action recognition model of the traditional ResNet3D or TSM (Temporal Shift Module) has an accuracy of only about 60%. Under the condition that the deployment and inference speed of the model does not differ much, the optimized action recognition model of the embodiment of the application can ensure accuracy and speed. At the same time, the improved model can effectively solve the overfitting and generalization problems of the traditional ResNet3D network for small sample data. In addition, in the case of insufficient facial feature recognition after down-sampling of the action collection picture, the comprehensive state evaluation method of the dual-camera provided by the embodiment of the application can improve the recognition accuracy of the myocardial infarction Levine feature to 99%. The action recognition model is not limited to the recognition of the myocardial infarction Levine feature, and can be extended to other action recognition scenes, such as the fall danger action monitoring of industrial plants, asthma, standing, sitting, and lying medical scenes, to provide a more robust discrimination model.

[0057] On the basis of the foregoing embodiment, the embodiment of the application provides an action recognition device based on the improved ResNet network, which refers to Figure 4 The device mainly includes the following parts: The image acquisition module 402 is configured to acquire multi-modal biological image data of a target object. The basic action recognition module 404 is configured to recognize a basic action recognition result corresponding to the target object based on the multi-modal biological image data by using a pre-trained improved ResNet network, and the improved ResNet network includes a backbone network and a ResNet3D enhancement structure based on residual attention fusion. The pain recognition module 406 is configured to recognize a pain level recognition result corresponding to the target object based on the multi-modal biological image data by using a pre-trained pain recognition network. The target action recognition module 408 is configured to determine a target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result, and the target action recognition result is used to describe whether the target object makes an action conforming to a specified feature.

[0058] The action recognition device based on the improved ResNet network provided in the embodiments of the present application uses the improved ResNet network and the pain recognition network to process the multi-modal biological image data respectively, to obtain the basic action recognition result and the pain level recognition result, and the basic action recognition result and the pain level recognition result are used to comprehensively determine whether the target object makes an action conforming to a specified feature, so that the accuracy of action recognition is significantly improved.

[0059] In an embodiment, the specified feature includes a myocardial infarction Levine action feature.

[0060] In an embodiment, the multi-modal biological image data includes wide-view pose image data, and the basic action recognition module 404 is specifically configured to: extract a first feature map of the wide-view pose image data by using the backbone network; perform multi-stage residual attention fusion on the first feature map by using the ResNet3D enhancement structure based on residual attention fusion to obtain a second feature map, and recognize an initial action recognition result corresponding to the target object based on the second feature map.

[0061] In an embodiment, the ResNet3D enhancement structure based on residual attention fusion includes a plurality of residual networks, and at least one residual unit in the residual network is inserted into a ResNet attention fusion unit, and the basic action recognition module 404 is specifically configured to: perform the following operations on an input to-be-processed feature map by using a current residual network in the ResNet3D enhancement structure, the to-be-processed feature map being the first feature map output by the backbone network or the second feature map output by a previous residual network corresponding to the current residual network: perform residual processing on the to-be-processed feature map by using the residual unit to obtain a residual feature map; In the case that the residual unit inserts the ResNet attention fusion unit, the ResNet attention fusion unit is utilized to perform ResNet attention fusion on the residual feature map to obtain a target attention fusion feature map; The target attention fusion feature map is continuously processed by the next residual unit until a high-dimensional feature map of the target is obtained; the high-dimensional feature map is a feature map output by a terminal residual unit or a ResNet attention fusion unit inserted in the current residual network; The high-dimensional feature map and the to-be-processed feature map are fused to obtain a second feature map output by the current residual network.

[0062] In an embodiment, the ResNet attention fusion unit includes a convolution branch and an attention branch in parallel, and the attention branch includes a channel attention submodule, a spatial attention submodule and a temporal attention submodule; the base action recognition module 404 is specifically configured to: perform convolution processing on the residual feature map through the convolution branch to obtain a convolution feature map; perform channel attention processing, spatial attention processing and temporal attention processing on the residual feature map through the channel attention submodule, the spatial attention submodule and the temporal attention submodule in the attention branch, respectively, and fuse the obtained channel attention feature map, spatial attention feature map and temporal attention feature map to obtain an initial attention fusion feature map; fuse the convolution feature map and the initial attention fusion feature map based on a gating mechanism to obtain the target attention fusion feature map.

[0063] In an embodiment, the base action recognition module 404 is specifically configured to: determine gating fusion weights corresponding to the convolution branch and gating fusion weights corresponding to the attention branch based on the channel attention feature map, the spatial attention feature map and the temporal attention feature map; perform shape alignment processing on the initial attention fusion feature map; fuse the convolution feature map and the shape-aligned initial attention fusion feature map based on the gating fusion weights corresponding to the convolution branch and the gating fusion weights corresponding to the attention branch to obtain the target attention fusion feature map.

[0064] In an embodiment, the multi-modal biological image data further includes facial image data, and the pain level recognition result is obtained by a pain recognition network based on the facial image data; the target action recognition module 408 is specifically configured to: The basic action and its probability value contained in the basic action recognition result are fused with the pain level and its probability value contained in the pain level recognition result to obtain an intermediate action recognition result, and the intermediate action recognition result is used to judge a probability value of the target object making an action conforming to the specified feature. The continuity of the multiple intermediate action recognition results is detected, and a target action recognition result corresponding to the target object is determined based on a result of the continuity detection.

[0065] The implementation principle and the technical effects of the device provided in the embodiments of the present application are the same as those of the foregoing method embodiments, and for brevity of description, the part not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments.

[0066] The embodiments of the present application provide an electronic device, specifically, the electronic device includes a processor and a memory; the memory stores a computer program, and the computer program performs the method of any one of the above embodiments when executed by the processor.

[0067] Figure 5 The structure schematic diagram of the electronic device provided in the embodiments of the present application, the electronic device 100 includes: a processor 50, a memory 51, a bus 52 and a communication interface 53, the processor 50, the communication interface 53 and the memory 51 are connected through the bus 52; the processor 50 is used to execute the executable module stored in the memory 51, for example, a computer program.

[0068] The memory 51 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 53 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0069] The bus 52 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5 Only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0070] The memory 51 is used to store a program, and the processor 50 executes the program after receiving an execution instruction. The method executed by the device defined in the flow process of any one of the foregoing embodiments of the present application can be applied to the processor 50 or realized by the processor 50.

[0071] The processor 50 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuit of hardware in the processor 50 or by instructions in the form of software. The processor 50 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 51, and the processor 50 reads the information in the memory 51, and combines the hardware to complete the steps of the above method.

[0072] The computer program product of the readable storage medium provided by the embodiments of the present application includes a computer readable storage medium storing program codes, and the program codes include instructions for executing the method described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be described here.

[0073] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0074] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An action recognition method based on an improved ResNet network, characterized in that, include: Acquire multimodal biological image data of the target object; The improved ResNet network, which is pre-trained, identifies the basic action recognition results of the target object based on the multimodal biological image data. The improved ResNet network includes a backbone network and a ResNet3D enhancement structure based on residual attention fusion. The pain level of the target object is identified based on the multimodal biological image data using a pre-trained pain recognition network. Based on the basic action recognition result and the pain level recognition result, the target action recognition result corresponding to the target object is determined. The target action recognition result is used to determine whether the target object performs an action that conforms to the specified characteristics.

2. The action recognition method based on the improved ResNet network according to claim 1, characterized in that, The specified features include the Levine action feature for myocardial infarction.

3. The action recognition method based on the improved ResNet network according to claim 1 or 2, characterized in that, The multimodal biological image data includes wide-view pose image data; based on the multimodal biological image data, the initial action recognition result corresponding to the target object is identified using a pre-trained improved ResNet network, including: The first feature map of the wide-view pose image data is extracted through the backbone network. The ResNet3D enhancement structure based on residual attention fusion is used to perform multi-stage residual attention fusion on the first feature map to obtain a second feature map, and the initial action recognition result corresponding to the target object is identified based on the second feature map.

4. The action recognition method based on the improved ResNet network according to claim 3, characterized in that, The ResNet3D enhancement structure based on residual attention fusion includes multiple residual networks, and at least one residual unit in the residual network is inserted into the ResNet attention fusion unit; The second feature map is obtained by performing multi-stage residual attention fusion on the first feature map, including: The current residual network in the ResNet3D enhancement structure performs the following operation on the input feature map to be processed, where the feature map to be processed is either the first feature map output by the backbone network or the second feature map output by the previous residual network corresponding to the current residual network: The residual unit is used to perform residual processing on the feature map to be processed to obtain a residual feature map; When the ResNet attention fusion unit is inserted into the residual unit, the ResNet attention fusion unit is used to perform ResNet attention fusion on the residual feature map to obtain the target attention fusion feature map; The target attention fusion feature map is further processed by the next residual unit until a high-dimensional feature map is obtained; wherein the high-dimensional feature map is the feature map output by the residual unit at the end of the current residual network or the ResNet attention fusion unit inserted therein; The feature map to be processed and the high-dimensional feature map are fused to obtain the second feature map output by the current residual network.

5. The action recognition method based on the improved ResNet network according to claim 4, characterized in that, The ResNet attention fusion unit includes parallel convolutional branches and attention branches, and the attention branches include channel attention submodules, spatial attention submodules and temporal attention submodules; The residual feature map is fused using ResNet attention to obtain the target attention fusion feature map, including: The residual feature map is convolved through the convolution branch to obtain a convolutional feature map; The residual feature map is processed by the channel attention submodule, spatial attention submodule and temporal attention submodule within the attention branch, respectively, and the obtained channel attention feature map, spatial attention feature map and temporal attention feature map are fused to obtain an initial attention fusion feature map. Based on the gating mechanism, the convolutional feature map and the initial attention fusion feature map are gated and fused to obtain the target attention fusion feature map.

6. The action recognition method based on the improved ResNet network according to claim 5, characterized in that, Based on a gating mechanism, the convolutional feature map and the initial attention fusion feature map are gated and fused to obtain the target attention fusion feature map, including: Based on the channel attention feature map, the spatial attention feature map, and the temporal attention feature map, the gating fusion weights corresponding to the convolutional branches and the gating fusion weights corresponding to the attention branches are determined. The initial attention fusion feature map is then shaped and aligned. According to the gate fusion weights corresponding to the convolutional branch and the gate fusion weights corresponding to the attention branch, the convolutional feature map and the shape-aligned initial attention fusion feature map are gated and fused to obtain the target attention fusion feature map.

7. The action recognition method based on the improved ResNet network according to claim 1 or 2, characterized in that, The multimodal biological image data also includes facial image data, and the pain level recognition result is obtained by the pain recognition network based on the facial image data; Based on the basic action recognition results and the pain level recognition results, the target action recognition results corresponding to the target object are determined, including: The basic actions and their probability values ​​included in the basic action recognition result are fused with the pain levels and their probability values ​​included in the pain level recognition result to obtain intermediate action recognition results. The intermediate action recognition results are used to determine the probability value of the target object performing an action that conforms to the specified characteristics. The continuity detection is performed on the intermediate action recognition results of multiple frames to determine the target action recognition result corresponding to the target object based on the results of the continuity detection.

8. An action recognition device based on an improved ResNet network, characterized in that, include: The image acquisition module is used to acquire multimodal biological image data of the target object; The basic action recognition module is used to recognize the basic action recognition result of the target object based on the multimodal biological image data through a pre-trained improved ResNet network. The improved ResNet network includes a backbone network and a ResNet3D enhancement structure based on residual attention fusion. The pain recognition module is used to identify the pain level of the target object based on the multimodal biological image data through a pre-trained pain recognition network. The target action recognition module is used to determine the target action recognition result corresponding to the target object based on the basic action recognition result and the pain level recognition result. The target action recognition result is used to determine whether the target object has performed an action that conforms to the specified characteristics.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images