Method for training 2D convolutional neural network for user action recognition

Abstract

The purpose of each embodiment of the present disclosure is to provide a method, a computer program product and a computer program storage medium for training a 2D convolutional neural network for user action recognition. Among them, a computer device obtains a sample set of action images, and inputs the sample image sequence into a 2D convolutional neural network to train it for action recognition. The feature extraction process in the embodiment of the present disclosure is completely automatic and does not require human participation. The features learned by the action recognition model have better generalization performance. The action classification from the original video frame input to the key point is an end-to-end process. The algorithm is simple and involves few links. By adding the channel dimension and the front and back feature sequence dimension in the middle, the features of the parameter classification are richer and more dimensional, so the final action recognition effect is better and the accuracy is higher.

Description

Technical Field

[0001] The various embodiments of the present disclosure relate to the fields of computer vision and artificial intelligence, and in particular to a technology for training a 2D convolutional neural network for user action recognition. Background Art

[0002] The standardization of movements is particularly important during sports rehabilitation. Therefore, how to correctly conduct rehabilitation training through exercise is a crucial issue. Irregular movements will lead to half the effort with twice the results in the rehabilitation process, or poor results. In more serious cases, it will lead to the failure of rehabilitation and even serious treatment risks. Hiring a professional rehabilitation therapist is a cost comparison for the person being treated. On the other hand, since the person being treated may have some problems with his or her movements, it is not very appropriate to go to a special offline institution for rehabilitation from the perspective of cost and convenience.

[0003] In order to solve the above series of problems, some digital rehabilitation systems have begun to try to use wearable devices for assistance, such as sports bracelets, sensor devices, etc. This requires users to purchase specific equipment and actively participate, which is cumbersome. In addition, the installation of auxiliary sensors and other equipment has strong coupling, high cost, and is inconvenient. Summary of the invention

[0004] The purpose of each embodiment of the present disclosure is to provide a method, a computer program product, and a computer program storage medium for training a 2D convolutional neural network for user action recognition.

[0005] According to one aspect of the present disclosure, a method for training a 2D convolutional neural network for user action recognition is provided, wherein the method comprises the following steps:

[0006] Acquire an action image sample set, wherein the sample image sequence includes a standard action image sequence and a non-standard action image sequence, wherein each of the sample image sequences includes a plurality of continuous images and is annotated with corresponding action classification labels, and wherein each sample image is annotated with human body key points;

[0007] The sample image sequence is input into a 2D convolutional neural network to train it for action recognition, wherein:

[0008] Performing multi-channel feature extraction on the sample image sequence to obtain a feature map corresponding to each channel, wherein the multi-channel includes an I frame channel, an MV channel, and an RGBR channel;

[0009] The obtained feature maps are spliced ​​and fused, and the key points of the human body are recognized for the sample image sequence by combining the feature importance of each feature and the attention mechanism of the key points;

[0010] The splicing and fusion specifically includes:

[0011] The feature maps extracted from the I frame channel, MV channel, and RGBR channel are spliced ​​in the channel dimension and a convolution operation is performed on the spliced ​​feature maps to obtain a fused feature map;

[0012] The feature importance of each feature is calculated based on the following method:

[0013] E ij =ReLU <C r |W s |H t >

[0014] Among them, E ij represents the feature importance of key point i relative to any adjacent key point j, C r represents the channel, W s Indicates width, H t Indicates height;

[0015] The key point attention mechanism is calculated based on the following method:

[0016]

[0017] Among them, A ij is the relative attention value between key point i and any adjacent key point j, N i is the set of neighboring key points of key point i, k is the index of any key point in the set Ni, E ik is the attention value of key point i and any key point k in the set Ni;

[0018] Combining the temporal characteristics of the identified human body key points, performing action classification and recognition on the sample image sequence;

[0019] Calculating a loss function based on the recognition result of the action classification and the action classification label annotated by the corresponding sample image sequence;

[0020] The above steps are repeated until the loss function converges, and the training of the 2D convolutional neural network is completed.

[0021] In some embodiments, the step of constructing the action image sample set includes:

[0022] Video recording based on standard and non-standard movements;

[0023] Label the human body key points, action classification labels and product pipelines corresponding to the standardized action image sequences;

[0024] According to the product pipeline to which each standard action image sequence belongs, it is stored in the corresponding feature template.

[0025] Among them, it also includes:

[0026] Label the human body key points, action classification labels, error types and product pipelines corresponding to the non-standard action image sequences;

[0027] Each non-standard action image sequence is stored in a corresponding feature template according to the product pipeline to which it belongs.

[0028] In some embodiments, before performing multi-channel feature extraction on the sample image sequence, the sample image sequence is also normalized.

[0029] In some embodiments, the multi-channel feature extraction specifically includes:

[0030] The sample image sequence is input into three ResNet18 models to extract feature maps of the I frame channel, MV channel and RGBR channel respectively.

[0031] The key points of the human body are identified based on the following method:

[0032]

[0033] Among them, P is the convolution kernel parameter of the spatial dimension, M p is the relative attention value A ij The spatial attention matrix, X in is the fused feature map, W p is the feature weight.

[0034] In some embodiments, the method further comprises:

[0035] Acquire multiple consecutive video frames of the user's action;

[0036] According to the multiple continuous video frames, the action classification of the user action is identified by the trained 2D convolutional neural network.

[0037] The multiple continuous video frames are extracted from the acquired user action video at a certain frequency.

[0038] According to one aspect of the present disclosure, a computer program product is also provided, comprising computer program instructions, wherein when the computer program instructions are executed by a computer device, the computer device is configured to execute a method for training a 2D convolutional neural network for user action recognition according to one aspect of the present disclosure.

[0039] According to one aspect of the present disclosure, a computer program storage medium is also provided, which stores computer executable instructions for reading database data into GPU video memory. When the computer executable instructions are executed by a computer device, the computer device is configured to execute a method for training a 2D convolutional neural network for user action recognition according to one aspect of the present disclosure.

[0040] With the development of CNN technology, the embodiments of the present disclosure collect user video data through mobile phone cameras, identify key points of the human body through computer vision human body key point capture technology, and then judge human body movements, and then analyze human body movements, which has extremely important and irreplaceable value in clinical practice. In this way, rehabilitation training can be carried out conveniently, efficiently and at a low cost.

[0041] The embodiments of the present disclosure take depth video as the research object, construct a 2D convolutional neural network, and introduce multi-channel feature extraction in the constructed convolutional neural network structure. By performing convolution on each channel separately, the extracted features are richer and the processing is more detailed. In the process of processing a single channel, the historical sequence feature information before and after the current feature is referenced. The training parameters are dynamically adjusted during the training process through the forward propagation and back propagation mechanisms, and finally the most effective model output is obtained.

[0042] The feature extraction process in the disclosed embodiment is completely automatic and does not require human intervention. The features learned by the action recognition model have better generalization performance. The process from raw video frame input to key point action classification is an end-to-end process. The algorithm is simple and involves few links. By adding channel dimensions and front and back feature sequence dimensions in the middle, the features of parameter classification are richer and have more dimensions. Therefore, the final action recognition effect is better and the accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0044] Figure 1 A flowchart of a method for training a 2D convolutional neural network for user action recognition according to an embodiment of the present disclosure is shown;

[0045] Figure 2 A schematic diagram showing a normalization process for RGB values ​​according to an example of the present disclosure;

[0046] Figure 3 A schematic diagram showing calculation of feature importance and key point attention values ​​based on concatenated and fused feature maps according to an example of the present disclosure.

[0047] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0048] The specific embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0049] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments of the present disclosure are described as devices represented by block diagrams and processes or methods represented by flow charts. Although the flow charts describe the operating processes of the various embodiments of the present disclosure as sequential processing, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The processes of the various embodiments of the present disclosure can be terminated when their operations are completed, but can also include additional steps not shown in the flow charts. The processes of the various embodiments of the present disclosure can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0050] The methods illustrated by the flow charts and the devices illustrated by the block diagrams discussed below may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium. (One or more) processors may perform the necessary tasks.

[0051] Similarly, it will also be understood that any flow charts, flow diagrams, state transition diagrams, and the like represent various processes that can be fully described as program code stored in a computer readable medium and thereby executed by a computer device or processor, whether or not such computer device or processor is explicitly shown.

[0052] As used herein, the term "storage medium" may refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, kernel memory, magnetic disk storage media, optical storage media, flash memory devices and / or other machine-readable media for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing and / or containing instructions and / or data.

[0053] A code segment may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program descriptions. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or storage contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including storage sharing, information passing, token passing, network transmission, etc.

[0054] The "computer device" referred to in this context refers to an electronic device that can perform predetermined processing procedures such as numerical calculations and / or logical calculations by running predetermined programs or instructions, and may include at least a processor and a memory, wherein the processor executes program instructions pre-stored in the memory to perform the predetermined processing process, or the predetermined processing process is performed by hardware such as ASIC, FPGA, DSP, or a combination of the above two.

[0055] The above-mentioned "computer device" is usually expressed in the form of a general-purpose computer device, and its components may include but are not limited to: one or more processors or processing units, system memory. The system memory may include a computer-readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The "computer device" may further include other removable / non-removable, volatile / non-volatile computer-readable storage media. The memory may include at least one computer program product having a set (e.g., at least one) of program modules that are configured to perform the functions and / or methods of the various embodiments of the present disclosure. The processor executes various functional applications and data processing by running the program stored in the memory.

[0056] For example, a computer program for executing various functions and processes of multiple embodiments of the present disclosure is stored in the memory, and when the processor executes the corresponding computer program, multiple embodiments of the present disclosure are implemented.

[0057] Typically, the computer device may be, for example, a user device or a network device, or even a combination of the two. The user device includes, but is not limited to, a personal computer (PC), a laptop, a mobile terminal, etc. The mobile terminal includes, but is not limited to, a smart phone, a tablet computer, etc. The network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of computers or network servers based on cloud computing (Cloud Computing), wherein cloud computing is a type of distributed computing, a super virtual computer consisting of a group of loosely coupled computer sets. The computer device may be operated alone to implement the various embodiments of the present disclosure, or may be connected to a network and implement the various embodiments of the present disclosure through interactive operations with other computer devices in the network. The network in which the computer device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, etc.

[0058] It should be noted that the user equipment, network equipment and network are only examples. Other existing or future computing devices or networks that are applicable to the embodiments of the present disclosure should also be included in the scope of protection of the present disclosure and are included here by reference.

[0059] The specific structural and functional details disclosed herein are merely representative and are for the purpose of describing exemplary embodiments of the present disclosure. However, the various embodiments of the present disclosure may be implemented in many alternative forms and should not be interpreted as being limited to only the embodiments set forth herein.

[0060] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0061] The terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments. Unless the context clearly indicates otherwise, the singular forms "one", "one" and "item" used herein are also intended to include plural numbers. It should also be understood that the terms "include" and / or "comprise" used herein specify the existence of stated features, integers, steps, operations, units and / or components, without excluding the existence or addition of one or more other features, integers, steps, operations, units, components and / or combinations thereof.

[0062] It should also be noted that, in some alternative implementations, the functions / actions mentioned may occur in a different order than that indicated in the accompanying drawings. For example, depending on the functions / actions involved, two figures shown in succession may actually be performed substantially simultaneously or may sometimes be performed in the reverse order.

[0063] See also Figure 1 , which shows a model training process according to an embodiment of the present disclosure, which is used to train a 2D convolutional neural network for user action recognition.

[0064] The action recognition model adopted in the present invention is a 2D convolutional neural network (CNN). The 2D convolutional neural network trained with the action image sample set can be used for user action recognition in the present invention.

[0065] See also Figure 1 In step S10, the computer device obtains a sample set of action images, wherein the sample image sequence includes a standard action image sequence and a non-standard action image sequence, wherein each of the sample image sequences includes a plurality of continuous images and is annotated with a corresponding action classification label, and wherein each sample image is annotated with a human body key point; in step S20, the computer device inputs the sample image sequence into a 2D convolutional neural network to train it for action recognition.

[0066] Specifically, step S10 is a sample construction stage, in which an action image sample set for model training is constructed.

[0067] The 2D convolutional neural network trained with the action image sample set can be considered as an action recognition model, which is used to identify the action classification of user actions.

[0068] According to one embodiment, the action image sample set in the present disclosure can directly obtain an existing action image sample set, as long as it includes a standard action image sequence and a non-standard action image sequence, each action image sequence is annotated with a corresponding action classification label, and each action image is annotated with human body key points.

[0069] According to another embodiment, each embodiment of the present disclosure directly constructs an action image sample set to be used for training a 2D convolutional neural network to perform an action recognition task.

[0070] Specifically, the computer device obtains videos shot according to standard actions and non-standard actions, marks the human body key points, action classification labels and product pipelines corresponding to the standard action image sequences, and stores them in corresponding feature templates according to the product pipelines to which each standard action image sequence belongs.

[0071] Furthermore, non-standard actions can be simply labeled as "non-standard actions", or they can be labeled in detail like standard actions. For example, the human body key points, action classification labels, error types and product pipelines corresponding to the non-standard action image sequences are labeled, and the non-standard action image sequences are stored in the corresponding feature templates according to the product pipelines to which they belong.

[0072] In one example, firstly, a video of a standard action and a non-standard action is taken, and a plurality of continuous image frames are extracted from each video at a specific frequency. For example, a certain frame rate (for example, 30 frames are extracted per second) can be set to ensure the continuity of the action. A continuous sequence of image frames corresponds to an action. For human key points, the key point positions (for example, head, shoulder, elbow, wrist, knee, ankle, etc.) are defined. The specific positions of the human key points are marked on each frame of the image, and key point coordinate data are generated. Here, for example, annotation tools (such as LabelMe, VIA, etc.) can be used to mark each image frame.

[0073] According to the actions shown in the image sequence, each image sequence is assigned an action classification label. For example, a standard action can be labeled as "standard forward bend" and a non-standard action can be labeled as "wrong forward bend".

[0074] Determine the product pipeline to which the action belongs based on the body part involved. For example, actions for the lumbar spine can be labeled as "lumbar pipeline", actions for the cervical spine can be labeled as "cervical pipeline", etc. Assign one or more relevant product pipeline tags to each action image sequence to indicate the product pipeline field to which these actions belong. The features extracted from the canonical action image sequence of each product pipeline are stored as feature templates. These feature templates are used for subsequent model training and action recognition.

[0075] In another example, taking rehabilitation training as an example, during the data collection process, different categories of standard and non-standard movements should be filmed and recorded according to different rehabilitation training product pipelines. Rehabilitation movement video clips should include common movements in rehabilitation training, such as bending, stretching, rotation, etc. Each video clip should capture movements shot from different angles and perspectives to increase data diversity.

[0076] Each product pipeline corresponds to one or more specific groups of actions. Common product pipelines include lumbar spine, cervical spine, spine, pelvis, etc. Standard actions corresponding to each pipeline, such as bending in lumbar training, raising head in cervical training, and lateral bending in spine training. Record the specific execution details of each action, such as angle, speed, number of repetitions, etc. Common non-standard actions corresponding to each pipeline, such as incorrect bending posture in lumbar training, insufficient raising head in cervical training, etc. Record the specific details of each wrong action, such as error type, angle deviation, speed change, etc. You can also mark the different stages of the action, such as: start, in progress, end, etc., to help the system understand the time series characteristics of the action. Mark the angle changes of key joints, which is very important for accurately evaluating the correctness and quality of the action. Mark the amplitude range of the action, such as the height of the arm raising, the angle of the leg bending, etc. Mark the execution speed of the action, slow, medium, fast, etc., to help the system identify different action speeds.

[0077] Assign a category label to each standard action image sequence. For example, "Lumbar spine - standard bending", "Cervical spine - standard head raising". Assign a category label to each non-standard action image sequence. For example, "Lumbar spine - insufficient non-standard bending", "Cervical spine - insufficient non-standard head raising". Human posture estimation tools (such as OpenPose, MediaPipe, etc.) can be used to annotate the key points of the human body in each frame of the image, such as the head, shoulders, elbows, wrists, hips, knees, ankles, etc. A specific error type description can also be given for each non-standard action. For example, for the lumbar bending action, it is labeled "insufficient bending angle" or "arms are not straight".

[0078] By collecting and labeling the standard and non-standard actions of different product pipelines in detail, the action recognition model can learn more detailed action features, thereby providing more accurate and effective action recognition in rehabilitation training.

[0079] In addition, data enhancement techniques such as rotation, scaling, and mirroring can be used to increase the number of sample images and improve the generalization ability of the model.

[0080] Step S20 is a model training phase, in which a 2D convolutional neural network is trained for action recognition.

[0081] According to an example of the present disclosure, the training process of a 2D convolutional neural network includes the following steps:

[0082] 1) Perform multi-channel feature extraction on the input sample image sequence to obtain the feature map corresponding to each channel, where the multi-channel includes I frame channel, MV channel and RGBR channel;

[0083] 2) The obtained feature maps are spliced ​​and fused, and the key points of the human body are recognized in the sample image sequence by combining the feature importance of each feature and the attention mechanism of the key points;

[0084] 3) Combine the temporal characteristics of the identified human key points to perform action recognition on the sample image sequence;

[0085] 4) Calculate the loss function based on the action recognition results and the action classification labels annotated by the corresponding sample image sequence;

[0086] 5) Repeat the above steps 1)-4) until the loss function converges, and the training of the 2D convolutional neural network is completed.

[0087] Specifically, in step 1), multiple 2D residual networks (ResNet18) are used to extract features of corresponding channels of the input sample image sequence to obtain feature maps corresponding to each channel.

[0088] In one embodiment, three ResNet18s extract feature maps for the sample image sequence in the I frame channel, MV channel and RGBR channel respectively.

[0089] I-frame channel (Intra-coded frame): I-frame contains complete image information, similar to static images, can provide detailed spatial information, and is independent of other frames.

[0090] MV channel (Motion Vector): Motion vector (MV) describes the motion information between video frames, capturing the motion mode and direction.

[0091] RGBR channel (RGB residual frame): The RGB residual frame represents the changes between frames and provides more detailed information, especially small changes and motion details.

[0092] ResNet18 is an 18-layer residual neural network, consisting of 8 residual blocks using 2D convolution in 4 stages. The network structure is relatively shallow. In order to use the convolution layer to fully extract features, the 8 2D residual blocks can all use 3*3 convolution, that is, the convolution kernel parameter is 3*3. Among them, the convolution layer generally refers to a network layer used to complete the weighted summation of local pixel values ​​and nonlinear activation. In order to reduce the number of channels of the convolution layer operation and thus reduce the number of parameters, each 2D residual block can use the bottleneck design concept, that is, each residual block is composed of 3 convolution layers (the convolution kernel parameters are 1*1, 3*3 and 1*1 respectively), and the first layer and the last layer are used to compress and restore the image channel respectively.

[0093] Since I frame, MV and RGBR cannot be directly concatenated together in terms of data features, three ResNet18s can be used to process I frame, MV and RGBR separately to obtain feature maps of the same dimension, which can be recorded as I frame feature map, MV feature map and RGBR feature map respectively. MV feature map and RGBR feature map are collectively referred to as P frame feature map. Among them, the same dimension can refer to the consistency of C*H*W, where C represents channel, H represents height, and W represents width, and * can also be represented as ×. Therefore, the dimension of the I frame feature map is C1×H×W, the dimension of the MV feature map is C2×H×W, and the dimension of the RGBR feature map is C3×H×W. After processing, feature maps of the same dimension can be spliced ​​through the first splicing layer, for example, splicing in the channel C dimension.

[0094] Furthermore, in some embodiments, the image sequence needs to be normalized before being input into ResNet18.

[0095] Specifically, the normalization process uses the mean-variance operation to remove the average brightness of the image, making the algorithm less interested in the image brightness and more focused on its content, and removes common parts to highlight individual differences.

[0096] Assume x 1 ~x n is the pixel value data in the original image, then the pixel average is

[0097] The feature normalization stage uses the following mean-variance formula (1) to perform feature normalization:

[0098] D(X)=E[X 2 ]-EX] 2 Formula (1)

[0099] Among them, E[X] represents the expected pixel value (i.e., the average value), E[X 2 ] represents the expectation of the square of the pixel value.

[0100]

[0101]

[0102] Normalization by difference

[0103]

[0104] The original RGB pixel value range of the image can be converted from (0 to 255) to (0 to 1). Figure 2As shown, the values ​​of the original RGB values ​​after normalization are all in the range of (0 to 1).

[0105] In some embodiments, in addition to normalization, other preprocessing operations may be performed before inputting the image sequence into the multi-channel feature map for extraction, including filtering and noise reduction, eliminating artifacts, image stretching, image rotation, masking, etc., so as to improve the diversity of sample data so as to extract features from a "cleaner" image.

[0106] In the prior art, only one frame of image is usually extracted at a specific frequency as input for feature extraction during the initial image feature processing. The embodiment of the present disclosure extracts multiple consecutive frames of image when extracting at a specific frequency, and then performs subsequent processing on the multiple frames of image after performing mean variance calculation. This can remove some common parts to highlight the differences, so that the subsequent feature extraction can better highlight the characteristics of the image and facilitate subsequent recognition.

[0107] Next, in step 2), the obtained feature maps are spliced ​​and fused, and the key points of the human body are recognized in the sample image sequence by combining the feature importance of each feature and the attention mechanism of the key points.

[0108] The embodiment of the present disclosure uses multiple channels to extract features from the sample image sequence respectively, and then uses the extraction results of the multiple channels to comprehensively evaluate the final aggregated feature data set, and finally extracts the most prominent feature information.

[0109] Among them, the I frame feature map with dimensions of C1×H×W, the MV feature map with dimensions of C2×H×W, and the RGBR feature map with dimensions of C3×H×W are concatenated in the channel dimension to form a new feature map with dimensions of (C1+C2+C3)×H×W. The concatenated feature map is convolved to further extract fusion features. The convolution operation combines features from different channels to generate a more comprehensive feature representation.

[0110] According to the overall feature map after splicing and fusion, calculate the feature importance value E of the corresponding feature (key point) ij :

[0111] E ij =ReLU <C r |W s |H t > Formula (5)

[0112] Among them, C r represents the channel, W s Indicates width, H t Indicates height;

[0113] ReLU (Rectified Linear Unit) is an activation function that is used to turn negative values ​​into zero, thereby retaining positive values, which helps to increase nonlinear features and highlight important features.

[0114] Figure 3 The feature channels shown correspond to the three dimensions of CWH in formula (5). The importance value of each feature is calculated by the feature importance formula (5).

[0115] Further, the relative attention value between the key point and its neighboring key points is calculated according to the feature importance. In some examples, the relative attention value A between the key point i and any key point j adjacent to it in the spatial dimension is ij :

[0116]

[0117] Among them, N i is the set of neighboring key points of key point i, k is the index of any key point in the set Ni, E ik is the attention value of key point i and any key point k in the set Ni.

[0118] exist Figure 3 After calculating the importance values ​​of each feature, all the features and feature importance results are given to the softmax function to calculate the attention value between each key point and its adjacent key points.

[0119] Next, based on the calculated feature importance values, key point attention values, and spatial attention matrix, it is finally determined which key point features should be output.

[0120]

[0121] Wherein, P is the convolution kernel parameter of the spatial dimension. In the embodiment of the present disclosure, P is 3, M p is the relative attention value A ij The spatial attention matrix, X in is the fused feature map, W p is the feature weight.

[0122] After calculation of the above formula (7), the feature information F is finally extracted out The key points in each image can be identified. These key point features can be represented as a set of two-dimensional coordinates or a high-dimensional feature vector obtained through further processing.

[0123] According to the calculation of the above formula (4)-formula (7), that is, Func1(x1, x2...xn)-》Func2(c,h,w)-》Func3(E ij)-》Func4(A ij )-》Result, the initial data input is the pixel data value of multiple frames of the image. After the mean square error, it is input into formula (5) to process the output of the relu function of each channel respectively, and the feature importance value of each channel is calculated. Then, the feature importance value is used to calculate the attention value through formula (6), and finally, the final output result, that is, the key point feature, is calculated through formula (7).

[0124] The next step 3) will further combine the temporal characteristics of the identified human key points to perform action recognition on the sample image sequence.

[0125] Here, the extracted key point features are first input into RNN (Recurrent Neural Network) or LSTM (Long Short-Term Memory) for time series modeling.

[0126] Arrange the key point feature vectors in the sample image sequence containing continuous video frames in chronological order to form a time series. Assume that each sample image sequence contains T frames, and the key point feature vector extracted from each frame is x t (where t is the index of the frame, t∈[1,T]), then the input sequence is: [x 1 ,x 2 ,...,x T ].

[0127] Use RNN or LSTM to model the input sequence. RNN or LSTM can capture the temporal dependencies and contextual information between video frames.

[0128] In some examples, a sequence of keypoint features is input into the LSTM, which models these features in time series and outputs the hidden state at each moment. Finally, the LSTM outputs the hidden state at the last moment, which captures the dynamic information in the entire sample image sequence.

[0129] Subsequently, the hidden state at the last moment is input into the fully connected layer of the 2D convolutional neural network for action classification. The fully connected layer generates the probability distribution of each action category through an activation function, such as Softmax, and selects the category with the highest probability as the final action recognition result.

[0130] In some embodiments, ResNet18 can also be used to perform convolution operations on each frame in the sample image sequence to extract high-dimensional feature vectors. These feature vectors represent the static spatial information of each frame of the image.

[0131] The hidden state of the last moment of the LSTM output is concatenated with the convolutional features of each frame. The concatenated feature vector contains a comprehensive feature vector of temporal features and spatial features. The fully connected layer linearly transforms these comprehensive feature vectors to generate the probability distribution of each action category, and selects the category with the highest probability as the final action recognition result.

[0132] In addition to using conventional image processing techniques, the embodiments of the present invention enrich the data granularity of image feature extraction by splitting multiple channels and adding time series frames, so that the identification of human key points is more accurate and reliable based on richer feature extraction, and better motion recognition accuracy is achieved, thereby expanding the scope of application of intelligent interaction scenarios.

[0133] The above steps 1)-3) are the forward propagation of the model training process, which is used to predict the action classification. The next step 4) will calculate the loss function and update the network parameters based on the action recognition results and the action classification labels annotated by the corresponding sample image sequence.

[0134] In some examples, the cross entropy loss function is used to measure the difference between the model prediction and the true label. The loss function formula is as follows:

[0135]

[0136] Among them, y i is the one-hot encoding representation of the true label, is the probability predicted by the model.

[0137] Subsequently, the gradient is calculated by the back-propagation algorithm, and the network parameters of the 2D convolutional neural network are updated using optimization algorithms, such as Adam (Adaptive Moment Estimation) and SGD (Stochastic Gradient Descent).

[0138] Repeat the above steps 1)-4) until the loss function converges and the training of the 2D convolutional neural network is completed. When the next update cycle or other update conditions are reached, the 2D convolutional neural network will update the training according to the new sample set.

[0139] In some embodiments, a group of convolutional neural networks can be constructed first, which includes N convolutional neural networks, and the structure of each convolutional neural network includes: a convolution layer, a pooling layer, and a fully connected layer. Design multiple groups of structural parameters of deep convolutional neural networks, including layer weights and layer parameters. Determine the sample data set and the test data set: use the sample data set to learn and train multiple groups of deep convolutional neural networks to obtain multiple groups of learning and training results; calculate multiple groups of error values ​​based on the multiple groups of learning and training results; when the multiple groups of error values ​​are greater than or equal to the set error threshold value, use the back propagation feedback method to optimize the structural parameters to obtain multiple groups of qualified target deep convolutional neural networks; use the test sample to test each target deep convolutional neural network to obtain multiple accuracy test values; rank the multiple accuracy test values ​​from high to low, and select the top 50% of the target deep convolutional neural networks in the ranking results for the next round of learning and training, until only one target deep convolutional neural network is left, which is used as the deep convolutional neural network model of the present disclosure.

[0140] New training data is used to train the weights and parameters of the layers in the deep convolutional neural network model whose recognition ability is to be enhanced, and the weights and parameters of other layers except the layers whose recognition ability is to be enhanced are frozen.

[0141] Designing the structure of the deep convolutional neural network model may include determining the input layer structure of the deep convolutional neural network model and determining the hidden layer structure of the deep convolutional neural network model. The input layer structure includes a convolutional layer, and the length of the input layer is a preset length of the sample image sequence. The hidden layer structure includes a convolutional layer, a pooling layer, a convolutional layer, a pooling layer, a fully connected layer, a neuron discarding layer, a fully connected layer, and a neuron discarding layer arranged in sequence.

[0142] Among them, the convolution layer of the input layer of the deep convolutional neural network model includes an activation function and 32 convolution neurons, and the convolution width is 8; each convolution layer of the hidden layer of the deep convolutional neural network model includes an activation function and 32 convolution neurons, the convolution width is 8, and the pooling coefficient of the pooling layer is 8; each fully connected layer of the hidden layer of the deep convolutional neural network model includes 80 neurons and an activation function, and the discard probability of each neuron discard layer is 0.2.

[0143] Each time the next round of learning and training is carried out, it includes: adjusting the ratio of the training data set and the test data set in the training database to reduce the number of training samples and increase the number of test samples.

[0144] Furthermore, the proportions of the training data set and the test data set in the training database can be adjusted respectively. Specifically, determine whether the proportion of the training data set in the database is greater than the proportion of the test data set in the training database; if so, double the amount of data in the test data set as a new test set for the next round of learning and training, and use the ratio value obtained by subtracting 1 from the proportion of the training data set in the entire data set as the proportion of the new training data set in the training database for the next round of learning and training; otherwise, double the amount of data in the training data set as a new training data set for the next round of learning and training, and use the ratio value obtained by subtracting 1 from the proportion of the test data set in the entire data set as the proportion of the new test data set in the training database for the next round of learning and training.

[0145] The data in the training database includes different lead data and / or data obtained by mixing data from different databases.

[0146] After the model is trained, it can be used to identify user actions, which is also called model prediction.

[0147] According to some embodiments of the present disclosure, a computer device obtains a plurality of continuous video frames of a user action; and the computer device recognizes the action classification of the user action through a trained 2D convolutional neural network based on the obtained plurality of continuous video frames.

[0148] In some embodiments, a computer device collects a video of a user's movements, such as a video of a user's movements during rehabilitation training, and selects a plurality of continuous frames, such as the first 30 frames, to perform motion analysis to capture the temporal dynamic information of the movements. In some examples, these continuous video frames may also be processed by normalization, noise reduction, and the like. Subsequently, these continuous frames are input into a trained motion recognition model to obtain the corresponding motion classification. In some examples, the output motion classification may also include error types, such as "cervical vertebra - non-standard head raise is not enough". In some examples, when the output motion classification is "cervical vertebra - non-standard head raise", correction suggestions may also be provided, such as "head raise 45 degrees".

[0149] It should be noted that the various embodiments of the present disclosure may be implemented in software and / or a combination of software and hardware, for example, may be implemented using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the various embodiments of the present disclosure may be executed by a processor to implement the steps or functions described above. Similarly, the software program of the various embodiments of the present disclosure (including related data structures) may be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive, or a floppy disk and the like. In addition, some steps or functions of the various embodiments of the present disclosure may be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0150] In addition, at least a portion of the embodiments of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computing device, can call or provide the method and / or technical solution according to the embodiments of the present disclosure through the operation of the computing device. The program instruction for calling / providing the method of the embodiments of the present disclosure may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computing device that runs according to the program instruction.

[0151] For those skilled in the art, it is obvious that the embodiments of the present disclosure are not limited to the details of the above exemplary embodiments, and the embodiments of the present disclosure can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present disclosure. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present disclosure is limited by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the embodiments of the present disclosure. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A method for training a 2D convolutional neural network for user action recognition, wherein: The method comprises the following steps: Acquire an action image sample set, wherein the sample image sequence includes a standard action image sequence and a non-standard action image sequence, wherein each of the sample image sequences includes a plurality of continuous images and is annotated with corresponding action classification labels, and wherein each sample image is annotated with human body key points; The sample image sequence is input into a 2D convolutional neural network to train it for action recognition, wherein: Performing multi-channel feature extraction on the sample image sequence to obtain a feature map corresponding to each channel, wherein the multi-channel includes an I frame channel, an MV channel, and an RGBR channel; The obtained feature maps are spliced ​​and fused, and the key points of the human body are recognized for the sample image sequence by combining the feature importance of each feature and the attention mechanism of the key points; The splicing and fusion specifically includes: The feature maps extracted from the I frame channel, MV channel, and RGBR channel are spliced ​​in the channel dimension and a convolution operation is performed on the spliced ​​feature maps to obtain a fused feature map; The feature importance of each feature is calculated based on the following method: HAVE BEEN ij =ReLU <C r |W s |H t > Among them, E ij represents the feature importance of key point i relative to any adjacent key point j, C r represents the channel, W s Indicates width, H t Indicates height; The attention mechanism of the key points is calculated based on the following method: Among them, A ij is the relative attention value between key point i and any adjacent key point j, N i is the set of neighboring key points of key point i, E ik is the attention value of key point i and any key point k in the set Ni; Combining the temporal characteristics of the identified human body key points, performing action classification and recognition on the sample image sequence; Calculating a loss function based on the recognition result of the action classification and the action classification label annotated by the corresponding sample image sequence; The above steps are repeated until the loss function converges, and the training of the 2D convolutional neural network is completed.

2. The method according to claim 1, wherein: The steps of constructing the action image sample set include: Video recording based on standard and non-standard movements; Label the human body key points, action classification labels and product pipelines corresponding to the standardized action image sequences; According to the product pipeline to which each standard action image sequence belongs, it is stored in the corresponding feature template.

3. The method according to claim 2, wherein: Also includes: Label the human body key points, action classification labels, error types and product pipelines corresponding to the non-standard action image sequences; Each non-standard action image sequence is stored in a corresponding feature template according to the product pipeline to which it belongs.

4. The method according to claim 1, wherein: Before performing multi-channel feature extraction on the sample image sequence, the sample image sequence is also normalized.

5. The method according to claim 1, wherein: The multi-channel feature extraction specifically includes: The sample image sequence is input into three ResNet18 models to extract feature maps of the I frame channel, MV channel and RGBR channel respectively.

6. The method according to claim 1, wherein: The method further includes: Acquire multiple consecutive video frames of the user's action; According to the multiple continuous video frames, the action classification of the user action is identified by the trained 2D convolutional neural network.

7. The method according to claim 6, wherein: The plurality of continuous video frames are extracted from the acquired user action video at a certain frequency.

8. A computer program product comprising computer program instructions, wherein: When the computer program instructions are executed by a computer device, the computer device is configured to perform the method according to any one of claims 1 to 7.

9. A computer program storage medium storing computer executable instructions, wherein when the computer executable instructions are executed by a computer device, the computer device is configured to perform the method according to any one of claims 1 to 7.

Images