A motion recognition method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a method, device, electronic device and storage medium for action recognition, the method comprising: obtaining multiple frames of continuous infrared images; the infrared images are images of a designated part of a target object taken by an infrared camera, the target object is deployed with multiple light-capturing balls, each light-capturing ball corresponds to a designated part of the target object; from the multiple frames of continuous infrared images, determining a target image group containing multiple frames of target images; inputting the multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group. The method provided by the embodiment of the present invention simplifies the processing of action recognition of the target object, reduces the requirements for the use scenario of action recognition, and at the same time, can achieve a higher action recognition accuracy in a use scenario with lower requirements.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an action recognition method, device, electronic equipment and storage medium. Background Art

[0002] Currently, there are many technologies for recognizing human actions, such as recognizing the actions of target objects in video images only through image processing technology, or recognizing the actions of target objects through light capture technology.

[0003] At present, the method of identifying the target object's action through light capture technology is mainly used in film and television production. The recognition process needs to be completed in a professional studio, such as Figure 1 As shown:

[0004] A plurality of infrared cameras 103 are installed at different positions in the studio, and the actors wear special light-capturing clothes 101, on which a plurality of light-capturing balls 102 with strong reflective ability are deployed. During the shooting process, the infrared camera 103 emits infrared light and receives the infrared light reflected by the light-capturing balls 102, and shoots infrared video images in different directions. After obtaining the infrared video images in different directions, the spatial position of the light-capturing balls 102 is calculated through image processing techniques such as image fusion, and then the actions of the actors are identified. However, this method is not only costly but also requires the processing of multiple video images, the algorithm is complex, and the requirements for the use scenario are high.

[0005] The recognition of the action of the target object in the video screen through image processing technology is mainly based on the image processing technology, which recognizes the action of the target object contained in the visible light image, such as RGB (red, green, blue) image. However, since the visible light image is greatly affected by the environment and the image quality is unstable, it is easy to affect the accuracy of action recognition. For example, the action recognition of the target object contained in the strongly exposed visible light image will reduce the accuracy of the recognition result due to the low image quality. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a motion recognition method, device, electronic device and storage medium, so as to improve the accuracy of motion recognition while simplifying the processing of motion recognition.

[0007] To achieve the above object, an embodiment of the present invention provides a method for motion recognition, comprising:

[0008] Acquire multiple frames of continuous infrared images; the infrared images are images taken by an infrared camera and include a designated part of a target object, wherein the target object is deployed with multiple light-capturing balls, wherein each light-capturing ball corresponds to a designated part of the target object;

[0009] Determine a target image group including multiple frames of target images from multiple frames of continuous infrared images;

[0010] Input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images containing specified parts of the sample object.

[0011] Furthermore, determining a target image group including multiple frames of target images from multiple frames of continuous infrared images includes:

[0012] One frame of infrared image is selected from a plurality of continuous frames of infrared images at intervals of a preset number of frames as a target image, thereby obtaining a target image group consisting of the plurality of frames of target images.

[0013] Furthermore, the pre-trained action recognition model includes: a feature extraction network layer, a difference feature calculation layer, a feature splicing layer, an action classification network layer and an output layer;

[0014] The step of inputting a plurality of target image frames in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group includes:

[0015] Inputting multiple frames of target images in the target image group into a feature extraction network layer of a pre-trained action recognition model;

[0016] The feature extraction network layer extracts the light capture features of multiple frames of target images respectively to obtain multiple light capture feature information;

[0017] The difference feature calculation layer calculates the difference of the light capture features of two adjacent target images in the target image group according to the light capture feature information to obtain a plurality of difference feature information;

[0018] The feature stitching layer stitches the plurality of light capture feature information and the plurality of difference feature information to obtain stitching feature information;

[0019] The action classification network layer determines the probability that the action corresponding to the splicing feature information belongs to each preset action type;

[0020] The output layer outputs the action type with the highest probability as the action type of the target object corresponding to the target image group.

[0021] Furthermore, the feature extraction network layer is:

[0022] Visual Geometry Group Network VGG, or, Residual Neural Network ResNet, or lightweight deep neural network MobileNet.

[0023] Furthermore, the action classification network layer in the action recognition model includes: a preset number of fully connected layers; wherein the input feature dimension of the first fully connected layer of the action classification network layer is s×(2N-1); the output feature dimension of the last fully connected layer of the action classification network layer is 1×n; N represents the number of target images, and n represents the number of action types;

[0024] The output layer in the action recognition model includes: a softmax layer.

[0025] Furthermore, the action recognition model is obtained by training based on the training sample set using the following steps:

[0026] Inputting multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained, and obtaining the action type of the sample object corresponding to the sample image group as an output result;

[0027] Based on the output result, adjusting the parameters of the current neural network model to be trained to obtain a new neural network model to be trained, completing one iteration, and returning to the step of inputting the multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained;

[0028] When the number of iterations reaches a preset number of iterations, or if the loss function value of the current neural network model to be trained is less than a preset loss function threshold, the training is terminated and the current neural network model to be trained is determined as an action recognition model.

[0029] Furthermore, the feature extraction network layer of the neural network model to be trained is a predetermined image feature extraction network layer;

[0030] The adjusting the parameters of the current neural network model to be trained based on the output result includes:

[0031] Based on the output result, the parameters of the action classification network layer of the current neural network model to be trained are adjusted.

[0032] Furthermore, the action types of the target object include: kicking, raising hands, running, walking, pushing, pulling, jumping and meaningless actions.

[0033] In order to achieve the above object, an embodiment of the present invention further provides a motion recognition device, comprising:

[0034] An infrared image acquisition module is used to acquire multiple frames of continuous infrared images; the infrared image is an image taken by an infrared camera containing a designated part of a target object, and the target object is deployed with multiple light capture balls, wherein each light capture ball corresponds to a designated part of the target object;

[0035] An image group determination module is used to determine a target image group including multiple frames of target images from multiple frames of continuous infrared images;

[0036] An action recognition module is used to input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images containing specified parts of the sample object.

[0037] Furthermore, the image group determination module is specifically used to select one infrared image frame from multiple continuous infrared images at intervals of a preset number of frames as the target image, thereby obtaining a target image group consisting of multiple target images.

[0038] Furthermore, the pre-trained action recognition model includes: a feature extraction network layer, a difference feature calculation layer, a feature splicing layer, an action classification network layer and an output layer;

[0039] The action recognition module is specifically used to input multiple frames of target images in the target image group into the feature extraction network layer of a pre-trained action recognition model; the feature extraction network layer extracts light capture features of the multiple frames of target images respectively to obtain multiple light capture feature information; the difference feature calculation layer calculates the difference in light capture features of two adjacent target images in the target image group according to the light capture feature information to obtain multiple difference feature information; the feature splicing layer splices multiple light capture feature information and multiple difference feature information to obtain spliced ​​feature information; the action classification network layer determines the probability that the action corresponding to the spliced ​​feature information belongs to each preset action type; the output layer outputs the action type with the largest probability as the action type of the target object corresponding to the target image group.

[0040] Furthermore, the feature extraction network layer is:

[0041] Visual Geometry Group Network VGG, or, Residual Neural Network ResNet, or lightweight deep neural network MobileNet.

[0042] Furthermore, the action classification network layer in the action recognition model includes: a preset number of fully connected layers; wherein the input feature dimension of the first fully connected layer of the action classification network layer is s×(2N-1); the output feature dimension of the last fully connected layer of the action classification network layer is 1×n; N represents the number of target images, n represents the number of action types, and s represents the dimension of light capture feature information;

[0043] The output layer in the action recognition model includes: a softmax layer.

[0044] Furthermore, the device further includes: a model training module;

[0045] The model training module is used to obtain the action recognition model by training based on the training sample set using the following steps:

[0046] Inputting multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained, and obtaining the action type of the sample object corresponding to the sample image group as an output result;

[0047] Based on the output result, adjusting the parameters of the current neural network model to be trained to obtain a new neural network model to be trained, completing one iteration, and returning to the step of inputting the multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained;

[0048] When the number of iterations reaches a preset number of iterations, or if the loss function value of the current neural network model to be trained is less than a preset loss function threshold, the training is terminated and the current neural network model to be trained is determined as an action recognition model.

[0049] Furthermore, the feature extraction network layer of the neural network model to be trained is a predetermined image feature extraction network layer;

[0050] The model training module adjusts the parameters of the action classification network layer of the current neural network model to be trained based on the output result.

[0051] Furthermore, the action types of the target object include: kicking, raising hands, running, walking, pushing, pulling, jumping and meaningless actions.

[0052] In order to achieve the above object, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0053] Memory, used to store computer programs;

[0054] The processor is used to implement any of the above-mentioned action recognition method steps when executing the program stored in the memory.

[0055] In order to achieve the above-mentioned purpose, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned action recognition method steps is implemented.

[0056] In order to achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned steps of the action recognition method.

[0057] Beneficial effects of the embodiments of the present invention:

[0058] By adopting the method provided by the embodiment of the present invention, by deploying multiple light-capturing balls on the target object, it is only necessary to use one infrared camera to shoot multiple frames of continuous infrared images of the target object, obtain a target image group containing multiple frames of target images, and then use a pre-trained action recognition model to identify the action of the target object in the multiple frames of target images, and determine the action type of the target object. Therefore, compared with the existing action recognition method, the method provided by the embodiment of the present invention simplifies the processing of action recognition of the target object and reduces the requirements of the use scenario of action recognition, that is, the target object does not need to wear specific light-capturing clothes, and does not require a high-demand use scenario. It is only necessary to stick multiple light-capturing balls on the target object, collect images of the target object through an infrared camera, and then process the collected images to realize the action recognition of the target object. At the same time, because the method provided by the embodiment of the present invention uses light capture technology, it can achieve a high action recognition accuracy in such a low-demand use scenario.

[0059] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0061] Figure 1 It is a schematic diagram of professional light capture data collection in the prior art;

[0062] Figure 2 A flow chart of a motion recognition method provided by an embodiment of the present invention;

[0063] Figure 3 Another flow chart of the action recognition method provided by the embodiment of the present invention;

[0064] Figure 4a A schematic diagram of a target object with a light capture ball deployed in the action recognition method provided by an embodiment of the present invention;

[0065] Figure 4b A schematic diagram of a target object with a light capture ball deployed thereon and an infrared image collected for the target object with the light capture ball deployed thereon provided by an embodiment of the present invention;

[0066] Figure 5a A schematic diagram of the structure of an action recognition model provided by an embodiment of the present invention;

[0067] Figure 5b A schematic diagram of a target image processed by an action recognition model provided by an embodiment of the present invention;

[0068] Figure 5c A schematic diagram of the structure of the action classification network layer in the action recognition model provided by an embodiment of the present invention;

[0069] Figure 6 A flowchart of a training action recognition model provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram of the structure of a motion recognition device provided by an embodiment of the present invention;

[0071] Figure 8 Another structural schematic diagram of a motion recognition device provided by an embodiment of the present invention;

[0072] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0074] Since the existing motion recognition method has complex algorithms and is difficult to apply to other fields, in order to simplify the processing of motion recognition and expand the application scenarios of motion recognition, the embodiment of the present invention provides a motion recognition method, such as Figure 2 As shown, including:

[0075] Step 201, obtain multiple frames of continuous infrared images; the infrared image is an image taken by an infrared camera containing a designated part of a target object, and the target object is deployed with multiple light capture balls, wherein each light capture ball corresponds to a designated part of the target object. The target object may be: a person, an animal, etc. The light capture ball deployed for the target object may be a light capture ball.

[0076] Step 202: determining a target image group including multiple frames of target images from multiple frames of continuous infrared images.

[0077] Step 203, input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images containing specified parts of the sample object.

[0078] By adopting the method provided by the embodiment of the present invention, by deploying multiple light-capturing balls on the target object, it is only necessary to use one infrared camera to shoot multiple frames of continuous infrared images of the target object, obtain a target image group containing multiple frames of target images, and then use a pre-trained action recognition model to identify the action of the target object in the multiple frames of target images, and determine the action type of the target object. Therefore, compared with the existing action recognition method, the method provided by the embodiment of the present invention simplifies the processing of action recognition of the target object and reduces the requirements of the use scenario of action recognition, that is, the target object does not need to wear specific light-capturing clothes, and does not require a high-demand use scenario. It is only necessary to stick multiple light-capturing balls on the target object, collect images of the target object through an infrared camera, and then process the collected images to realize the action recognition of the target object. At the same time, because the method provided by the embodiment of the present invention uses light capture technology, it can achieve a high action recognition accuracy in such a low-demand use scenario.

[0079] The action recognition method and device provided by the embodiments of the present invention are described in detail below through specific embodiments.

[0080] In one embodiment of the present application, Figure 3 As shown, another process of the action recognition method includes the following steps:

[0081] Step 301, acquiring multiple frames of continuous infrared images.

[0082] In this step, multiple light capture balls can be deployed at designated parts of the target object, and each light capture ball corresponds to a designated part of the target object. The light capture ball can be a reflective light capture ball, and the target object can be a person or an animal, etc. If the target object is a person, the designated part of the target object can include: wrists, elbows, ankles, knees, feet, shoulders, etc. Multiple light capture balls can be attached to multiple designated parts of the target object. For details, please refer to Figure 4a , the target object 401's wrists, elbows, ankles, knees, feet, shoulders and other designated parts are deployed with light capture balls 402. Specifically, refer to Figure 4a , the light catching ball 402 can be attached to each designated part of the target object 401 in the order of the arrows. For example, a light catching ball 402 can be attached to the pelvis of the target object 401, starting from the pelvis, and then attached to the designated part of the target object 401 in the order of the arrows. Figure 4a In the direction of the middle arrow, a light capture ball 402 is pasted on each of the following parts: "spine 1", "spine 2", "spine 3", "neck", "head", "left clavicle", "left shoulder", "left elbow", "left wrist", "left hand", "right clavicle", "right shoulder", "right elbow", "right wrist", "right hand" and so on; starting from the pelvis, follow the steps of Figure 4a In the direction of the middle arrow, a light capture ball 402 is pasted on each of the "left hip", "left knee", "left ankle", "left foot", "right hip", "right knee", "right ankle", "right foot" and other parts downwards. Finally, 24 light capture balls 402 can be pasted on the above 24 body parts of the target object 401.

[0083] An infrared camera, such as a Kineck DK camera, can be used to shoot a target object on which multiple light-catching balls are deployed, and obtain multiple frames of continuous images containing a specified part of the target object. Figure 4b As shown, light capture balls 402 are deployed on the left wrist, right wrist, left elbow and right elbow of the target object 403 , and an infrared camera can be used to capture an infrared image 410 of the target object 403 deployed with the light capture balls.

[0084] Step 302 , selecting one infrared image frame at a preset frame interval from a plurality of continuous infrared image frames as a target image, and obtaining a target image group consisting of a plurality of target image frames.

[0085] The preset number of frames can be set according to actual applications, for example, the preset number of frames can be set to 2, 3 or 4, etc.

[0086] For example, if 10 consecutive infrared images are acquired and the preset number of frames is 2, the first infrared image can be determined as the target image, and one infrared image can be determined as the target image every 2 infrared images, and finally the first, third, fifth, seventh and ninth frames of the 10 consecutive infrared images are determined as the target images. Furthermore, the determined five target images can be regarded as a target image group.

[0087] In the embodiment of the present invention, all of the multiple frames of continuous infrared images captured by the infrared camera may be used as the target image.

[0088] Step 303: input the multiple frames of target images in the target image group into a feature extraction network layer of a pre-trained action recognition model.

[0089] In the embodiment of the present invention, refer to Figure 5a ,The pre-trained action recognition model includes: feature extraction network layer, ,differential feature calculation layer, feature concatenation layer, action classification network layer and ,output layer.

[0090] Among them, the feature extraction network layer can be: VGG (Visual Geometry Group Network), or ResNet (Residual Neural Network), or MobileNet (Lightweight Deep Neural Network).

[0091] Step 304, a feature extraction network layer extracts light capture features of multiple frames of target images respectively to obtain multiple light capture feature information.

[0092] In the embodiment of the present invention, after inputting multiple target images in the target image group into the feature extraction network layer of the pre-trained action recognition model, the feature extraction network layer can be used to extract the features of each target image as light capture feature information. Moreover, the extracted light capture feature information is a multi-dimensional vector.

[0093] For example, if the target image group contains 5 frames of target images, and the 5 frames of target images are respectively the following: the t-2th frame of infrared image, the t-1th frame of infrared image, the tth frame of infrared image, the t+1th frame of infrared image, and the t+2th frame of infrared image. The w×h×1-dimensional pixel matrix of each frame of target image can be input into the feature extraction network layer, and the light capture features of the 5 frames of target images are extracted as follows: I t-2 ,I t-1 ,I t ,I t+1 and I t+2 . And, the extracted light capture features are all preset s-dimensional vectors, s≠0. Wherein, w is the number of horizontal pixels in the pixel matrix of the target image, and w is the number of vertical pixels in the pixel matrix of the target image.

[0094] Step 305, the difference feature calculation layer calculates the difference between the light capture features of two adjacent target images in the target image group according to the light capture feature information to obtain a plurality of difference feature information.

[0095] In this step, if the number of light capture features extracted is N, N-1 difference feature information can be calculated.

[0096] For example, refer to Figure 5b , if the target image group contains 5 frames of target images, the 5 frames of target images are respectively the following: the t-2th frame of infrared image, the t-1th frame of infrared image, the tth frame of infrared image, the tth frame of infrared image, the t+1th frame of infrared image and the t+2th frame of infrared image in the multiple frames of continuous infrared images collected by the infrared camera, and the light capture feature information of the 5 frames of target images extracted by the feature extraction network layer are respectively: I t-2 ,I t-1 ,I t ,I t+1 and I t+2 , the difference feature calculation layer can calculate the difference feature information according to the extracted light capture feature information:

[0097] M t =I t -I t-1 ;

[0098] M t-1 =I t-1 -I t-2 ;

[0099] M t+1 =I t+1 -I t ;

[0100] M t+2 =I t+2 -I t+1 ;

[0101] Based on the five light capture feature information, four difference feature information can be calculated: M t 、M t-1 、M t+1 and M t+2 Among them, the difference feature information is also an s-dimensional vector.

[0102] Step 306, feature stitching layer, stitching multiple light capture feature information and multiple difference feature information to obtain stitching feature information.

[0103] In this step, multiple light capture feature information and multiple difference feature information can be spliced ​​into one spliced ​​feature information. If there are N light capture feature information and N-1 difference feature information, and both the light capture feature information and the difference feature information are s-dimensional vectors, then s×(2N-1)-dimensional spliced ​​feature information can be obtained.

[0104] For example, refer to Figure 5b , if there are 5 light capture feature information: I t-2 ,I t-1 ,I t ,I t+1 and I t+2 , 4 difference feature information: M t 、M t-1 、M t+1 and M t+2 , where the light capture feature information and the difference feature information are both s-dimensional vectors. The 5 light capture feature information and the 4 difference feature information can be superimposed to obtain s×9-dimensional splicing feature information.

[0105] Step 307, the action classification network layer determines the probability that the action corresponding to the spliced ​​feature information belongs to each preset action type.

[0106] In an embodiment of the present invention, the action classification network layer in the action recognition model may include a preset number of fully connected layers. Wherein, under the premise of ensuring that the dimension of the output of the last fully connected layer of the action classification network layer is 1×n, the preset number can be specifically determined according to the actual application situation, for example, the action classification network layer can be set to 6 fully connected layers or 10 fully connected layers, etc. n represents the number of action types of the target object.

[0107] For example, refer to Figure 5c , if the action classification network layer in the action recognition model includes 6 fully connected layers:

[0108] First fully connected layer: the input feature dimension is s×(2N-1), the number of neurons is (2N-1)s, and the output feature dimension is 1×(2N-1)s; N represents the number of target images; N represents the number of target images and also represents the number of light capture feature information;

[0109] Second fully connected layer: the input feature dimension is 1×(2N-1)s, the number of neurons is 2(2N-1)s, and the output feature dimension is 1×2(2N-1)s;

[0110] The third fully connected layer: the input feature dimension is 1×2(2N-1)s, the number of neurons is 2(2N-1)s, and the output feature dimension is 1×2n, where n represents the number of action types;

[0111] Fourth fully connected layer: the input feature dimension is 1×2n, the number of neurons is 2n, and the output feature dimension is 1×2n;

[0112] The fifth fully connected layer: the input feature dimension is 1×2n, the number of neurons is 2n, and the output feature dimension is 1×n;

[0113] The sixth fully connected layer: the input feature dimension is n, the number of neurons is n, and the output feature dimension is 1×n.

[0114] The feature dimension of the sixth fully connected layer output is 1×n, which represents the probability value corresponding to each of the n preset action types of the target object. Figure 5b , the features of the sixth fully connected layer output are: [p1, p2, ..., p n-1 , p n ], where p1, p2, …, p n-1 and p n are the probability values ​​corresponding to each of the n preset action types of the target object, and p1+p2+…+p n-1 +p n =1.

[0115] Step 308: the output layer outputs the action type with the highest probability as the action type of the target object corresponding to the target image group.

[0116] The output layer in the action recognition model includes: softmax layer. The softmax layer has an input feature dimension of 1×n and an output feature dimension of 1×n. The input of the softmax layer is: the output of the last fully connected layer of the action classification network layer in the action recognition model, that is, the probability value corresponding to each of the n preset action types of the target object; for example, refer to Figure 5b , the features of the sixth fully connected layer output are: [p1, p2, ..., p n-1 , p n ].

[0117] Based on the probability value corresponding to each preset action type among the n preset action types of the input target object, the softmax layer can generate a classification vector that retains the maximum probability value and sets other probability values ​​to 0.

[0118] For example, the input features of the softmax layer are: [p1, p2, ..., p n-1 , p n ], where p1, p2, …, p n-1 and p nare the probability values ​​corresponding to each of the n preset action types of the target object. If p1 is p1, p2, …, p n-1 and p n The softmax layer can generate a classification vector [1, 0, …, 0, 0] with the maximum probability value retained as 1 and other probability values ​​set to 0.

[0119] Based on the classification vector output by the softmax layer, the preset action type corresponding to the classification vector can be used as the action type of the target object corresponding to the target image group.

[0120] The preset action type corresponding to the classification vector is the preset action type of the target object corresponding to the maximum probability value. For example, the preset action type of the target object corresponding to the classification vector [1, 0, ..., 0, 0] is the preset action type of the target object corresponding to the maximum probability value p1.

[0121] Reference Figure 5b , among the n preset action types of the target object, the probability value corresponding to each preset action type is: [p1, p2, …, p n-1 , p n ], the output layer can output the preset action type with the largest probability value among the probability values ​​of each preset action type as the action type of the identified target object, that is, max{p1,p2,...,p n}The corresponding preset action type is output as the action type of the identified target object.

[0122] In the embodiment of the present invention, the preset action types of the target object include: kicking, raising hands, running, walking, pushing, pulling, jumping and meaningless actions, etc.

[0123] For example, if the target object has 8 types of actions: kicking, raising hands, running, walking, pushing, pulling, jumping, and meaningless actions, then n = 8; if the features output by the softmax layer are: [0.1, 0.1, 0.01, 0.03, 0.04, 0.05, 0.6, 0.06]; among them, the probability corresponding to the kicking action is 0.1, the probability corresponding to the hand raising action is 0.1, the probability corresponding to the running action is 0.01, the probability corresponding to the walking action is 0.03, the probability corresponding to the pushing action is 0.04, the probability corresponding to the pulling action is 0.05, the probability corresponding to the jumping action is 0.6, and the probability corresponding to the meaningless action is 0.04. The output layer can output the action type with the highest probability: jumping action, as the action type of the target object corresponding to the target image group.

[0124] According to the method provided by the embodiment of the present invention, by deploying multiple light capture balls on the target object, only one infrared camera is needed to shoot multiple frames of continuous infrared images for the target object, and a target image group containing multiple frames of target images is obtained. Then, through the pre-trained action recognition model, light capture feature information is extracted from the multiple frames of target images in the target image group respectively, and difference feature information is calculated based on the extracted light capture feature information, and then the light capture feature information and the difference feature information are spliced ​​to obtain spliced ​​feature information. Then, the probability that the action corresponding to the spliced ​​feature information belongs to each preset action type is determined, and the action type with the highest probability is determined to be the action type of the target object. Compared with the existing action recognition method, the method provided by the embodiment of the present invention extracts the light capture feature information of multiple frames of continuous target images and combines the difference feature information to identify the action. On the one hand, it simplifies the processing of action recognition of the target object and reduces the requirements of the use scene of action recognition. On the other hand, it also improves the accuracy of identifying the action of the target object. In addition, there is no need to be in a professional studio, and the target object does not need to wear specific light capture clothes. It only needs to stick multiple light capture balls on the target object, and the action recognition of the target object can be realized by an infrared camera, which expands the use scene of action recognition.

[0125] In the embodiment of the present invention, see Figure 6 , a process for training an action recognition model, including:

[0126] Step 601: Input multiple frames of sample images of a sample image group in a training sample set into a neural network model to be trained, and obtain the action type of the sample object corresponding to the sample image group as an output result.

[0127] The training sample set includes a plurality of sample image groups. The multiple frames of sample images in each sample image group are infrared images collected by an infrared camera, and a preset number of frames are spaced between each two adjacent frames of sample images in each sample image group.

[0128] The neural network model to be trained includes: a feature extraction network layer, a difference feature calculation layer, a feature concatenation layer, an action classification network layer and an output layer. Among them, the feature extraction network layer can use a predetermined image feature extraction network layer, such as a VGG network, or a ResNet network, or a MobileNet network.

[0129] In this step, after the sample images are input into the neural network model to be trained, the feature extraction network layer can extract the light capture feature information of each sample image; the difference feature calculation layer calculates the difference in light capture feature information of two adjacent frames of sample images in the same sample image group based on the light capture feature information of the extracted sample images to obtain the difference feature information; the feature splicing layer splices the light capture feature information and the difference feature information of the extracted sample images to obtain the splicing feature information of the sample image group; the action classification network layer determines the probability that the action corresponding to the splicing feature information of the sample image group belongs to each preset action type; the output layer outputs the action type with the largest probability as the action type of the sample object corresponding to the sample image group, as the output result.

[0130] Step 602, based on the output result, adjust the parameters of the current neural network model to be trained to obtain a new neural network model to be trained, complete one iteration, and return to the step of inputting multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained.

[0131] In the embodiment of the present invention, the parameters of the action classification network layer of the current neural network model to be trained are adjusted based on the output result.

[0132] Step 603, when the number of iterations reaches a preset number of iterations, or if the loss function value of the current neural network model to be trained is less than a preset loss function threshold, the training is terminated and the current neural network model to be trained is determined as an action recognition model.

[0133] The preset number of iterations and the preset loss function threshold can both be set according to the actual training situation, wherein the setting of the preset number of iterations satisfies: after a preset number of iterations, the current neural network model to be trained converges; the setting of the preset loss function threshold satisfies: if the loss function value of the current neural network model to be trained is less than the preset loss function threshold, the current neural network model to be trained converges.

[0134] Based on the same inventive concept, according to the motion recognition method provided by the above embodiment of the present invention, another embodiment of the present invention also provides a motion recognition device, whose structural schematic diagram is shown in FIG. Figure 7 As shown, specifically including:

[0135] The infrared image acquisition module 701 is used to acquire multiple frames of continuous infrared images; the infrared image is an image taken by an infrared camera containing a specified part of the target object, and the target object is deployed with multiple light capture balls, wherein each light capture ball corresponds to a specified part of the target object.

[0136] The image group determination module 702 is used to determine a target image group including multiple frames of target images from multiple frames of continuous infrared images.

[0137] The action recognition module 703 is used to input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target person object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images containing specified parts of the sample object.

[0138] It can be seen that, by using the action recognition device provided by the embodiment of the present invention, by deploying multiple light capture balls on the target object, it is only necessary for one infrared camera to shoot multiple frames of continuous infrared images for the target object, obtain a target image group containing multiple frames of target images, and then identify the action of the target object in the multiple frames of target images through a pre-trained action recognition model to determine the action type of the target object. Therefore, compared with the existing action recognition method, the method provided by the embodiment of the present invention simplifies the processing of action recognition of the target object and reduces the requirements of the use scenario of action recognition, that is, the target object does not need to wear specific light capture clothes, and does not need a high-demand use scenario. It only needs to stick multiple light capture balls on the target object, collect images of the target object through an infrared camera, and then process the collected images to realize the action recognition of the target object. At the same time, because the method provided by the embodiment of the present invention uses light capture technology, it can achieve a high action recognition accuracy in such a low-demand use scenario.

[0139] Furthermore, the image group determination module 702 is specifically configured to select one infrared image frame from a plurality of continuous infrared images at intervals of a preset number of frames as a target image, thereby obtaining a target image group consisting of a plurality of target images.

[0140] Furthermore, the pre-trained action recognition model includes: a feature extraction network layer, a difference feature calculation layer, a feature concatenation layer, an action classification network layer and an output layer;

[0141] The action recognition module 703 is specifically used to input multiple frames of target images in the target image group into the feature extraction network layer of the pre-trained action recognition model; extract the light capture features of the multiple frames of target images respectively to obtain multiple light capture feature information, and the light capture feature information is an s-dimensional vector; the difference feature calculation layer calculates the difference in light capture features of two adjacent target images in the target image group according to the light capture feature information to obtain multiple difference feature information; the feature splicing layer splices the multiple light capture feature information and the multiple difference feature information to obtain spliced ​​feature information; the action classification network layer determines the probability that the action corresponding to the spliced ​​feature information belongs to each preset action type; the output layer outputs the action type with the largest probability as the action type of the target object corresponding to the target image group.

[0142] Furthermore, the feature extraction network layer is: VGG, or,ResNet, or,MobileNet.

[0143] Furthermore, the action classification network layer in the action recognition model includes: a preset number of fully connected layers; wherein the input feature dimension of the first fully connected layer of the action classification network layer is s×(2N-1); the output feature dimension of the last fully connected layer of the action classification network layer is 1×n; N represents the number of target images;

[0144] The output layer in the action recognition model includes: softmax layer.

[0145] Further, the action classification network layer in the action recognition model includes: a first fully connected layer to a sixth fully connected layer;

[0146] First fully connected layer: the input feature dimension is s×(2N-1), the number of neurons is (2N-1)s, and the output feature dimension is 1×(2N-1)s;

[0147] Second fully connected layer: the input feature dimension is 1×(2N-1)s, the number of neurons is 2(2N-1)s, and the output feature dimension is 1×2(2N-1)s;

[0148] The third fully connected layer: the input feature dimension is 1×2(2N-1)s, the number of neurons is 2(2N-1)s, and the output feature dimension is 1×2n, where n represents the number of action types;

[0149] Fourth fully connected layer: the input feature dimension is 1×2n, the number of neurons is 2n, and the output feature dimension is 1×2n;

[0150] The fifth fully connected layer: the input feature dimension is 1×2n, the number of neurons is 2n, and the output feature dimension is 1×n;

[0151] The sixth fully connected layer: the input feature dimension is n, the number of neurons is n, and the output feature dimension is 1×n;

[0152] The output layer in the action recognition model includes: a softmax layer;

[0153] The softmax layer has an input feature dimension of 1×n and an output feature dimension of 1×n.

[0154] For further information, see Figure 8 , the action recognition device also includes: a model training module 801;

[0155] The model training module 801 is used to train the action recognition model based on the training sample set by adopting the following steps:

[0156] Inputting multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained, and obtaining the action type of the sample object corresponding to the sample image group as an output result;

[0157] Based on the output result, the parameters of the current neural network model to be trained are adjusted to obtain a new neural network model to be trained, one iteration is completed, and the step of inputting the multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained is returned;

[0158] When the number of iterations reaches a preset number of iterations, or if the loss function value of the current neural network model to be trained is less than a preset loss function threshold, the training is terminated and the current neural network model to be trained is determined as an action recognition model.

[0159] Furthermore, the feature extraction network layer of the neural network model to be trained is a predetermined image feature extraction network layer;

[0160] The model training module 801 adjusts the parameters of the action classification network layer of the current neural network model to be trained based on the output results.

[0161] Furthermore, the target object's action types include: kicking, raising hands, running, walking, pushing, pulling, jumping, and meaningless actions.

[0162] It can be seen that, by using the device provided by the embodiment of the present invention, by deploying multiple light capture balls on the target object, only one infrared camera is needed to shoot multiple frames of continuous infrared images for the target object, obtain a target image group containing multiple frames of target images, and then extract light capture feature information from the multiple frames of target images in the target image group through a pre-trained action recognition model, calculate the difference feature information based on the extracted light capture feature information, and then splice the light capture feature information and the difference feature information to obtain spliced ​​feature information. Then determine the probability that the action corresponding to the spliced ​​feature information belongs to each preset action type, and determine the action type with the highest probability as the action type of the target object. Compared with the existing action recognition method, the method provided by the embodiment of the present invention extracts the light capture feature information of multiple frames of continuous target images and combines the difference feature information to identify the action. On the one hand, it simplifies the processing of action recognition of the target object and reduces the requirements of the use scenario of action recognition. On the other hand, it also improves the accuracy of identifying the action of the target object. Moreover, there is no need for a professional studio, and the target object does not need to wear specific light-capturing clothing. It is only necessary to stick multiple light-capturing balls on the target object and use an infrared camera to realize motion recognition of the target object, which expands the application scenarios of motion recognition.

[0163] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0164] Memory 903, used for storing computer programs;

[0165] The processor 901 is used to execute the program stored in the memory 903 to implement the following steps:

[0166] Acquire multiple frames of continuous infrared images; the infrared images are images taken by an infrared camera and include a designated part of a target object, wherein the target object is deployed with multiple light-capturing balls, wherein each light-capturing ball corresponds to a designated part of the target object;

[0167] Determine a target image group including multiple frames of target images from multiple frames of continuous infrared images;

[0168] Input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images containing specified parts of the sample object.

[0169] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0170] The communication interface is used for communication between the above electronic device and other devices.

[0171] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0172] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0173] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned action recognition methods are implemented.

[0174] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any action recognition method in the above embodiments.

[0175] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0176] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0177] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for motion recognition, characterized in that: include: Acquire multiple frames of continuous infrared images; The infrared image is an image taken by an infrared camera and includes a designated part of a target object, wherein a plurality of light-capturing balls are deployed on the target object, wherein each light-capturing ball corresponds to a designated part of the target object; Determine a target image group including multiple frames of target images from multiple frames of continuous infrared images; Input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images of specified parts of the sample object taken by an infrared camera.

2. The method according to claim 1, characterized in that The step of determining a target image group including multiple frames of target images from multiple frames of continuous infrared images includes: One frame of infrared image is selected from a plurality of continuous frames of infrared images at intervals of a preset number of frames as a target image, thereby obtaining a target image group consisting of the plurality of frames of target images.

3. The method according to claim 1, characterized in that The pre-trained action recognition model includes: a feature extraction network layer, a difference feature calculation layer, a feature splicing layer, an action classification network layer and an output layer; The step of inputting a plurality of target image frames in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group includes: Inputting multiple frames of target images in the target image group into a feature extraction network layer of a pre-trained action recognition model; The feature extraction network layer extracts the light capture features of multiple frames of target images respectively to obtain multiple light capture feature information; The difference feature calculation layer calculates the difference of the light capture features of two adjacent target images in the target image group according to the light capture feature information to obtain a plurality of difference feature information; The feature stitching layer stitches the plurality of light capture feature information and the plurality of difference feature information to obtain stitching feature information; The action classification network layer determines the probability that the action corresponding to the splicing feature information belongs to each preset action type; The output layer outputs the action type with the highest probability as the action type of the target object corresponding to the target image group.

4. The method according to claim 3, characterized in that The action classification network layer in the action recognition model includes: a preset number of fully connected layers; wherein the input feature dimension of the first fully connected layer of the action classification network layer is s×(2N-1); the output feature dimension of the last fully connected layer of the action classification network layer is 1×n; N represents the number of target images, n represents the number of action types, and s represents the dimension of light capture feature information; The output layer in the action recognition model includes: a softmax layer.

5. The method according to claim 3, characterized in that: The action recognition model is obtained by training based on the training sample set using the following steps: Inputting multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained, and obtaining the action type of the sample object corresponding to the sample image group as an output result; Based on the output result, adjusting the parameters of the current neural network model to be trained to obtain a new neural network model to be trained, completing one iteration, and returning to the step of inputting the multiple frames of sample images of the sample image group in the training sample set into the neural network model to be trained; When the number of iterations reaches a preset number of iterations, or if the loss function value of the current neural network model to be trained is less than a preset loss function threshold, the training is terminated and the current neural network model to be trained is determined as an action recognition model.

6. The method according to claim 5, characterized in that The feature extraction network layer of the neural network model to be trained is a predetermined image feature extraction network layer; The adjusting the parameters of the current neural network model to be trained based on the output result includes: Based on the output result, the parameters of the action classification network layer of the current neural network model to be trained are adjusted.

7. A motion recognition device, characterized in that: include: An infrared image acquisition module, used for acquiring multiple frames of continuous infrared images; The infrared image is an image taken by an infrared camera and includes a designated part of a target object, wherein a plurality of light-capturing balls are deployed on the target object, wherein each light-capturing ball corresponds to a designated part of the target object; An image group determination module is used to determine a target image group including multiple frames of target images from multiple frames of continuous infrared images; An action recognition module is used to input multiple frames of target images in the target image group into a pre-trained action recognition model to obtain the action type of the target object corresponding to the target image group; wherein the action recognition model is trained based on a training sample set, and the training sample set includes: multiple sample image groups, the action type of the sample object corresponding to each sample image group, each sample image group includes multiple frames of sample images, and the sample images in the sample image group are images of specified parts of the sample object taken by an infrared camera.

8. The device according to claim 7, characterized in that The image group determination module is specifically used to select one infrared image frame from multiple continuous infrared images at intervals of a preset number of frames as a target image, thereby obtaining a target image group consisting of multiple target images.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images