Behavior and posture recognition method for occluded parts of the human body

Abstract

The present invention discloses a method, medium and device for identifying the behavior and posture of an occluded part of a human body, wherein the method comprises: obtaining a video to be detected and extracting a video frame of the video to be detected; obtaining a current video frame according to the order of the video frames, and judging whether the current video frame is the first frame of the video to be detected; if not, inputting the current video frame into a feature extraction main network to extract the first frame feature corresponding to the current video frame through the feature extraction main network; obtaining the second frame feature of the previous frame video frame, and performing feature fusion on the first frame feature corresponding to the current video frame and the second frame feature of the previous frame video frame; inputting the feature fusion result into a feature extraction refinement network to extract the corresponding refined feature, and using the refined feature as the second frame feature of the current video frame; the method can prevent the loss of key points of the human body due to large human body movement amplitude and occlusion, and improve the accuracy of human body key point recognition.

Description

Technical Field

[0001] The present invention relates to the field of machine recognition technology, and in particular to a method for recognizing the behavior and posture of an obscured part of a human body, a computer-readable storage medium, and a computer device. Background Art

[0002] In related technologies, when extracting key points from a person, most simply input an image into a detection network to extract the key points corresponding to the person in the image. However, it is understandable that when a person is in motion, their movements are large and easily obscured. Therefore, this recognition method often results in missing key points in the motion state, leading to inaccurate recognition results. Summary of the Invention

[0003] The present invention aims to at least partially address one of the technical problems in the aforementioned technologies. To this end, one object of the present invention is to provide a method for identifying human body gestures from occluded areas, which can prevent the loss of key points on the human body due to large movements and occlusions, thereby improving the accuracy of key point recognition.

[0004] A second object of the present invention is to provide a computer-readable storage medium.

[0005] A third object of the present invention is to provide a computer device.

[0006] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a method for behavioral posture recognition of occluded parts of the human body, comprising the following steps: obtaining a video to be detected and extracting video frames of the video to be detected; obtaining a current video frame according to the order of the video frames, and judging whether the current video frame is the first frame of the video to be detected; if not, inputting the current video frame into the feature extraction main network to extract the first frame features corresponding to the current video frame through the feature extraction main network; obtaining the second frame features of the previous frame video frame, and performing feature fusion on the first frame features corresponding to the current video frame and the second frame features of the previous frame video frame; inputting the feature fusion results into the feature extraction refinement network to extract corresponding refined features, and using the refined features as the second frame features of the current video frame.

[0007] According to the behavioral posture recognition method of the occluded part of the human body according to the embodiment of the present invention, first, a video to be detected is obtained, and the video frame of the video to be detected is extracted; then, the current video frame is obtained according to the order of the video frames, and it is determined whether the current video frame is the first frame of the video to be detected; then, if not, the current video frame is input into the feature extraction main network to extract the first frame feature corresponding to the current video frame through the feature extraction main network; then, the second frame feature of the previous frame video frame is obtained, and the first frame feature corresponding to the current video frame and the second frame feature of the previous frame video frame are feature fused; then, the feature fusion result is input into the feature extraction refinement network to extract the corresponding refined features, and the refined features are used as the second frame features of the current video frame; thereby preventing the loss of human key points due to large human body movement and occlusion, and improving the accuracy of human key point recognition.

[0008] In addition, the method for identifying the behavior and posture of the occluded part of the human body according to the above embodiment of the present invention may also have the following additional technical features:

[0009] Optionally, if the current video frame is the first frame of the video to be detected, the current video frame is input into the feature extraction main network to output the first frame feature corresponding to the current video frame through the feature extraction main network, and the first frame feature is input into the feature extraction refinement network, and the refined feature output by the feature extraction refinement network is used as the second frame feature of the current video frame.

[0010] Optionally, the first frame feature corresponding to the current video frame and the second frame feature of the previous video frame are feature fused, including: obtaining the influence coefficient of the previous video frame, and adjusting the second frame feature of the previous video frame according to the influence coefficient to obtain the feature to be fused; adjusting the size of the feature to be fused according to the first convolution so that the size of the feature to be fused is equal to the size of the first frame feature of the current video frame; amplifying the number of feature map channels of the feature to be fused according to the second convolution so that the number of feature map channels of the feature to be fused is equal to the number of feature map channels of the first frame feature of the current video frame; adding the amplified feature to be fused and the feature map of the first frame feature of the current video frame on the same dimension to complete the feature fusion of the first frame feature corresponding to the current video frame and the second frame feature of the previous video frame.

[0011] Optionally, the training process of the feature extraction main network and the feature extraction refinement network includes: obtaining a first data set and training based on the first data set to obtain a pre-trained model; obtaining a second data set and continuing to train the pre-trained model based on the second data set to obtain a final detection model.

[0012] Optionally, the first data set is a COCO data set, wherein training is performed based on the first data set to obtain a pre-trained model, including: jittering and amplifying the COCO data set to obtain a pre-trained data set; during the training process of the pre-trained model, the original image in the COCO data set is input as the current video frame, and the image in the pre-trained data set is input as the previous video frame; wherein jittering and amplifying the COCO data set includes: randomly translating randomly given key points of any image in the COCO data set; randomly rotating all key points of any image in the COCO data set; and scaling the size of the key points of any image in the COCO data set.

[0013] Optionally, the second dataset is a PoseTrack dataset.

[0014] To achieve the above-mentioned objectives, the second aspect of the present invention proposes a computer-readable storage medium, which stores a behavior posture recognition program for an obscured part of a human body, so that when the behavior posture recognition program for an obscured part of a human body is executed by a processor, the behavior posture recognition method for an obscured part of a human body as described above is implemented.

[0015] According to an embodiment of the present invention, a computer-readable storage medium stores a program for recognizing the behavior and posture of an obscured human body part, so that when the program is executed by a processor, the method for recognizing the behavior and posture of an obscured human body part as described above is implemented. This prevents the loss of key points of the human body due to large human body movement and occlusion, and improves the accuracy of key point recognition.

[0016] To achieve the above-mentioned objectives, the third aspect of the present invention proposes a computer device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the program, it implements the above-mentioned method for behavioral posture recognition of obscured parts of the human body.

[0017] According to an embodiment of the present invention, a computer device stores a program for recognizing the behavior and posture of an obscured part of a human body in a memory, so that when the program is executed by a processor, the method for recognizing the behavior and posture of an obscured part of a human body as described above is implemented. This prevents the loss of key points of a human body due to large human body movements and occlusion, and improves the accuracy of key point recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 2. A flow chart of a method for identifying a behavior posture of an occluded part of a human body according to an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of the COCO dataset augmentation process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0021] In the related art, when extracting key points of the human body, in a moving state, this recognition method often results in the loss of key points of the human body, resulting in inaccurate final recognition results. According to the behavioral posture recognition method of the occluded part of the human body of an embodiment of the present invention, first, a video to be detected is obtained, and the video frame of the video to be detected is extracted; then, the current video frame is obtained according to the order of the video frames, and it is determined whether the current video frame is the first frame of the video to be detected; then, if not, the current video frame is input into the feature extraction main network to extract the first frame feature corresponding to the current video frame through the feature extraction main network; then, the second frame feature of the previous frame is obtained, and the first frame feature and the second frame feature are feature fused; then, the feature fusion result is input into the feature extraction refinement network to extract the corresponding refined feature, and the refined feature is used as the second frame feature of the current video frame; thereby preventing the loss of key points of the human body due to large human motion amplitude and occlusion, and improving the accuracy of human key point recognition.

[0022] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0023] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0024] Figure 1 FIG. 1 is a flow chart of a method for identifying a behavior posture of an obscured part of a human body according to an embodiment of the present invention. Figure 1 As shown, the method for identifying the behavior and posture of the occluded part of the human body includes the following steps:

[0025] S101: Obtain a video to be detected and extract video frames of the video to be detected.

[0026] S102: Obtain a current video frame according to the order of the video frames, and determine whether the current video frame is the first frame of the video to be detected.

[0027] S103: If not, the current video frame is input into the feature extraction main network to extract the first frame feature corresponding to the current video frame through the feature extraction main network.

[0028] In some embodiments, if the current video frame is the first frame of the video to be detected, the current video frame is input into the feature extraction main network to output the first frame features corresponding to the current video frame through the feature extraction main network, and the first frame features are input into the feature extraction refinement network, and the refined features output by the feature extraction refinement network are used as the second frame features of the current video frame.

[0029] That is to say, the video frames of the video to be detected are extracted from the video to be detected, and these video frames are sorted according to the time axis. When performing human key point detection, the current video frame must be obtained first; if the current video frame is the first frame, the current video frame is input into the feature extraction main network to extract the corresponding first frame features (i.e., overall features); then, the first frame features are input into the feature extraction refinement network to extract the corresponding refined features, which are used as the second frame features (i.e., local features) of the current video frame.

[0030] S104: Acquire the second frame feature of the previous video frame, and perform feature fusion on the first frame feature corresponding to the current video frame and the second frame feature of the previous video frame.

[0031] In some embodiments, feature fusion is performed on the first frame feature corresponding to the current video frame and the second frame feature of the previous video frame, including: obtaining the influence coefficient of the previous video frame, and adjusting the second frame feature of the previous video frame according to the influence coefficient to obtain the feature to be fused; adjusting the size of the feature to be fused according to the first convolution so that the size of the feature to be fused is equal to the size of the first frame feature of the current video frame; amplifying the number of feature map channels of the feature to be fused according to the second convolution so that the number of feature map channels of the feature to be fused is equal to the number of feature map channels of the first frame feature of the current video frame; adding the amplified feature to be fused and the feature map of the first frame feature of the current video frame on the same dimension to complete the feature fusion of the first frame feature corresponding to the current video frame and the second frame feature of the previous video frame.

[0032] As an example, Figure 2 As shown in the figure, when obtaining the key points of the human body of the current video frame, the key points may be lost due to the human body being in motion. Therefore, by performing feature fusion on the first frame features of the current video frame and the second frame features of the previous video frame, the key point detection capability of the current video frame is enhanced by supplementing the key point information of the previous frame. Preferably, the influence coefficient a of the previous video frame is first obtained (for example, the value of a is 0.1). The influence coefficient is used to determine the degree of influence of the second frame features of the previous video frame on the fused features. Then, the second frame features output by the previous video frame are synthesized into a feature map by superposition, and the feature map is multiplied by the influence coefficient to obtain the feature to be fused; then, the size of the feature to be fused is adjusted to the same size as the first frame feature of the current video frame using 3*3 convolution; then, the number of channels of the feature to be fused is expanded to the same number of channels of the feature map of the first frame feature of the current video frame using 1*1 convolution. Then, the feature maps of the same dimension are added to complete the feature fusion.

[0033] S105: Input the feature fusion result into the feature extraction and refinement network to extract corresponding refined features, and use the refined features as the second frame features of the current video frame.

[0034] There are many ways to train the feature extraction main network and the feature extraction refinement network.

[0035] In some embodiments, the training process of the feature extraction main network and the feature extraction refinement network includes: obtaining a first data set and training based on the first data set to obtain a pre-trained model; obtaining a second data set and continuing to train the pre-trained model based on the second data set to obtain a final detection model.

[0036] In some embodiments, the first data set is a COCO data set, wherein training is performed based on the first data set to obtain a pre-trained model, including: jittering and amplifying the COCO data set to obtain the pre-trained data set; during the training process of the pre-trained model, the original image in the COCO data set is input as the current video frame, and the image in the pre-trained data set is input as the previous video frame; wherein jittering and amplifying the COCO data set includes: randomly translating the randomly given key points of any image in the COCO data set; randomly rotating all key points of any image in the COCO data set; and scaling the size of the key points of any image in the COCO data set.

[0037] As an example, Figure 2 As shown in the figure, first, the COCO dataset is obtained. Then, for a single image in the COCO dataset, a random number k of key points to be changed are randomly assigned, and these k key points are randomly slightly translated (for example, for a 46*46 feature map, the translation is within 5 pixels). All key points in the entire image are randomly slightly rotated, and the width of all key points is enlarged. Thus, during the model training process, the adjusted feature map can be used as the input of the previous video frame of the current frame, and the pre-trained model is obtained after training.

[0038] In some embodiments, the second dataset is a PoseTrack dataset.

[0039] As an example, in order to increase the diversity of samples, first, a pre-trained model is trained on the expanded COCO dataset; then, the pre-trained model is used to continue training on the PoseTrack dataset to obtain the final detection model.

[0040] In summary, according to the behavioral posture recognition method of the occluded part of the human body according to an embodiment of the present invention, first, a video to be detected is obtained, and the video frame of the video to be detected is extracted; then, the current video frame is obtained according to the order of the video frames, and it is determined whether the current video frame is the first frame of the video to be detected; then, if not, the current video frame is input into the feature extraction main network to extract the first frame feature corresponding to the current video frame through the feature extraction main network; then, the second frame feature of the previous frame video frame is obtained, and the first frame feature corresponding to the current video frame and the second frame feature of the previous frame video frame are feature fused; then, the feature fusion result is input into the feature extraction refinement network to extract the corresponding refined features, and the refined features are used as the second frame features of the current video frame; thereby preventing the loss of human key points due to large human body movement and occlusion, and improving the accuracy of human body key point recognition.

[0041] In order to implement the above-mentioned embodiments, the second aspect of the present invention proposes a computer-readable storage medium, which stores a behavior posture recognition program for an obscured part of a human body, so that when the behavior posture recognition program for an obscured part of a human body is executed by a processor, the behavior posture recognition method for an obscured part of a human body as described above is implemented.

[0042] According to an embodiment of the present invention, a computer-readable storage medium stores a program for recognizing the behavior and posture of an obscured human body part, so that when the program is executed by a processor, the method for recognizing the behavior and posture of an obscured human body part as described above is implemented. This prevents the loss of key points of the human body due to large human body movement and occlusion, and improves the accuracy of key point recognition.

[0043] In order to implement the above-mentioned embodiments, the third aspect of the present invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the program, it implements the above-mentioned method for behavioral posture recognition of obscured parts of the human body.

[0044] According to an embodiment of the present invention, a computer device stores a program for recognizing the behavior and posture of an obscured part of a human body in a memory, so that when the program is executed by a processor, the method for recognizing the behavior and posture of an obscured part of a human body as described above is implemented. This prevents the loss of key points of a human body due to large human body movements and occlusion, and improves the accuracy of key point recognition.

[0045] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0047] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0048] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0049] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0050] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0051] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0052] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0053] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0054] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediary. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or diagonally above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0055] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0056] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for identifying the behavior and posture of an occluded part of a human body, characterized in that: The following steps are involved: Obtaining a video to be detected and extracting a video frame of the video to be detected; Obtaining a current video frame according to the order of the video frames, and determining whether the current video frame is the first frame of the video to be detected; If not, inputting the current video frame into the feature extraction main network to extract the first frame feature corresponding to the current video frame through the feature extraction main network; Obtain the second frame features of the previous video frame, and perform feature fusion on the first frame features corresponding to the current video frame and the second frame features of the previous video frame; Inputting the feature fusion result into a feature extraction and refinement network to extract corresponding refined features, and using the refined features as second frame features of the current video frame; If the current video frame is the first frame of the video to be detected, the current video frame is input into the feature extraction main network to output the first frame feature corresponding to the current video frame through the feature extraction main network, and the first frame feature corresponding to the current video frame is input into the feature extraction refinement network, and the refined feature output by the feature extraction refinement network is used as the second frame feature of the current video frame.

2. The method for identifying the behavior and posture of an obscured part of a human body according to claim 1, wherein: Performing feature fusion on the first frame feature corresponding to the current video frame and the second frame feature of the previous video frame, including: Obtaining an influence coefficient of a previous video frame, and adjusting a second frame feature of the previous video frame according to the influence coefficient to obtain a feature to be fused; Adjusting the size of the feature to be fused according to the first convolution so that the size of the feature to be fused is equal to the size of the first frame feature of the current video frame; Amplifying the number of feature map channels of the feature to be fused according to the second convolution so that the number of feature map channels of the feature to be fused is equal to the number of feature map channels of the first frame feature of the current video frame; The amplified features to be fused are added to the feature maps of the same dimension of the first frame features of the current video frame to complete the feature fusion of the first frame features corresponding to the current video frame and the second frame features of the previous video frame.

3. The method for identifying the behavior and posture of an obscured part of a human body according to claim 1, wherein: The training process of the feature extraction main network and the feature extraction refinement network both includes: Obtaining a first data set, and performing training based on the first data set to obtain a pre-trained model; A second data set is obtained, and the pre-trained model is further trained according to the second data set to obtain a final detection model.

4. The method for identifying the behavior and posture of an obscured part of a human body according to claim 3, wherein: The first data set is a COCO data set, wherein training is performed according to the first data set to obtain a pre-trained model, including: Performing jitter amplification on the COCO dataset to obtain a pre-training dataset; During the training process of the pre-trained model, the original image in the COCO dataset is input as the current video frame, and the image in the pre-trained dataset is input as the previous video frame; The jitter amplification of the COCO dataset includes: Randomly translate the key points randomly given in any image in the COCO dataset; Randomly rotate all key points of any image in the COCO dataset; Scale the keypoint size of any image in the COCO dataset.

5. The method for identifying the behavior and posture of an obscured part of a human body according to claim 3, wherein: The second dataset is the PoseTrack dataset.

6. A computer-readable storage medium, characterized in that A behavior posture recognition program for an obscured part of a human body is stored thereon, and when the behavior posture recognition program for an obscured part of a human body is executed by a processor, the behavior posture recognition method for an obscured part of a human body according to any one of claims 1 to 5 is implemented.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for recognizing the behavior and posture of an obscured part of a human body according to any one of claims 1 to 5 is implemented.

Images