Human body posture recognition method and device based on OpenPose and computer readable storage medium

Abstract

According to the OpenPose-based human body posture recognition method, whether human body skeleton information acquired based on OpenPose is corrected or not is determined according to whether multi-person posture recognition is performed or not. When the human body skeleton information is determined to be corrected, correction is carried out by means of the arm information and the palm information of each person, so that subsequent posture recognition errors caused by false detection of the human body skeletons of other persons in a multi-person application situation are avoided. The invention also provides a device for realizing the OpenPose-based human body posture recognition method and a computer readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a human posture recognition method, device and computer-readable storage medium based on OpenPose. Background Art

[0002] Generally, human posture recognition is usually performed in a top-down manner, that is, pedestrians are first identified, each person is segmented, and finally, the key points of each person are determined based on the segmented independent bodies.

[0003] OpenPose's human posture recognition uses a bottom-up approach, first finding all the key points of the human body in the image, and then assembling them into an independent human skeleton by matching. However, this recognition method has a flaw, that is, specific postures or multiple people can easily lead to misjudgment or cannot be correctly assembled into a human skeleton. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a human posture recognition method, device and computer-readable storage medium based on OpenPose, which can effectively improve the existing known OpenPose recognition defects.

[0005] An embodiment of the present invention provides a human posture recognition method based on OpenPose, the method comprising the following steps: acquiring an image containing a human body; acquiring human skeleton information in the image based on the OpenPose algorithm; judging whether there are multiple people according to the human skeleton information; when judging that there are multiple people, performing the following steps: identifying arm information belonging to each person according to the image; identifying palm information belonging to each person according to the image; judging whether to correct the human skeleton information according to the arm information and the palm information of each person; and recognizing the human postures of multiple people according to the corrected human skeleton information.

[0006] An embodiment of the present invention further provides a human posture recognition device based on OpenPose, the device comprising an image acquisition unit for acquiring images containing a human body; a processor; and a memory for storing a computer program, when the computer program is executed by the processor, the processor implements the human posture recognition method based on OpenPose.

[0007] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the human posture recognition method based on OpenPose is implemented.

[0008] Compared with the prior art, the human posture recognition method, device and computer-readable storage medium based on OpenPose provided by the present invention can effectively avoid incorrect splicing of the human skeleton and cause incorrect recognition of human posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 Flowchart of a human posture recognition method based on OpenPose according to an embodiment of the present invention.

[0010] Figure 2 FIG. 4 is a flow chart of identifying arm information of each person according to an embodiment of the present invention.

[0011] Figure 3 FIG. 4 is a flow chart of identifying palm information of each person according to an embodiment of the present invention.

[0012] Figure 4 FIG. 4 is a schematic diagram of palm recognition according to an embodiment of the present invention.

[0013] Figure 5 FIG. 4 is a hardware block diagram of a human posture recognition device based on OpenPose according to an embodiment of the present invention.

[0014] Main component symbols

[0015]

[0016] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding and implementation of the present invention by those skilled in the art of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the present invention provides many applicable inventive concepts, which can be implemented in a variety of specific forms. Those skilled in the art of the present invention can use the details described in these embodiments or other embodiments and other available structural, logical and electrical changes to implement the invention without departing from the spirit and scope of the present invention.

[0018] The present invention specification provides different embodiments to illustrate the technical features of different implementations of the present invention. Among them, the configuration of each component in the embodiment is for illustration purposes and is not intended to limit the present invention. And the partial repetition of the figure numbers in the embodiment is to simplify the description and does not mean the correlation between different embodiments. Among them, the same component numbers used in the drawings and the description represent the same or similar components. The drawings in this specification are simplified and are not drawn to exact proportions.

[0019] Furthermore, in describing some embodiments of the present invention, the specification describes the method and (or) program of the present invention in a specific order of steps. However, since the method and program are not necessarily implemented according to the specific order of steps described, they are not limited to the specific order of steps described. A person skilled in the art of the present invention will know that other orders are also possible implementation methods. Therefore, the specific order of steps described in the specification is not used to limit the scope of the patent application. Furthermore, the scope of the patent application for the method and (or) program of the present invention is not limited to the order of execution steps written therein, and a person skilled in the art of the present invention will understand that adjusting the order of execution steps does not deviate from the spirit and scope of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0021] See also Figure 1 , which is a flow chart of a human posture recognition method based on OpenPose according to an embodiment of the present invention. The method comprises the following steps:

[0022] Step S101, acquiring an image containing a human body.

[0023] In this embodiment, an image including a human body may be acquired via an image acquisition unit, such as a camera device.

[0024] Step S102: obtaining human skeleton information in the image based on the OpenPose algorithm.

[0025] In a feasible implementation, the OpenPose algorithm can be used to obtain multiple human key point information and corresponding human skeleton information in the image.

[0026] Step S103, judging whether there are multiple people according to the human skeleton information.

[0027] When it is determined that there are multiple persons, step S104 is executed; when it is determined that there are not multiple persons, that is, there is only single human skeleton information, step S107 is executed to identify the human posture according to the human skeleton information.

[0028] Step S104, identifying arm information belonging to each of the multiple persons.

[0029] In a feasible manner, key point information corresponding to the arm can be extracted from the multiple human key point information obtained in step S102, and then the arm information belonging to each person in the multiple people can be obtained, wherein the arm information can include the position information of the human key point corresponding to the arm.

[0030] Step S105, identifying palm information belonging to each of the multiple persons.

[0031] In a feasible manner, key point information corresponding to the palm can be extracted from the multiple human key point information obtained in step S102, and then palm information belonging to each of the multiple people can be obtained, wherein the palm information can include position information of the human key points corresponding to the palm.

[0032] Step S106, judging whether to modify the human skeleton information according to the arm information and the palm information of each person.

[0033] Because in multi-person application scenarios, the arm or palm of one person is often blocked by another person, therefore, based on the arm information and palm information of each person, it is determined whether to correct the human skeleton information to prevent erroneous human skeleton information from affecting the accuracy of human posture recognition.

[0034] Step S107, recognizing the human postures of multiple persons based on the corrected human skeleton information.

[0035] See also Figure 2 , which is a flowchart of identifying the arm information of each person according to an embodiment of the present invention, is an embodiment of step S104.

[0036] Step S201: locating multiple arms in the image based on the OpenPose algorithm.

[0037] Step S202, identifying vein features of multiple arms.

[0038] In one embodiment, a convolutional neural network can be used to identify the vein features of multiple arms. In a feasible implementation, the identified vein features are mainly elbow veins, because there are obvious and elastic blood vessels there.

[0039] In one embodiment, in order to better identify the vein features of multiple arms, the image may be preprocessed first, and the preprocessing is mainly to enhance the contrast of the image. In a feasible manner, the preprocessing may include the following steps: (1) performing histogram statistics on the image regions corresponding to the multiple arms, and counting the probability of each hue appearing on the arms; (2) increasing the saturation of the blue hue and the green hue in the image region; (3) preliminarily determining the vein locations on the multiple arms, and blackening the red part at the vein location; (4) converting the image region into a grayscale image and normalizing it to more clearly show the vein features.

[0040] Step S203, identifying arm information belonging to each person according to the vein features of the multiple arms.

[0041] In one embodiment, the arm information includes whether the arm is a left arm or a right arm and the outside or inside of the arm. Since the venous blood vessel distribution diagrams of the left and right arms of the human body are opposite, the left and right arms of each person can be identified based on the venous distribution image of the vein characteristics. Taking the cubital vein as an example, the cubital vein includes the cephalic vein, the basilic vein and the middle ulnar vein. According to the venous blood vessel distribution of the cubital vein, the arm can be identified as the left arm or the right arm. In addition, according to whether the intersection of the basilic vein and the middle ulnar vein is identified, it can be determined whether the arm is the inside arm or the outside arm relative to the human body. When the intersection of the basilic vein and the middle ulnar vein is identified, the arm is determined to be the inside arm; when the intersection of the basilic vein and the middle ulnar vein cannot be identified, the arm is determined to be the outside arm. Among them, the inside arm refers to the side close to the human body when the arm is naturally hanging; the outside arm refers to the side away from the human body when the arm is naturally hanging.

[0042] See also Figure 3 , which is a flow chart of identifying palm information of each person according to an embodiment of the present invention, is an embodiment of step S105.

[0043] Step S301: locating multiple palms in the image based on the OpenPose algorithm.

[0044] Step S302, obtaining a plurality of valley points between fingers of each palm.

[0045] In a feasible manner, the image may be binarized, and a plurality of valley points between fingers of each palm may be obtained according to the color difference of the binarized image.

[0046] Step S303: Acquire multiple wrist key points identified by the OpenPose algorithm.

[0047] Step S304: identifying the central area of ​​each palm according to the multiple valley points between the fingers and the multiple wrist key points of each palm.

[0048] like Figure 4 As shown, A, B, C, and D are multiple valley points between fingers of a single palm, and E is a wrist key point of the palm. The closed area formed by sequentially connecting the valley points between fingers and the wrist key points is the central area S of the palm.

[0049] Step S305, obtaining the position information of the thumb valley point of each palm.

[0050] In a feasible method, for each palm, the position information of the valley points between multiple fingers is first obtained, and then the sum of the distances between each valley point between fingers and the valley points between other fingers is obtained, and the thumb valley point among the multiple valley points between fingers is judged based on the sum of the distances, and the position information of the thumb valley point is obtained at the same time.

[0051] by Figure 4 For example, the two-dimensional coordinates of the valley points A, B, C, and D between fingers are A(x1, y1), B(x2, y2), C(x3, y3), and D(x4, y4). Using the distance formula between two points in the coordinate plane, we can get the distance between each valley point between fingers and the other three valley points. Point A is d AB ,d AC ,d AD , point B is d BA ,d BC ,d BD , point C is d CA ,d CB ,d CD , point D is d DA ,d DB ,d DC . Let d A =d AB +d AC +d AD ,d B =d BA +d BC +d BD ,d C =d CA +d CB +d CD ,d D =d DA +d DB +d DC . Take the largest one, d X =max(d A , d B , d c , d D ), and X∈(A, B, C, D), it is judged that point X is the thumb valley point. Figure 4 In the figure, point X is point A, that is, point A is the valley point of the thumb.

[0052] Step S306, identifying palm information belonging to each person according to the position information of the center area of ​​each palm and the thumb valley point of each palm, wherein the palm information includes the left hand or the right hand, the back of the hand or the palm.

[0053] Specifically, image recognition is performed on the image of the central area of ​​each palm to determine whether there are sweat hairs in the central area of ​​each palm. When sweat hairs are present in the central area, the central area is determined to be the back of the hand area, and when sweat hairs are not present in the central area, the central area is determined to be the palm area. Secondly, when the thumb valley point is located at point A and the central area of ​​the palm is the palm area, the palm is the right hand; when the thumb valley point is located at point D and the central area of ​​the palm is the palm area, the palm is the left hand; when the thumb valley point is located at point A and the central area of ​​the palm is the back of the hand area, the palm is the left hand; and when the thumb valley point is located at point D and the central area of ​​the palm is the back of the hand area, the palm is the right hand.

[0054] See also Figure 5 , which is a hardware block diagram of a human posture recognition device based on OpenPose according to an embodiment of the present invention. The device 400 includes a processor 402, a memory 404, a computer program 406, and an image acquisition unit 408. The device 400 is an electronic device. Those skilled in the art should understand that Figure 5 The composition of the device 400 shown does not constitute a limitation of the embodiment of the present invention. Figure 5 The device 400 shown in FIG. 4 is simplified for ease of description, and in different embodiments, may include fewer or more components than shown.

[0055] In one embodiment, the processor 402 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 402 is the control core (Control Unit) of the device 400, and uses various interfaces and lines to connect various components of the entire device 400, and executes or executes the computer program 406 or module stored in the memory 404, and calls the data stored in the memory 404 to execute various functions of the device 400 and process data, such as the human posture recognition method based on OpenPose.

[0056] In one embodiment, the memory 404 is used to store the codes and various data of the computer program 406, such as the target image capture method, and to achieve high-speed and automatic access to the program or data during the operation of the device 400. The memory 404 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable storage medium capable of carrying or storing data.

[0057] In one embodiment, the image acquisition unit 408 includes a camera device for acquiring images in real time for human posture recognition based on OpenPose.

[0058] In summary, the OpenPose-based human posture recognition method, device, and computer-readable storage medium of the present invention can effectively solve the occlusion problem of some parts of the human body in multi-person application scenarios and improve the accuracy of human posture recognition.

[0059] It is worth noting that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A human posture recognition method based on OpenPose, characterized in that: The following steps are involved: Acquire an image containing a human body; Based on the OpenPose algorithm, obtaining human skeleton information in the image; Determining whether there are multiple persons according to the human skeleton information; When it is determined that there are multiple people, perform the following steps: According to the image, identifying arm information belonging to each person; According to the image, identifying palm information belonging to each person; Determining whether to modify the human skeleton information according to the arm information and the palm information of each person; as well as According to the corrected human skeleton information, the human postures of multiple people are recognized.

2. The human posture recognition method based on OpenPose as claimed in claim 1, characterized in that: The step of identifying arm information belonging to each person according to the image further includes: Based on the OpenPose algorithm, multiple arms are located in the image; identifying vein features of the plurality of arms; According to the vein features of the multiple arms, the arm information of each person is identified.

3. The method for human posture recognition based on OpenPose as claimed in claim 2, characterized in that: The venous feature includes a cubital vein.

4. The method for human posture recognition based on OpenPose as claimed in claim 2, characterized in that: The step of identifying the vein features of the multiple arms also includes: using a convolutional neural network to identify the vein features of the multiple arms.

5. The method for human posture recognition based on OpenPose as claimed in claim 2, characterized in that: The step of identifying the vein features of the plurality of arms also includes: Performing histogram statistics on the image regions corresponding to the plurality of arms to count the probability of occurrence of each hue; increasing the saturation of blue hue and green hue in the image area; Determine the locations of the veins on the multiple arms, and perform blackening processing on the red parts of the vein locations; as well as The image region is converted into a grayscale image and normalized.

6. The method for human posture recognition based on OpenPose as claimed in claim 2, characterized in that: The step of identifying the arm information of each person according to the vein features of the multiple arms also includes: Depending on whether the intersection of the basilic vein and the middle ulnar vein is identified, it is determined whether each of the multiple arms is an inner arm or an outer arm relative to the human body.

7. The method for human posture recognition based on OpenPose as claimed in claim 1, characterized in that: The step of identifying palm information belonging to each person according to the image further includes: Based on the OpenPose algorithm, multiple palms are located in the image; Obtain multiple valley points between fingers of each palm; Get multiple wrist key points identified based on the OpenPose algorithm; Identify the central area of ​​each palm according to the multiple valley points between fingers of each palm and the multiple wrist key points; Obtain the position information of the thumb valley point of each palm; as well as Palm information belonging to each person is identified based on the position information of the central area of ​​each palm and the thumb valley point of each palm.

8. The method for human posture recognition based on OpenPose as claimed in claim 7, characterized in that: The step of identifying the palm information belonging to each person also includes: Performing image recognition on the image of the central area of ​​each palm to determine whether there is hair in the central area of ​​each palm; When it is determined that there are hairs in the central area, the central area is determined to be the back of the hand area; as well as When it is determined that there are no hairs in the central area, the central area is determined to be the palm area.

9. A human posture recognition device based on OpenPose, characterized in that: include: An image acquisition unit, used for acquiring images containing a human body; processor; as well as A memory for storing a computer program, wherein when the computer program is executed by the processor, the processor implements the human posture recognition method based on OpenPose as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the OpenPose-based human posture recognition method is implemented.

Images