A skeletal point occlusion detection method, device, equipment and storage medium

Abstract

The application provides a kind of skeleton point occlusion detection method, device, equipment and storage medium. First, the position information of each skeleton point in human skeleton is obtained by acquiring color image, and the skeleton point set with connection relationship is obtained using the position information of each skeleton point. Then, the target region is obtained by region construction of human skeleton according to the connection relationship between each skeleton point. Finally, the occlusion detection result is obtained by judging the positional relationship between skeleton point and target region. That is, the application only needs to calibrate the target region according to the skeleton point with connection relationship, and can directly judge whether the detected skeleton point is occluded according to the positional relationship between skeleton point and target region. Therefore, compared with the prior art, the calculation amount of the application is small, and the occlusion of the skeleton point can be quickly and effectively judged by simple logic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a skeleton point occlusion detection method, device, equipment and storage medium. BACKGROUND

[0002] In the fields of video monitoring and search, human-computer interaction, home fitness, smart care for the aged, virtual reality, etc., human pose estimation is needed to understand and analyze human behaviors. Generally, human pose is estimated based on the positional relationship of un-occluded skeleton points, but if the estimation is performed using occluded skeleton points, the obtained human pose deviates from the actual situation, so that the human pose estimation result is not accurate enough.

[0003] In the prior art, the judgment of skeleton point occlusion often adopts a human segmentation method based on deep learning, that is, a human image is subjected to pixel-level fine-grained segmentation to obtain multiple parts, and based on the segmentation of each part, it is judged whether each skeleton point is occluded; but this method not only depends on a large amount of training data, but also needs a complex network model for fine segmentation, so that the calculation amount is large and the calculation time is long, and the detection efficiency of skeleton point occlusion is low. SUMMARY

[0004] Therefore, the present application provides a skeleton point occlusion detection method, device, equipment and storage medium, which solves the problem of low detection efficiency of skeleton point occlusion in the prior art.

[0005] According to an embodiment of the present application, a skeleton point occlusion detection method is provided, comprising: acquiring a color image and a depth image which are aligned and include a target user's human skeleton; obtaining position information of each to-be-detected skeleton point in the human skeleton by using the color image, and obtaining a plurality of to-be-detected skeleton point sets having connection relationships according to the position information; wherein the connection relationship is that the connection of each to-be-detected skeleton point in the to-be-detected skeleton point set can form any part of the human torso; based on the alignment relationship between the color image and the depth image, constructing a region by using the position information between each to-be-detected skeleton point in the to-be-detected skeleton point set and the corresponding depth information, to obtain a plurality of target regions; judging the positional relationship between the to-be-detected skeleton point and the target region, to obtain an occlusion detection result of the to-be-detected skeleton point.

[0006] According to an embodiment of the present application, a skeleton point occlusion detection device is also provided, comprising: an image acquisition module, configured to acquire a color image and a depth image which are aligned and comprise a target user's body skeleton; a skeleton point set calculation module, configured to acquire position information of each to-be-detected skeleton point in the body skeleton by using the color image, and to acquire a plurality of to-be-detected skeleton point sets having connection relationships according to the position information; wherein the connection relationship is that connection of each to-be-detected skeleton point in the to-be-detected skeleton point set can form any part of the body trunk; a target region generation module, configured to perform region construction by using position information between each to-be-detected skeleton point in the to-be-detected skeleton point set and corresponding depth information, to obtain a plurality of target regions based on the alignment relationship between the color image and the depth image; and an occlusion detection module, configured to judge a positional relationship between the to-be-detected skeleton point and the target region, and to obtain an occlusion detection result of the to-be-detected skeleton point.

[0007] According to an embodiment of the present application, a terminal device is also provided, wherein the terminal device comprises a memory, a processor, and a skeleton point occlusion detection program stored in the memory and executable on the processor; when the processor executes the skeleton point occlusion detection program, the steps of the above-mentioned skeleton point occlusion detection method are implemented.

[0008] According to an embodiment of the present application, a computer readable storage medium is also provided, wherein the computer readable storage medium stores a skeleton point occlusion detection program; when the skeleton point occlusion detection program is executed by a processor, the steps of the above-mentioned skeleton point occlusion detection method are implemented.

[0009] The present application has the following advantages: the present application first acquires position information of each skeleton point in a body skeleton by using a color image, and acquires skeleton point sets having connection relationships by using the position information of each skeleton point; then target regions are obtained by performing region construction on the body skeleton according to the connection relationship between each skeleton point; finally, an occlusion detection result is obtained by judging a positional relationship between a skeleton point and a target region; that is, the present application only needs to calibrate target regions according to skeleton points having connection relationships, and can directly judge whether a to-be-detected skeleton point is occluded according to a positional relationship between the skeleton point and the target region. Therefore, compared with the prior art, the present application has a smaller amount of calculation, and can quickly and effectively judge an occlusion condition of a skeleton point by using simple logic. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A flowchart of a skeleton point occlusion detection method provided by the present application is shown in the figure;

[0011] Figure 2 A structure diagram of a human skeleton point model provided by an embodiment of the present application is shown in the figure;

[0012] Figure 3 A non-wrist skeleton point occlusion judgment model provided by an embodiment of the present application is shown in the figure;

[0013] Figure 4 A wrist skeleton point shielding judgment model is provided for an embodiment of the present application.

[0014] Figure 5 A left hip shielding by a left knee-left ankle located calf schematic diagram is provided for an embodiment of the present application.

[0015] Figure 6 A left hip shielding by a left knee-left ankle located calf schematic diagram is provided for an embodiment of the present application.

[0016] Figure 7 A left hip shielding by a left knee-left ankle located calf schematic diagram is provided for an embodiment of the present application.

[0017] Figure 8 An internal structure principle block diagram of a terminal device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the present application are described clearly and completely in combination with embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0019] The present application provides a skeleton point shielding detection method, device, equipment and storage medium, which only needs to calibrate a target area according to the skeleton points with a connection relationship, and then can judge whether the to-be-detected skeleton point is shielded according to the position relationship between the to-be-detected skeleton point and the target area. The method used in the present application has a small amount of calculation, and can quickly judge whether the to-be-detected skeleton point is shielded, solving the problem of low skeleton point shielding detection efficiency in the prior art.

[0020] For example, the corresponding skeleton point images of the left and right arms are collected, and the images include a left elbow skeleton point, a left wrist skeleton point, a right elbow skeleton point and a right wrist skeleton point. Since the skeleton point image is a 2D image, and there are only each skeleton point (a dot) and the connection relationship between each skeleton point on the 2D image, it is impossible to judge from the 2D image whether the right wrist is shielded by the left elbow and the left arm where the left wrist is located when the human body photo is taken. Therefore, to judge whether the right wrist is shielded by the left elbow and the left arm where the left wrist is located, the present embodiment calibrates the area formed by the left elbow skeleton point and the left wrist skeleton point as a target area, and then judges whether the right wrist skeleton point is in the target area to obtain the shielding condition of the right wrist.

[0021] Exemplary method

[0022] Figure 1A flowchart of a skeleton point occlusion detection method provided by the present application is shown in the figure. The method can be applied to a terminal device, which can be a terminal product with image acquisition function, such as a mobile phone, a computer, etc. As shown in the figure, Figure 1 The skeleton point occlusion detection method specifically includes the following steps:

[0023] S100, a color image and a depth image including a target user's body skeleton are acquired.

[0024] In one embodiment, a color image including a target user's body skeleton can be acquired by an acquisition device. Preferably, the color image can be an image in RGB format, YUV format or HSV format, which is not limited here.

[0025] In another embodiment, the acquisition device can also synchronously or in sequence acquire a depth image of the same scene as the color image. Further, after the depth image is acquired, each pixel point of the depth image and each corresponding pixel point of the color image are aligned one by one to obtain a depth image and a color image with an alignment relationship, i.e., the depth image and the color image can be converted to the same coordinate system through the alignment relationship.

[0026] S200, the position information of each to-be-detected skeleton point in the body skeleton is acquired by using the color image, and a plurality of to-be-detected skeleton point sets with connection relationship are acquired according to the position information of each to-be-detected skeleton point.

[0027] Figure 2 A position relationship structure diagram of a skeleton point in a body skeleton provided by the present embodiment is shown in the figure. L and R respectively represent left and right, and index numbers from 0 to 14 represent the positions of skeleton points of the human body, in order, head, neck, middle hip, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right ankle and left ankle. When images of the human body in different postures are acquired, a preset skeleton point detection algorithm can be used to perform skeleton point detection on the color image to obtain the position information of each to-be-detected skeleton point, and a skeleton point set with connection relationship is acquired according to the position information of each to-be-detected skeleton point, so as to facilitate subsequent human posture estimation. It should be noted that the preset skeleton point detection algorithm can be an existing algorithm such as AlphaPose or openpose, which is not limited here.

[0028] In one embodiment, acquiring a plurality of to-be-detected skeleton point sets with connection relationship according to the position information of each to-be-detected skeleton point more specifically includes: judging whether each to-be-detected skeleton point has a connection relationship according to the position information between each to-be-detected skeleton point, if the connection line between the to-be-detected skeleton points can form any part of the human body trunk, it is judged that the to-be-detected skeleton points have a connection relationship, and then the to-be-detected skeleton points with a connection relationship are collected in the order of the connection line that can form any part of the human body trunk to form a to-be-detected skeleton point set. FurtherFigure 2 As shown, the line connecting the to-be-detected skeletal points corresponding to index number 1 and index number 4 can form a left upper limb in the human torso, i.e., a to-be-detected skeletal point set having a connection relationship; the four to-be-detected skeletal points corresponding to index number 9, 11, and 13 can be sequentially connected to form a right leg in the human torso, so the four to-be-detected skeletal points also form a to-be-detected skeletal point set having a connection relationship; and the to-be-detected skeletal points corresponding to index number 5 and index number 6 cannot be sequentially connected to form any part of the human torso, so the to-be-detected skeletal points corresponding to index number 5 and index number 6 cannot form a to-be-detected skeletal point set having a connection relationship, and the same applies to other cases, which will not be described herein again.

[0029] S300, based on the alignment relationship between the color image and the depth image, using the position information between the to-be-detected skeletal points in the to-be-detected skeletal point set and the corresponding depth information, a region is constructed to obtain a plurality of target regions.

[0030] More specifically, the target region is a region constructed based on the to-be-detected skeletal point set, which corresponds to the limbs where each to-be-detected skeletal point is located. In an embodiment, taking a to-be-detected skeletal point set as an example, step S300 further includes steps S301 and S302:

[0031] S301, according to the alignment relationship between the color image and the depth image, the position information and the corresponding depth information of the first and last skeletal points in the to-be-detected skeletal point set are obtained, respectively, and a spherical model is constructed using the position information and the corresponding depth information to obtain a first region and a second region.

[0032] In some embodiments, after obtaining the position information of the first and last to-be-detected skeletal points in the to-be-detected skeletal point set, a spherical model is constructed with the respective positions of the first and last to-be-detected skeletal points as the center of the sphere and with a set radius R to obtain the first region and the second region. It should be noted that in this embodiment, the spherical model can also be an ellipsoidal model, which is not limited herein.

[0033] In an embodiment, the radius R is set to 2*X / hipDepth, where X is a constant value and hipDepth is the depth information of the first and last to-be-detected skeletal points. As can be seen from the above formula, the radius R changes with the distance of the target user from the camera, i.e., when the target user is close to the camera, the radius R increases, and the pixel area occupied by the target region also increases; when the target user is far away from the camera, the radius R decreases, and the pixel area occupied by the target region also decreases; that is, the target region constructed by the present application changes in real time with the distance of the target user from the camera, so that the occlusion of the skeletal points can be more accurately determined in the subsequent steps.

[0034] S302, a cylindrical model is constructed to enclose the first region and the second region to obtain a target region.

[0035] More specifically, a column model is constructed by connecting each of the to-be-detected skeletal points between the first region and the second region and taking the connection as an axis, and the column model can enclose the first region and the second region to form the target region. It should be understood that the present application is only described with one set of to-be-detected skeletal points, and when there are multiple sets of to-be-detected skeletal points, the above steps can be repeated, which will not be described here.

[0036] For example, as shown in FIG. 3, the first and last to-be-detected skeletal points in the set of to-be-detected skeletal points are the left shoulder skeletal point (joint1) and the left elbow skeletal point (joint2), and a spherical model is constructed with joint1 and joint2 as the centers and with a radius R to obtain the first region and the second region; then a column model is established with the first region or the second region as the base surface to enclose the first region and the second region to form the target region; wherein the height of the column model is the distance between the first region and the second region. It should be noted that the target region of the present embodiment can be a three-dimensional region, or the three-dimensional target region can be projected onto a plane and the region projected on the plane can be taken as the final target region. Figure 3 It should be noted that the target region obtained above is for a regular limb region. When the limb region in the target region needs to be connected to a palm or a foot, adaptive processing needs to be performed on the target region obtained based on the connection point (such as a wrist point or an ankle point). Specifically, when the target region to be processed has a wrist skeletal point, because the wrist skeletal point is the connection point of the arm and the palm and the diameter of the palm is generally larger than the diameter of the arm, spherical models are constructed with the first and last skeletal points as the centers and with different radii to obtain first regions and second regions of different sizes; wherein the size of the first region is larger than that of the second region. Further, a column model is constructed by extending from the second region to the first region to obtain the target region, wherein the extension length is greater than the length between each first region, as shown in FIG. 4.

[0037] As can be seen from FIG. 4, the first regions of different sizes are composed of two spherical bodies with different radii (i.e., R and R1), and the length of the second region is longer than the distance between the centers of the two spherical bodies by R2. Figure 4 Figure 4 S400, judging the positional relationship between the to-be-detected skeletal point and the target region to obtain the occlusion detection result of the to-be-detected skeletal point.

[0038] S400, judging the positional relationship between the to-be-detected skeletal point and the target region to obtain the occlusion detection result of the to-be-detected skeletal point.

[0039] ​In this embodiment, the positional relationship between the detected skeleton point and the target area includes two cases: the detected skeleton point is located outside the target area and the detected skeleton point is located inside the target area. When the detected skeleton point is located outside the target area, it means that the detected skeleton point is not occluded and no further judgment is needed; when the detected skeleton point is located inside the target area, it means that the detected skeleton point may be occluded and further judgment is needed.

[0040] In one embodiment, when the bone point to be detected is located inside the target region, step S400 includes the following steps S401 and S402:

[0041] S401, based on the alignment relationship between the color image and the depth image, obtain the depth value of the target region and the depth value of the bone point to be detected.

[0042] S402, compare the depth value of the target region with the depth value of the bone point to be detected. If the depth value corresponding to the bone point to be detected is less than the depth value of each pixel in the target region, then the bone point to be detected is not occluded; if the depth value corresponding to any pixel in the target region is less than the depth value corresponding to the bone point to be detected, then the bone point to be detected is occluded.

[0043] Specifically, such as Figure 3 As shown, joint3 is the current skeleton point to be detected, and joint1 and joint2 are the first and last skeleton points to be detected that make up the target region. The first and second regions constructed based on joint1 and joint2 are merged into the target region. If joint3 is within the target region, it is considered that the skeleton point may be occluded and further confirmation is needed. If joint3 is within the target region, the depth values ​​of the skeleton point to be detected and the target region need to be combined to determine the front-back relationship between the skeleton point to be detected and the target region, and based on this relationship, it can be determined whether the skeleton point is actually occluded. For example, when joint1, joint2, and joint3 are left shoulder-left elbow-right wrist respectively, if joint3 is found to be within the target region constructed based on joint1 and joint2 using the above method, it can be initially considered that the right wrist is occluded by the left shoulder-left elbow. Furthermore, the depth values ​​at the left shoulder, left elbow, and right wrist are obtained. If the depth of the right wrist is less than the depth values ​​of the left shoulder and left elbow, it is considered that the right wrist is not occluded; otherwise, it is considered that the right wrist is occluded.

[0044] It should be noted that S401 and S402 determine whether the bone point to be detected is occluded based on the depth of each pixel within the target area. In another embodiment, the occlusion of the bone point to be detected can also be determined based on the depth value corresponding to the axis formed by the beginning and end of the bone points to be detected in the target area. More specifically, this includes the following steps:

[0045] S403, obtaining a depth value corresponding to an axis formed by the two end skeleton points in the target region according to the alignment relationship between the depth image and the color image.

[0046] In one embodiment, the axis in the embodiment corresponds to a limb or a trunk of a human body, for example, the two end skeleton points in the target region are a left wrist skeleton point and a left elbow skeleton point, and the axis is a line connecting the left wrist skeleton point and the left elbow skeleton point (i.e., a left arm).

[0047] S404, comparing the depth value corresponding to the axis and the depth value corresponding to the skeleton point to be detected to obtain an occlusion detection result of the skeleton point to be detected.

[0048] More specifically, when the depth value corresponding to the skeleton point to be detected is less than the depth value corresponding to the axis, the skeleton point to be detected is not occluded; otherwise, the skeleton point to be detected is occluded.

[0049] In another embodiment, after obtaining the occlusion detection result of the skeleton point to be detected, human posture estimation can be performed using the occlusion detection result of the skeleton point to be detected, and more specifically, after step S400, the method further comprises:

[0050] S501, obtaining coordinate information of the skeleton point to be detected that is occluded in the occlusion detection result;

[0051] S502, removing the skeleton point to be detected that is occluded in the color image according to the coordinate information of the skeleton point to be detected that is occluded to obtain a skeleton point to be detected that is not occluded;

[0052] S503, obtaining depth information of the skeleton point to be detected that is not occluded based on the alignment relationship between the depth image and the color image;

[0053] S504, performing human posture estimation using the depth information of the skeleton point to be detected that is not occluded to obtain a three-dimensional human posture model.

[0054] Specifically, as shown in Figure 5 , Figure 6 and Figure 7 , which are, in sequence, three-dimensional human posture model result schematic diagrams obtained by using the embodiments provided in the present application; wherein, Figure 5 is a human posture model schematic diagram in which a left hip is occluded by a left elbow-left wrist; Figure 6 is a human posture model schematic diagram in which a left hip is occluded by a lower leg; Figure 7 is a human posture model schematic diagram in which a left shoulder is occluded by a left lower arm.

[0055] In summary, this invention only requires identifying the target region using connected skeletal points, and then determining whether the skeletal point to be detected is occluded based on its positional relationship with the target region. Compared with existing technologies, the method provided by this invention has a lower computational load, and can quickly and effectively determine whether the skeletal point to be detected is occluded through simple logic.

[0056] Exemplary apparatus

[0057] This embodiment also provides a skeletal point occlusion detection device, which includes the following components:

[0058] The image acquisition module is used to acquire aligned color and depth images, including the target user's human skeleton.

[0059] The skeleton point set calculation module is used to obtain the position information of each skeleton point to be detected in the human skeleton using color images, and to obtain several skeleton point sets with connection relationships based on the position information of each skeleton point to be detected; wherein, the connection relationship is that the connection of each skeleton point to be detected in the skeleton point set can constitute any part of the human torso.

[0060] The target region generation module is used to construct target regions based on the alignment relationship between the color image and the depth image, utilizing the connection relationship between each bone point to be detected in the bone point set and its corresponding depth information, thereby obtaining several target regions.

[0061] The occlusion detection module is used to determine the positional relationship between the bone point to be detected and the target area, and to obtain the occlusion detection result of the bone point to be detected.

[0062] It should be noted that the specific functions of each module are explained in the aforementioned skeletal point detection method, and will not be repeated here.

[0063] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 8 As shown. The terminal device may include one or more processors 800 ( Figure 8 (Only one is shown in the image), a memory 801, and a computer program 802 stored in the memory 801 and executable on one or more processors 800, such as a skeletal point occlusion detection program. When one or more processors 800 execute the computer program 802, they can implement the various steps in the method embodiment for skeletal point occlusion detection. Alternatively, when one or more processors 800 execute the computer program 802, they can implement the functions of the various modules / units in the embodiment of the skeletal point occlusion detection device, which is not limited here.

[0064] In one embodiment, the terminal device further comprises a collection device, which can include a depth camera and a two-dimensional camera, wherein the depth camera can be any one of a structured light camera, a TOF camera, a binocular camera, etc., and the two-dimensional camera can be any one of an infrared camera, a color camera, etc., for collecting a depth image and a two-dimensional image respectively, and the processor uses the collected depth image and two-dimensional image to implement each step in the above-described embodiment of the skeleton point occlusion detection method. It should be noted that the depth camera and the two-dimensional camera can be integrally arranged or independently arranged, and the collection device can be arranged in the terminal device or can perform data transmission with the terminal device through wireless communication or wired communication, which is not limited herein.

[0065] Those skilled in the art can understand that, Figure 8 The principle block diagram shown in the above embodiments is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the terminal device to which the present application scheme is applied. The specific terminal device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0066] In one embodiment, a terminal device is provided, which comprises a memory, a processor, and a skeleton point occlusion detection program stored in the memory and executable on the processor. When the processor executes the skeleton point occlusion detection program, the following operation instructions are implemented:

[0067] Obtaining a color image and a depth image that include an aligned human skeleton of a target user;

[0068] Obtaining position information of each to-be-detected skeleton point in the human skeleton from the color image, and obtaining a plurality of to-be-detected skeleton point sets having a connection relationship according to the position information; wherein the connection relationship is that the to-be-detected skeleton points in the to-be-detected skeleton point sets can connect to form any part of a human torso;

[0069] Based on the alignment relationship between the color image and the depth image, constructing a region using the position information between each to-be-detected skeleton point in the to-be-detected skeleton point set and the corresponding depth information, to obtain a plurality of target regions;

[0070] Judging the position relationship between the to-be-detected skeleton point and the target region, and obtaining an occlusion detection result of the to-be-detected skeleton point.

[0071] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0072] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0073] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / equipment and method can be implemented in other ways. For example, the apparatus / equipment embodiments described above are merely schematic, for example, the division of modules or units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0074] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0075] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0076] The integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on the above understanding, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting skeletal point occlusion, characterized in that, include: Acquire aligned color and depth images of the target user's human skeleton; The position information of each bone point to be detected in the human skeleton is obtained using the color image, and several sets of bone points to be detected with connection relationships are obtained based on the position information; wherein, the connection relationship is that the connection of each bone point to be detected in the set of bone points to be detected can form any part of the human torso. Based on the alignment relationship between the color image and the depth image, regions are constructed using the positional information and corresponding depth information of each bone point in the set of bone points to be detected, resulting in several target regions. Determine the positional relationship between the bone point to be detected and the target region to obtain the occlusion detection result of the bone point to be detected; Based on the alignment relationship between the color image and the depth image, regions are constructed using the positional information and corresponding depth information of the bone points in the set of bone points to be detected, resulting in several target regions, including: Based on the alignment relationship between the color image and the depth image, the position information and corresponding depth information of the first and last skeleton points in the skeleton point set to be detected are obtained respectively. A spherical model is then constructed using the position information and corresponding depth information to obtain each first region and its corresponding second region. The center of the spherical model is the position of the first and last skeleton points to be detected, and the radius R of the spherical model is: X is a constant value, hipDepth is the depth information, and the target area will change in real time as the target user is closer to the camera; A columnar model is constructed to enclose each of the first regions and the corresponding second regions to obtain the target regions.

2. The skeletal point occlusion detection method as described in claim 1, characterized in that, The step of determining the positional relationship between the skeleton point to be detected and the target region to obtain the occlusion detection result of the skeleton point to be detected includes: When the bone point to be detected is located outside the target area, the bone point to be detected in the occlusion detection result is not occluded.

3. The skeletal point occlusion detection method as described in claim 1, characterized in that, The step of determining the positional relationship between the skeleton point to be detected and the target region to obtain the occlusion detection result of the skeleton point to be detected includes: When the bone point to be detected is located inside the target region, the depth value of the target region and the depth value of the bone point to be detected are obtained according to the alignment relationship between the color image and the depth image. By comparing the depth value of the target region with the depth value of the bone point to be detected, the occlusion detection result of the bone point to be detected is obtained.

4. The skeletal point occlusion detection method as described in claim 3, characterized in that, The step of comparing the depth value of the target region with the depth value of the bone point to be detected to obtain the occlusion detection result of the bone point to be detected includes: If the depth values ​​of the bone points to be detected are all less than the depth values ​​of each pixel in the target region, then the bone points to be detected are not occluded. If the depth value of any pixel in the target region is less than the depth value of the bone point to be detected, then the bone point to be detected is occluded.

5. The skeletal point occlusion detection method as described in claim 1, characterized in that, The step of determining the positional relationship between the skeleton point to be detected and the target region to obtain the occlusion detection result of the skeleton point to be detected includes: When the bone point to be detected is located inside the target region, the depth value corresponding to the axis formed by the first and last bone points to be detected in the target region is obtained according to the alignment relationship between the depth image and the color image. By comparing the depth value corresponding to the axis and the depth value corresponding to the bone point to be detected, the occlusion detection result of the bone point to be detected is obtained.

6. The skeletal point occlusion detection method as described in claim 5, characterized in that, The step of comparing the depth value corresponding to the axis and the depth value corresponding to the bone point to be detected to obtain the occlusion detection result of the bone point to be detected includes: If the depth value corresponding to the bone point to be detected is less than the depth value corresponding to the axis, then the bone point to be detected is not occluded. If the depth value corresponding to the bone point to be detected is greater than the depth value corresponding to the axis, then the bone point to be detected is occluded.

7. The skeletal point occlusion detection method as described in claim 1, characterized in that, After determining the positional relationship between the skeleton point to be detected and the target region, and obtaining the occlusion detection result of the skeleton point to be detected, the method further includes: Obtain the coordinate information of the occluded bone points to be detected in the occlusion detection results; Based on the coordinate information of the occluded bone points to be detected, the occluded bone points to be detected in the color image are removed to obtain the unoccluded bone points to be detected. Based on the alignment relationship between the depth image and the color image, the depth value of the unoccluded bone point to be detected is obtained; Human pose estimation is performed using the depth values ​​of the unoccluded bone points to be detected, resulting in a three-dimensional human pose model.

8. A skeletal point occlusion detection device, characterized in that, The device comprises the following components: The image acquisition module is used to acquire aligned color and depth images, including the target user's human skeleton. The skeleton point set calculation module is used to obtain the position information of each skeleton point to be detected in the human skeleton using the color image, and obtain a number of skeleton point sets to be detected with connection relationship based on the position information; wherein, the connection relationship is that the connection of each skeleton point to be detected in the skeleton point set to be detected can form any part of the human torso. The target region generation module is used to construct a region based on the alignment relationship between the color image and the depth image, using the position information between each bone point to be detected in the bone point set to be detected and its corresponding depth information, to obtain a number of target regions. An occlusion detection module is used to determine the positional relationship between the bone point to be detected and the target area, and to obtain the occlusion detection result of the bone point to be detected. The alignment relationship between the color image and the depth image is used to construct regions using the positional information and corresponding depth information of the bone points in the set of bone points to be detected, resulting in several target regions, including: Based on the alignment relationship between the color image and the depth image, the position information and corresponding depth information of the first and last skeleton points in the skeleton point set to be detected are obtained respectively. A spherical model is then constructed using the position information and corresponding depth information to obtain each first region and its corresponding second region. The center of the spherical model is the position of the first and last skeleton points to be detected, and the radius R of the spherical model is: X is a constant value, hipDepth is the depth information, and the target area will change in real time as the target user is closer to the camera; A columnar model is constructed to enclose each of the first regions and the corresponding second regions to obtain the target regions.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a skeletal point occlusion detection program stored in the memory and executable on the processor. When the processor executes the skeletal point occlusion detection program, it implements the steps of the skeletal point occlusion detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a skeletal point occlusion detection program, which, when executed by a processor, implements the steps of the skeletal point occlusion detection method as described in any one of claims 1-7.

Images