Acupuncture point positioning method and device, electronic equipment and storage medium

Abstract

The invention provides an acupuncture point positioning method and device, electronic equipment and a storage medium, and relates to the technical field of medical treatment, and the method comprises the steps: collecting a human body image of a target person through global RGBD camera equipment; when it is determined that the occlusion area exists, pixel information of the marked occlusion area is complemented based on the target image, and a complemented human body image is obtained; and inputting the complemented human body image into the acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model. According to the invention, when it is detected that there is a sheltered area in the human body image, the pixel information of the sheltered area can be complemented based on the local unsheltered image and / or the complete unsheltered human body image, and then acupoint positioning is carried out on the complemented human body image through the acupoint positioning model. Therefore, the original acupuncture point positioning model can be prevented from being influenced by the shielding area, and the accuracy of acupuncture point positioning is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technologies, and particularly to a method, a device, an electronic device, and a storage medium for acupoint positioning. Background Art

[0002] In the field of traditional Chinese medicine (TCM) medical treatment, acupoint positioning is a crucial technology. As the basis of the acupuncture, massage, and meridian acupoint theory, the accurate positioning of acupoints is of great significance for the research and development of TCM medical equipment and clinical applications.

[0003] In related technologies, professionals are usually invited to label acupoints on a certain number of sample human body images. Based on these labeled data, a machine learning algorithm is used to train an acupoint positioning model, and the trained acupoint positioning model is used to identify and position acupoints in the input human body image.

[0004] However, in the above-mentioned related technologies, when there is an occlusion area in the human body image that occludes a human body part, the trained acupoint positioning model cannot accurately identify and position the occluded acupoints, thereby reducing the accuracy of acupoint positioning. Summary of the Invention

[0005] The present invention provides a method, a device, an electronic device, and a storage medium for acupoint positioning to solve the defect of reducing the accuracy of acupoint positioning in the prior art.

[0006] The present invention provides a method for acupoint positioning, including the following steps.

[0007] Collect a human body image of a target person through a global RGBD camera device; Detect whether there is an occlusion area in the human body image that occludes a human body part; In the case of determining that there is an occlusion area, mark the occlusion area in the human body image; Based on a target image, complete the pixel information of the marked occlusion area in the human body image to obtain a completed human body image, where the target image includes a local unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; Input the completed human body image into an acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model.

[0008] According to the method for acupoint positioning provided by the present invention, the target image includes a local unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device; Completing the pixel information in the occluded area marked in the human body image based on the target image to obtain a completed human body image, including: Based on the relative pose relationship between the global RGBD camera device and the local RGBD camera device, registering each pixel information in the locally unoccluded image of the target human body part in the occluded area marked in the human body image to obtain the completed human body image, where the relative pose relationship is determined based on the first external parameter of the global RGBD camera device and the second external parameter of the local RGBD camera device.

[0009] According to a acupoint positioning method provided by the present invention, the method further includes: When the local RGBD camera device is a camera device with dynamically changing position and each time a acupoint positioning request is received, determining the second external parameter of the local RGBD camera device in real time; When the local RGBD camera device is a camera device with fixed position, obtaining the pre-stored second external parameter of the local RGBD camera device.

[0010] According to a acupoint positioning method provided by the present invention, the target image includes a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; Completing the pixel information in the occluded area marked in the human body image based on the target image to obtain a completed human body image, including: Based on the transformation relationship between the complete unoccluded human body image and the human body image, registering each pixel information of the target human body part in the complete unoccluded human body image in the occluded area marked in the human body image to obtain the completed human body image.

[0011] According to a acupoint positioning method provided by the present invention, the method further includes: Determining at least three identical key points of the target person in the complete unoccluded human body image and the human body image; Based on the positions of the identical key points in the complete unoccluded human body image and the positions of the identical key points in the human body image, determining the transformation relationship between the complete unoccluded human body image and the human body image.

[0012] According to a acupoint positioning method provided by the present invention, detecting whether there is an occluded area in the human body image that occludes the human body part, including: Dividing the human body image based on the human body part to obtain first sub-images corresponding to each human body part; Divide the complete unobstructed human body image based on human body parts to obtain second sub-images corresponding to each human body part; For each of the human body parts, compare the first sub-image corresponding to the human body part with the second sub-image corresponding to the human body part to obtain a comparison result; When the comparison result indicates that the first sub-image corresponding to the human body part and the second sub-image corresponding to the human body part are different, determine that there is an occlusion area in the human body image that occludes the human body part.

[0013] According to a point location method provided by the present invention, when it is determined that there is an occlusion area, marking the occlusion area includes: When it is determined that there is an occlusion area, set the pixel values of each pixel point in the occlusion area to a preset pixel value to mark the occlusion area.

[0014] The present invention also provides a point location device, including: An acquisition unit for acquiring a human body image of a target person through a global RGBD camera device; A detection unit for detecting whether there is an occlusion area in the human body image that occludes a human body part; A marking unit for marking the occlusion area in the human body image when it is determined that there is an occlusion area; A complementing unit for complementing the pixel information of the marked occlusion area in the human body image based on a target image to obtain a complemented human body image, where the target image includes a local unobstructed image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or a complete unobstructed human body image of the target person acquired by the global RGBD camera device before the human body image; A location unit for inputting the complemented human body image into a point location model to obtain a point location result output by the point location model.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the point location method as described in any one of the above when executing the computer program.

[0016] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the point location method as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the acupoint positioning method as described in any one of the above.

[0018] For the acupoint positioning method, device, electronic device and storage medium provided by the present invention, a human body image of a target person is collected by a global RGBD camera device, and it is detected whether there is an occlusion area that occludes a human body part in the human body image. In the case where it is determined that there is an occlusion area, the occlusion area in the human body image is marked, and based on a target image, the pixel information of the marked occlusion area in the human body image is complemented to obtain a complemented human body image. The target image includes a partial unoccluded image of a target human body part obtained by photographing the target human body part corresponding to the occlusion area by a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; the complemented human body image is input into an acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model. It can be seen that when the present invention detects an occlusion area that occludes a human body part in the human body image, it can complement the pixel information of the occlusion area based on the partial unoccluded image and / or the complete unoccluded human body image, and then perform acupoint positioning on the complemented human body image through the acupoint positioning model. Since the input to the acupoint positioning model is the complemented human body image, the original acupoint positioning model can be kept unaffected by the occlusion area, thereby improving the accuracy of acupoint positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is one of the flow diagrams of the acupoint positioning method provided by the embodiments of the present invention.

[0021] Figure 2 is the second flow diagram of the acupoint positioning method provided by the embodiments of the present invention.

[0022] Figure 3 is one of the overall flow diagrams of the acupoint positioning method provided by the embodiments of the present invention.

[0023] Figure 4 is the second overall flow diagram of the acupoint positioning method provided by the embodiments of the present invention.

[0024] Figure 5 is the structural diagram of the acupoint positioning device provided by the embodiments of the present invention.

[0025] Figure 6 It is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0026] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0027] The following Figures 1 - 4 describes the acupoint positioning method of the present invention. The execution subject of the acupoint positioning method may be an electronic device such as a terminal, a tablet computer, a computer, a robotic arm, etc., or an acupoint positioning device provided in the electronic device, and the acupoint positioning device may be implemented by software, hardware or a combination of both.

[0028] Figure 1 is one of the flowcharts of the acupoint positioning method provided by the embodiment of the present invention. As Figure 1 shown, the acupoint positioning method includes the following steps: Step 101: Collect a human body image of a target person through a global RGBD (Red Green Blue Depth) camera device.

[0029] Exemplarily, when it is necessary to position the acupoints of a target person, the global RGBD camera device may be controlled to take a real-time photo of the target person to obtain a real-time photo of the human body image of the target person.

[0030] It should be noted that the human body image captured by the global RGBD camera device is an image including RGB information and depth information, and a point cloud is generated based on the RGB information and the depth information. That is, the coordinates of each pixel point in the three-dimensional space are calculated based on the depth information, so that each pixel point corresponds to a point in the point cloud. The generated point cloud not only contains the three-dimensional shape information of the human body, but also can be combined with the RGB image to assign color information to each point, thereby generating a colored point cloud.

[0031] Step 102: Detect whether there is an occlusion area in the human body image that occludes the human body part.

[0032] Exemplarily, a complete unobstructed human body image can be captured in advance by a global RGBD camera device. When the human body image is obtained, the human body image can be compared with the complete unobstructed human body image to determine whether there are some areas in the human body image that are occluded. If the occluded areas include human body parts, then the areas are determined as the occluding areas that occlude the human body parts.

[0033] It should be noted that when it is determined that the information of multiple target areas in the human body image is different from the information of the corresponding areas in the complete unobstructed human body image, and each target area includes human body parts, then multiple occluding areas are determined. Subsequently, the information of each occluding area can be complemented. Alternatively, the human body image can be input into an occlusion detection model to detect whether there is occlusion in the human body through the occlusion detection model. Among them, the occlusion detection model is trained based on the following method: obtaining multiple sample human body images including sample human body occluded parts, and marking the occluded parts of the sample human bodies in each sample human body image to obtain the labels corresponding to each sample human body image. Inputting each sample human body image into the initial occlusion detection model to obtain the predicted occlusion results corresponding to each sample human body image output by the initial occlusion detection model. Constructing a loss function based on the predicted occlusion results and the corresponding labels, and iteratively optimizing the model parameters of the initial occlusion detection model based on the loss function until the convergence condition is reached, and finally obtaining the occlusion detection model. The present invention does not limit this.

[0034] Step 103, in the case where an occluding area is determined to exist, mark the occluding area in the human body image.

[0035] Exemplarily, when it is determined that there is an occluding area in the human body image, the detected occluding area can be marked on the human body image with a preset color or a preset pattern to remove the information of the occluding area.

[0036] Step 104, based on the target image, complement the pixel information of the marked occluding area in the human body image to obtain a complemented human body image. The target image includes a partial unobstructed image of the target human body part obtained by photographing the target human body part corresponding to the occluding area by a local RGBD camera device, and / or a complete unobstructed human body image of the target person collected by the global RGBD camera device before the human body image.

[0037] Among them, the pixel information includes RGB information and depth information. Then, complementing the pixel information of the occluding area includes complementing the RGB information and the depth information of the occluding area.

[0038] Exemplarily, after removing the information of the occluded area, it is necessary to complete the information of the occluded area to obtain a complete unoccluded human body image of the target person. Specifically, in the case of including a local RGBD camera device, the local RGBD camera device can be controlled to rotate or move to the target human body part for shooting to obtain a local unoccluded image of the target human body part, and then the pixel information of the occluded area marked in the human body image can be completed based on the local unoccluded image to obtain a completed human body image; in the case of not including a local RGBD camera device, the pixel information of the occluded area marked in the human body image can be completed based on the complete unoccluded human body image of the target person pre-shot by the global RGBD camera device to obtain a completed human body image. Or, in the case of including a local RGBD camera device, the pixel information of the occluded area marked in the human body image is completed based on the local unoccluded image to obtain a first completed human body image, and at the same time, the pixel information of the occluded area marked in the human body image is completed based on the complete unoccluded human body image of the target person shot by the global RGBD camera device to obtain a second completed human body image, and the first completed human body image and the second completed human body image are fused to obtain a completed human body image. The specific fusion technology needs to consider various factors, such as color, texture, lighting, etc., to ensure that the obtained completed human body image after fusion has a natural transition and no obvious seams.

[0039] Step 105: Input the completed human body image into the acupoint positioning model to obtain the acupoint positioning result output by the acupoint positioning model.

[0040] Exemplarily, when obtaining the completed human body image, the completed human body image is input into the acupoint positioning model, and the acupoint positioning model analyzes the features of the completed human body image to obtain the acupoint positioning result in the completed human body image.

[0041] It should be noted that the specific training method of the acupoint positioning model is as follows: Obtain a plurality of unoccluded sample images including sample persons, mark the acupoints of the sample persons in the unoccluded sample images to obtain the acupoint labels corresponding to the unoccluded sample images, input the unoccluded sample images into the initial acupoint positioning model to obtain the predicted acupoint positioning result output by the initial acupoint positioning model, construct a loss function based on the predicted acupoint positioning result and the acupoint labels corresponding to the unoccluded sample images, and iteratively optimize the model parameters of the initial acupoint positioning model based on the loss function until the convergence condition is reached to obtain the acupoint positioning model.

[0042] It should be noted that the initial acupoint positioning model can be a deep neural network (DNN), a convolutional neural network (CNN), etc. The embodiments of the present invention do not limit the initial acupoint positioning model.

[0043] It should be noted that the present invention can be applied to any scenario for positioning the acupoints of a target user. For example, when massaging a target user and it is necessary to locate the acupoints of the target user, it can be applied to manual massage or an automatic massage scenario using a robotic arm, etc. The present invention does not limit this.

[0044] It should be noted that the global RGBD camera device in the present invention can also be replaced by a global RGB camera device, and the local RGBD camera device can also be a local RGB camera device. When including a global RGB camera device and a local RGB camera device, the present invention can effectively complete the 2D human body occlusion information; when including a global RGBD camera device and a local RGBD camera device, the present invention can effectively complete the 3D human body occlusion information.

[0045] The acupoint positioning method provided by the present invention collects a human body image of a target person through a global RGBD camera device, detects whether there is an occlusion area that occludes a human body part in the human body image, and when it is determined that there is an occlusion area, marks the occlusion area in the human body image, and based on a target image, completes the pixel information of the marked occlusion area in the human body image to obtain a completed human body image. The target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; inputs the completed human body image into the acupoint positioning model to obtain the acupoint positioning result output by the acupoint positioning model. It can be seen that when the present invention detects that there is an occlusion area that occludes a human body part in the human body image, it can complete the pixel information of the occlusion area based on the partial unoccluded image and / or the complete unoccluded human body image, and then perform acupoint positioning on the completed human body image through the acupoint positioning model. Since the input to the acupoint positioning model is the completed human body image, the original acupoint positioning model can be kept unaffected by the occlusion area, thereby improving the accuracy of acupoint positioning. In addition, since the input to the acupoint positioning model is the completed human body image, when constructing the acupoint positioning model, the occlusion scenario does not need to be considered, and the focus can be on improving and optimizing the acupoint positioning performance of the unoccluded human body, thereby simplifying the construction process of the acupoint positioning model, avoiding simulating or enumerating countless occlusion scenarios, and circumventing the dependence on a large amount of occlusion data.

[0046] In one embodiment, the target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occluded area through a local RGBD camera device; step 104 above completes the pixel information of the occluded area marked in the human body image based on the target image to obtain a completed human body image, which can be specifically implemented in the following manner: Based on the relative pose relationship between the global RGBD camera device and the local RGBD camera device, register the pixel information in the partial unoccluded image of the target human body part in the occluded area marked in the human body image to obtain the completed human body image, and the relative pose relationship is determined based on the first extrinsic parameters of the global RGBD camera device and the second extrinsic parameters of the local RGBD camera device.

[0047] Exemplarily, the first extrinsic parameters include the extrinsic parameter matrix of the global RGBD camera device where represents the first rotation matrix, represents the first translation vector; the second extrinsic parameters include the extrinsic parameter matrix of the local RGBD camera device where represents the second rotation matrix, represents the second translation vector, calculate the relative rotation matrix based on the following formula (1) and calculate the relative translation vector based on the following formula (2) : (1) (2) where is the transpose of the first rotation matrix of the global RGBD camera device.

[0048] Then, represent the relative pose relationship between the global RGBD camera device and the local RGBD camera device based on the following formula (3) : (3) After obtaining the relative pose relationship , use the calculated relative pose relationship to convert the pixel information of each pixel point in the partial unoccluded image to the coordinate system of the global RGBD camera device, and according to the converted coordinates, map the pixel information of the partial unoccluded image to the corresponding occluded area in the human body image. Specifically, interpolation methods (such as bilinear interpolation) can be used to process the pixel values of non-integer coordinates, which will not be elaborated in this invention.

[0049] In this embodiment, when the target image includes a locally unoccluded image of the target human body part corresponding to the occluded area captured by the local RGBD camera device, the pixel information in the locally unoccluded image of the target human body part can be registered in the occluded area marked in the human body image based on the relative pose relationship between the global RGBD camera device and the local RGBD camera device, so as to obtain a completed human body image, realizing the automatic completion of the occluded area. Moreover, since the completion is based on the locally unoccluded image of the target human body part, the accuracy of the occluded area completion is improved, and further the accuracy of acupoint location is improved.

[0050] In one embodiment, the acupoint location method further includes the following steps: When the local RGBD camera device is a camera device with dynamically changing position, and each time a request for acupoint location is received, the second external parameters of the local RGBD camera device are determined in real time; When the local RGBD camera device is a camera device with a fixed position, the pre-stored second external parameters of the local RGBD camera device are obtained.

[0051] Exemplarily, when the local RGBD camera device is a camera device with dynamically changing position, since the position and orientation of the local RGBD camera device change over time, each time a request for acupoint location is received, the second external parameters of the local RGBD camera device need to be updated in real time; when the local RGBD camera device is a camera device with a fixed position, since the position and orientation of the local RGBD camera device remain unchanged, only the pre-stored external parameters need to be used as the second external parameters of the local RGBD camera device.

[0052] It should be noted that when the local RGBD camera device is a camera device with dynamically changing position, for example, the local RGBD camera device is set on a robotic arm, and the position of the local RGBD camera device changes with the movement of the robotic arm. The spatial position of the end of the robotic arm can be obtained in real time through the robotic arm control software or interface. Based on the relative position relationship between the end of the robotic arm and the local RGBD camera device installed on the robotic arm, the spatial position of the local RGBD camera device can be calculated in real time. When the positions of both the local RGBD camera device and the global RGBD camera device are fixed positions, the position of the local RGBD camera device can be obtained in advance based on the positioning module, or the position of the local RGBD camera device can be manually input, and the position of the local RGBD camera device is determined as the spatial position of the local RGBD camera device. The present invention does not limit this.

[0053] In one embodiment, the target image includes a complete unoccluded body image of the target person captured by the global RGBD camera device before the body image; step 104 above complements the pixel information of the occluded area marked in the body image based on the target image to obtain a complemented body image, which can be specifically implemented in the following manner: Based on the transformation relationship between the complete unoccluded body image and the body image, register the pixel information of each target body part in the complete unoccluded body image in the occluded area marked in the body image to obtain the complemented body image.

[0054] Exemplarily, the transformation relationship can be a mathematical formula or a mapping table, which is used to register the pixel information of each target body part in the complete unoccluded body image at the corresponding position in the occluded area marked in the body image, and finally obtain the complemented body image.

[0055] In this embodiment, based on the complete unoccluded body image, and based on the transformation relationship between the complete unoccluded body image and the body image, register the pixel information of each target body part in the complete unoccluded body image in the occluded area marked in the body image to obtain the complemented body image, realizing the automatic complement of the occluded area information in the body image, and further improving the efficiency and accuracy of acupoint positioning.

[0056] In one embodiment, Figure 2 is the second flowchart of the acupoint positioning method provided by the embodiment of the present invention. As Figure 2 shown, the acupoint positioning method further includes the following steps: Step 201, determine at least three identical key points of the target person in the complete unoccluded body image and the body image.

[0057] Exemplarily, the key points can be joints of the target person (for example, shoulder joints, elbow joints, etc.), facial features (such as the corners of the eyes, the corners of the mouth, etc.) or other significant feature points. Specifically, use a human joint point or key point detection algorithm to identify all the key points of the target person in the complete unoccluded body image, and identify all the key points of the target person in the body image, and determine the identical key points between all the key points identified from the complete unoccluded body image and all the key points identified from the body image. Among them, the human joint point or key point detection algorithm can use a human pose estimation algorithm, such as MMpose (an open-source toolbox for pose estimation based on Pytorch, a member of the OpenMMLab project, which includes rich algorithms such as 2D multi-person pose estimation, 2D hand pose estimation, 2D face key point detection, 133-key-point full-body human pose estimation, etc., as well as related components and modules), etc. The present invention does not limit this.

[0058] Step 202: Based on the positions of the same key points in the complete unoccluded human body image and the positions of the same key points in the human body image, determine the transformation relationship between the target person in the complete unoccluded human body image and the target person in the human body image.

[0059] Exemplarily, the same key points (such as the left shoulder, right shoulder, left knee, right knee, etc.) are detected in the complete unoccluded human body image and the partially occluded human body image respectively. These algorithms will output the two-dimensional coordinates of each key point in the image. Since they are the same key points, they can be directly matched one by one according to the type of the key points (such as the left shoulder matching the left shoulder, the right knee matching the right knee). For each pair of matched key points, calculate the displacement vector (coordinate difference) of the matched key points in the complete unoccluded human body image and the partially occluded human body image, and calculate the average displacement vector of all pairs of matched key points as the overall translation vector; select two pairs of key points (such as the left shoulder and the right shoulder), calculate the vectors of these two pairs of key points in the complete unoccluded human body image and the partially occluded human body image, calculate the rotation angle through the vector angle formula, construct a rotation matrix according to the calculated rotation angle, and determine the translation vector and the rotation matrix as the transformation relationship between the target person in the complete unoccluded human body image and the target person in the human body image.

[0060] Step 203: Based on the transformation relationship between the complete unoccluded human body image and the human body image, register the pixel information of the target human body part in the complete unoccluded human body image in the occluded area marked in the human body image to obtain the complemented human body image.

[0061] In this embodiment, at least three same key points of the target person in the complete unoccluded human body image and the human body image are determined. Based on the positions of the same key points in the complete unoccluded human body image and the positions of the same key points in the human body image, the transformation relationship between the complete unoccluded human body image and the human body image can be automatically determined, which is convenient for subsequently registering the pixel information of the target human body part in the complete unoccluded human body image in the occluded area marked in the human body image based on this pixel conversion relationship to obtain the complemented human body image, realizing the automatic complement of the occluded area information in the human body image, and further improving the efficiency and accuracy of acupoint positioning.

[0062] In one embodiment, to detect whether there is an occluded area in the human body image that occludes the human body part in the above step 102, it can be specifically implemented in the following manner: Divide the human body image based on human body parts to obtain first sub-images corresponding to the respective human body parts; divide the complete unoccluded human body image based on human body parts to obtain second sub-images corresponding to the respective human body parts; for each of the human body parts, compare the first sub-image corresponding to the human body part with the second sub-image corresponding to the human body part to obtain a comparison result; in the case where the comparison result indicates that the first sub-image corresponding to the human body part and the second sub-image corresponding to the human body part are different, determine that there is an occlusion area in the human body image that occludes the human body part.

[0063] Exemplarily, when a human body image is obtained, the human body image can be divided based on human body parts, where the human body parts include the head, torso, upper limbs, lower limbs, etc., to obtain first sub-images corresponding to the respective human body parts; and the complete unoccluded human body image is also divided based on human body parts to obtain second sub-images corresponding to the respective human body parts. For each human body part, extract a first feature from the first sub-image corresponding to the human body part and extract a second feature from the second sub-image corresponding to the human body part. The first feature and the second feature can be edges, textures, shapes, colors, etc., and these features can reflect the uniqueness and differences of the human body parts. Compare the extracted first feature and second feature between the first sub-image and the second sub-image corresponding to the human body part. The comparison process includes steps such as feature point matching and similarity calculation of feature vectors. By calculating the similarity or distance between the features, the similarity degree of the first sub-image and the second sub-image corresponding to the human body part in the human body part can be evaluated. According to the comparison result, if there are significant differences between the first sub-image and the second sub-image in the human body part (i.e., the comparison result indicates that the first sub-image and the second sub-image corresponding to the human body part are different), it can be considered that there is an occlusion area in the human body image that occludes the human body part. Using the same method, it can be determined whether other human body parts in the human body image are occluded.

[0064] In this embodiment, the human body image and the complete unoccluded human body image are divided based on human body parts, and it is determined whether there is an occlusion area in the human body image that occludes each human body part, and the occlusion area is further subdivided, which further improves the accuracy of determining the occlusion area.

[0065] In one embodiment, in the case where it is determined that there is an occlusion area in step 103 above, mark the occlusion area, which can be specifically implemented in the following manner: In the case where it is determined that there is an occlusion area, set the pixel values of each pixel point in the occlusion area to a preset pixel value to mark the occlusion area.

[0066] Among them, the pixel values of each pixel point include RGB values and depth values, and the present invention does not limit this.

[0067] Exemplarily, a preset pixel value for characterizing the pixel points of the occluded area is pre-stored. The preset pixel value can be set based on the specific scenario. When it is determined that there is an occluded area, the pixel values of each pixel point in the occluded area are set to the preset pixel value, so as to distinguish the occluded area and the non-occluded area through the preset pixel value.

[0068] In this embodiment, when it is determined that there is an occluded area, the pixel values of each pixel point in the occluded area are set to the preset pixel value to mark the occluded area, which is convenient for the electronic device to distinguish the occluded area and the non-occluded area in the human body image, and further improves the accuracy of information completion.

[0069] Figure 3 is one of the overall flowcharts of the acupoint positioning method provided by the embodiments of the present invention. As Figure 3 shown, when the target image includes a partial unoccluded image of the target human body part corresponding to the occluded area captured by the local RGBD camera device, the relative pose relationship between the global RGBD camera device and the local RGBD camera device is determined based on the first external parameter of the global RGBD camera device and the second external parameter of the local RGBD camera device. In application, the human body image of the target person is collected by the global RGBD camera device, and it is detected whether there is an occluded area that occludes the human body part in the human body image. When it is determined that there is an occluded area, the occluded area in the human body image is marked. Based on the relative pose relationship between the global RGBD camera device and the local RGBD camera device, the information in the partial unoccluded image of the target human body part is registered in the occluded area marked in the human body image to obtain a complemented human body image. The complemented human body image is input into the acupoint positioning model to obtain the acupoint positioning result output by the acupoint positioning model.

[0070] It should be noted that Figure 3 the local RGBD camera device in

[0071] Figure 4 is the second overall flowchart of the acupoint positioning method provided by the embodiments of the present invention. As Figure 4As shown, when the target image includes a complete unoccluded human body image of the target person collected by a global RGBD camera device before the human body image, at least three identical key points of the target person in the complete unoccluded human body image and the human body image are determined; based on the positions of the identical key points in the complete unoccluded human body image and the positions of the identical key points in the human body image, the transformation relationship between the complete unoccluded human body image and the human body image is determined. In application, a human body image of the target person is collected by a global RGBD camera device, and whether there is an occlusion area that occludes a human body part in the human body image is detected. When it is determined that there is an occlusion area, the occlusion area in the human body image is marked. Based on the transformation relationship between the complete unoccluded human body image and the human body image, the pixel information of each target human body part in the complete unoccluded human body image is registered in the marked occlusion area in the human body image to obtain a complemented human body image. The complemented human body image is input into an acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model.

[0072] The acupoint positioning device provided by the present invention is described below. The acupoint positioning device described below can be correspondingly referred to the acupoint positioning method described above.

[0073] Figure 5 is a schematic structural diagram of the acupoint positioning device provided by an embodiment of the present invention. As Figure 5 shown, the acupoint positioning device 500 includes a collection unit 501, a detection unit 502, a marking unit 503, a complementing unit 504, and a positioning unit 505; where: The collection unit 501 is configured to collect a human body image of a target person through a global RGBD camera device; The detection unit 502 is configured to detect whether there is an occlusion area that occludes a human body part in the human body image; The marking unit 503 is configured to mark the occlusion area in the human body image when it is determined that there is an occlusion area; The complementing unit 504 is configured to complement the pixel information of the marked occlusion area in the human body image based on a target image, and obtain a complemented human body image. The target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; The positioning unit 505 is configured to input the complemented human body image into an acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model.

[0074] The acupoint positioning device provided by the present invention captures a human body image of a target person through a global RGBD camera device, detects whether there is an occlusion area that occludes a human body part in the human body image, and in the case of determining that there is an occlusion area, marks the occlusion area in the human body image, and based on a target image, completes the pixel information of the marked occlusion area in the human body image to obtain a completed human body image. The target image includes a local unoccluded image of a target human body part corresponding to the occlusion area captured by a local RGBD camera device, and / or a complete unoccluded human body image of the target person captured by the global RGBD camera device before the human body image; the completed human body image is input into an acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model. It can be seen that when the present invention detects an occlusion area that occludes a human body part in the human body image, it can complete the pixel information of the occlusion area based on the local unoccluded image and / or the complete unoccluded human body image, and then perform acupoint positioning on the completed human body image through the acupoint positioning model. Since the input to the acupoint positioning model is the completed human body image, the original acupoint positioning model can be kept unaffected by the occlusion area, thereby improving the accuracy of acupoint positioning.

[0075] Based on any of the above embodiments, the target image includes a local unoccluded image of a target human body part corresponding to the occlusion area captured by a local RGBD camera device; the completion unit 504 is specifically configured to: Based on the relative pose relationship between the global RGBD camera device and the local RGBD camera device, register each pixel information in the local unoccluded image of the target human body part in the occlusion area marked in the human body image to obtain the completed human body image, and the relative pose relationship is determined based on the first external parameter of the global RGBD camera device and the second external parameter of the local RGBD camera device.

[0076] Based on any of the above embodiments, the acupoint positioning device 500 further includes: A first determination unit, configured to, when the local RGBD camera device is a camera device with a dynamically changing position and each time an acupoint positioning request is received, determine the second external parameter of the local RGBD camera device in real time; An acquisition unit, configured to, when the local RGBD camera device is a camera device with a fixed position, acquire the pre-stored second external parameter of the local RGBD camera device.

[0077] Based on any of the above embodiments, the target image includes a complete unoccluded human body image of the target person captured by the global RGBD camera device before the human body image; the completion unit 504 is further specifically configured to: Based on the transformation relationship between the complete unoccluded human body image and the human body image, register the pixel information of each pixel of the target human body part in the complete unoccluded human body image in the occluded area marked in the human body image, and obtain the complemented human body image.

[0078] Based on any of the above embodiments, the acupoint positioning device 500 further includes: A second determination unit, configured to determine at least three identical key points of the target person in the complete unoccluded human body image and the human body image; A third determination unit, configured to determine the transformation relationship between the complete unoccluded human body image and the human body image based on the positions of the respective identical key points in the complete unoccluded human body image and the positions of the respective identical key points in the human body image.

[0079] Based on any of the above embodiments, the detection unit 502 is specifically configured to: Divide the human body image based on human body parts to obtain first sub-images corresponding to the respective human body parts; Divide the complete unoccluded human body image based on human body parts to obtain second sub-images corresponding to the respective human body parts; For each of the human body parts, compare the first sub-image corresponding to the human body part and the second sub-image corresponding to the human body part to obtain a comparison result; In the case where the comparison result indicates that the first sub-image corresponding to the human body part and the second sub-image corresponding to the human body part are different, determine that there is an occluded area in the human body image that occludes the human body part.

[0080] Based on any of the above embodiments, the marking unit 503 is specifically configured to: In the case where an occluded area is determined, set the pixel value of each pixel point in the occluded area to a preset pixel value to mark the occluded area.

[0081] Figure 6 It is a schematic physical structure diagram of an electronic device provided by an embodiment of the present invention. As Figure 6 shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute the acupoint positioning method, and the method includes: collecting a human body image of a target person through a global RGBD camera device; Detect whether there is an occlusion area in the human body image that occludes the human body part; In the case of determining that there is an occlusion area, mark the occlusion area in the human body image; Based on the target image, complete the pixel information of the marked occlusion area in the human body image to obtain a completed human body image. The target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; Input the completed human body image into the acupoint positioning model to obtain the acupoint positioning result output by the acupoint positioning model.

[0082] In addition, when the logical instructions in the above-mentioned memory 630 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0083] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the acupoint positioning method provided by the above-mentioned various methods. The method includes: collecting a human body image of a target person through a global RGBD camera device; Detect whether there is an occlusion area in the human body image that occludes the human body part; In the case of determining that there is an occlusion area, mark the occlusion area in the human body image; Based on the target image, complete the pixel information of the occluded area marked in the human body image to obtain a completed human body image. The target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occluded area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; Input the completed human body image into the acupoint positioning model to obtain the acupoint positioning result output by the acupoint positioning model.

[0084] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the acupoint positioning method provided by the above-mentioned various methods. The method includes: collecting a human body image of a target person through a global RGBD camera device; Detect whether there is an occluded area in the human body image that occludes the human body part; In the case of determining that there is an occluded area, mark the occluded area in the human body image; Based on the target image, complete the pixel information of the occluded area marked in the human body image to obtain a completed human body image. The target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occluded area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; Input the completed human body image into the acupoint positioning model to obtain the acupoint positioning result output by the acupoint positioning model.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0086] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for acupoint location, characterized in that, Including: Collecting a human body image of a target person through a global RGBD camera device; Detecting whether there is an occlusion area in the human body image that occludes a human body part; When it is determined that there is an occlusion area, marking the occlusion area in the human body image; Based on a target image, complementing pixel information of the marked occlusion area in the human body image to obtain a complemented human body image, where the target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; Inputting the complemented human body image into an acupoint location model to obtain an acupoint location result output by the acupoint location model.

2. The acupoint positioning method according to claim 1, wherein The target image includes a partial unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device; The complementing pixel information of the marked occlusion area in the human body image based on the target image to obtain a complemented human body image includes: Based on the relative pose relationship between the global RGBD camera device and the local RGBD camera device, registering each pixel information in the partial unoccluded image of the target human body part in the marked occlusion area in the human body image to obtain the complemented human body image, where the relative pose relationship is determined based on the first external parameter of the global RGBD camera device and the second external parameter of the local RGBD camera device.

3. The acupoint positioning method according to claim 2, wherein, The method further includes: When the local RGBD camera device is a camera device with a dynamically changing position, and in each case of receiving an acupoint location request, determining the second external parameter of the local RGBD camera device in real time; When the local RGBD camera device is a camera device with a fixed position, obtaining the pre-stored second external parameter of the local RGBD camera device.

4. The acupoint location method according to claim 1, wherein, The target image includes a complete unoccluded human body image of the target person collected by the global RGBD camera device before the human body image; The complementing pixel information of the marked occlusion area in the human body image based on the target image to obtain a complemented human body image includes: Based on the transformation relationship between the complete unoccluded human body image and the human body image, registering each pixel information of the target human body part in the complete unoccluded human body image in the marked occlusion area in the human body image to obtain the complemented human body image.

5. The acupoint location method according to claim 4, characterized in that The method further includes: Determining at least three same key points of the target person in the complete unoccluded human body image and the human body image; Based on the positions of each of the same key points in the complete unoccluded human body image and the positions of each of the same key points in the human body image, determining the transformation relationship between the complete unoccluded human body image and the human body image.

6. The acupoint location method according to claim 1, wherein, The detecting whether there is an occlusion area in the human body image that occludes a human body part includes: Divide the human body image based on human body parts to obtain first sub-images corresponding to the respective human body parts; Divide the complete unoccluded human body image based on human body parts to obtain second sub-images corresponding to the respective human body parts; For each of the human body parts, compare the first sub-image corresponding to the human body part and the second sub-image corresponding to the human body part to obtain a comparison result; In the case where the comparison result indicates that the first sub-image corresponding to the human body part and the second sub-image corresponding to the human body part are different, determine that there is an occlusion area in the human body image that occludes the human body part.

7. The acupoint location method according to any one of claims 1-6, characterized in that The marking of the occlusion area in the case of determining the existence of the occlusion area includes: In the case of determining the existence of the occlusion area, set the pixel values of the respective pixel points in the occlusion area to a preset pixel value to mark the occlusion area.

8. An acupoint positioning device, characterized in that Includes: An acquisition unit for acquiring a human body image of a target person through a global RGBD camera device; A detection unit for detecting whether there is an occlusion area in the human body image that occludes a human body part; A marking unit for marking the occlusion area in the human body image in the case of determining the existence of the occlusion area; A completion unit for completing the pixel information of the occlusion area marked in the human body image based on a target image to obtain a completed human body image, where the target image includes a locally unoccluded image of the target human body part obtained by photographing the target human body part corresponding to the occlusion area through a local RGBD camera device, and / or, a complete unoccluded human body image of the target person acquired by the global RGBD camera device before the human body image; A positioning unit for inputting the completed human body image into an acupoint positioning model to obtain an acupoint positioning result output by the acupoint positioning model.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the acupoint positioning method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the acupoint positioning method according to any one of claims 1 to 7.

Images