An acupoint positioning method based on whole body key point recognition and instance segmentation

Abstract

The application discloses an acupoint positioning method based on whole body key point recognition and instance segmentation, and belongs to the technical field of medical image processing. Based on the comprehensive solution of deep learning and digital image processing technology, the accuracy and efficiency of traditional Chinese medicine acupoint positioning theory are improved. The method adopts a targeted strategy to locate various whole body acupoints. Specifically, the whole body acupoints are divided into two categories: the first category is acupoints that can be positioned by key point position and using traditional Chinese medicine positioning method, and the second category is acupoints that have no obvious image features but can be positioned by part contour information. Different positioning methods are adopted according to different categories of acupoints. The application also adopts augmented reality technology to superimpose acupoint information on real body images, making acupoint positioning more visual. After verification, the method can significantly improve the positioning accuracy and efficiency of whole body acupoints, and provides a new and efficient acupoint positioning solution.

Description

Technical Field

[0001] This application belongs to the field of medical image processing technology, specifically relating to an acupoint location method based on whole-body key point recognition and instance segmentation. Background Technology

[0002] With the increasing emphasis placed on Traditional Chinese Medicine (TCM) by the state, medical treatments such as acupuncture are gradually gaining popularity and recognition, and are increasingly spreading globally. For doctors practicing acupuncture in clinical practice, long-term experience has led to accurate mastery of acupoint location on the body surface, enabling them to quickly and accurately locate specific acupoints. However, for beginners, the general public, and those engaged in TCM healthcare, acupoint location is not easy to master, and accuracy is even lower. Acupoint location is a crucial aspect of TCM therapy, and the accuracy of acupoint identification directly affects the final outcome. However, the human body has numerous acupoints: 362 main acupoints and 46 extra acupoints, totaling 408 acupoints, each with different functions in TCM. Therefore, accurately identifying acupoints is particularly difficult for non-professionals.

[0003] Currently, people generally locate acupoints by referring to traditional Chinese medicine books and illustrations of acupoints. However, these methods cannot adequately address individual differences and inevitably lead to inaccuracies in acupoint location.

[0004] Given the increasing demand for convenient and accurate acupoint location, it is extremely important to develop acupoint location technology based on key point detection and instance segmentation. Summary of the Invention

[0005] The purpose of this application is to provide an acupoint location method based on whole-body key point recognition and instance segmentation, which solves the problem in the prior art that the selection of acupoints is somewhat arbitrary and inaccurate due to differences in patients' height, weight, etc., and the physician's reliance on personal experience. This can solve at least one of the technical problems involved in the background art.

[0006] To solve the above-mentioned technical problems, the technical solution of this application is as follows:

[0007] In a first aspect, embodiments of this application provide a method for acupoint location based on whole-body key point recognition and instance segmentation, including:

[0008] Step 1: Read the video stream from the AR device, extract the image frames from the video stream, and have the part detection module detect the part categories of the image frames and output the determined part categories.

[0009] Step 2: Based on the characteristic types of acupoints in different parts of the human body, proceed to Step 3 for acupoints with significant characteristics and Step 4 for acupoints with insignificant characteristics.

[0010] Step 3: The key point detection module performs key point detection processing on the acupoints with significant features to obtain the key point coordinates. After accumulating the corresponding acupoint offset, the acupoint coordinates are obtained. Finally, the acupoint location and name are output.

[0011] Step 4: The instance segmentation module outputs the outline of the part, then finds features based on the proportion of the outline, accumulates the corresponding acupoint offset to obtain the acupoint coordinates, and finally outputs the acupoint location and name.

[0012] Step 5: Use augmented reality technology to overlay the acupoint locations and names output in Steps 3 and 4 onto a real human body image to achieve accurate acupoint positioning.

[0013] Optionally, the part detection module includes an arm processing component, a face processing component, a torso processing component, and a leg processing component.

[0014] Optionally, the part detection module is trained through the following steps:

[0015] A target detection dataset is constructed based on images and names of various parts of the human body;

[0016] The part detection module was trained using the constructed object detection dataset.

[0017] Optionally, the part detection module is implemented based on Faster R-CNN. It uses shared convolutional layers to extract features from images of various parts of the human body to obtain part feature maps. The obtained part feature maps are then fed into a region proposal network. The region proposal network generates bounding boxes to be detected, i.e., the locations of the specified regions of interest, and performs a first correction on the bounding boxes of the regions of interest. The region pooling layer selects the features corresponding to each region of interest on the part feature map based on the output of the region proposal network and sets the dimension to a fixed value. A fully connected layer is used to classify the bounding boxes and perform a second correction on the target bounding boxes.

[0018] Optionally, the backbone network of the key point detection module incorporates the concept of a residual ladder network to fuse intra-layer features, thereby obtaining a refined local representation. Specifically, this includes:

[0019] The residual staircase network replaces the downsampling convolutional layers in the backbone network. The residual staircase network consists of residual staircase blocks, which are connected together by dense element-wise summation. The residual staircase blocks first divide the features into four branches, and then process each branch using a 1×1 convolution. The output of the 1×1 convolution of each branch is further processed by a 3×3 convolution. Finally, the output features of the four branches are concatenated, and then the features are fused by a 1×1 convolution and connected using identity.

[0020] Optionally, the key point detection module is trained through the following steps:

[0021] Construct a keypoint localization dataset based on human keypoint images and keypoint names;

[0022] The keypoint detection module is trained using the constructed keypoint localization dataset.

[0023] Optionally, the key point detection module is implemented using HRNet, which is an improved version of the residual ladder network. It adopts a multi-resolution structure design, extracts key point features from human key point images at different resolutions by using multiple branch networks, fuses the key point features using a fusion layer to achieve high-resolution key point feature extraction, and uses a key point regression layer to predict the key point position to obtain the key point coordinates.

[0024] Optionally, the instance segmentation module is trained through the following steps:

[0025] An instance segmentation dataset is constructed based on human keypoint images, pixel-by-pixel annotations, and instance-level identifiers.

[0026] The instance segmentation module is trained using the constructed instance segmentation dataset.

[0027] Optionally, the instance segmentation module uses key point detection information as the alignment method for instance segmentation, namely, the alignment method based on human skeleton posture. By using the information contained in the human skeleton posture, the alignment method based on human skeleton posture straightens out various human movements and separates overlapping people.

[0028] Optionally, the instance segmentation module is implemented using a Mask R-CNN model based on the alignment method of human skeleton pose. First, the feature extractor extracts the features of the images of each part of the human body. Then, the candidate region generator generates candidate target regions. Next, the classifier classifies the candidate target regions, the regressor locates the candidate target regions, and finally, the segmenter segments the candidate target regions to obtain the outline of the human body parts. Then, the acupoint coordinates are obtained by searching according to the outline path and combining the key point coordinates.

[0029] In a second aspect, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0030] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0031] The beneficial effects of this application are as follows:

[0032] 1. This application presents a comprehensive solution based on deep learning and digital image processing technologies to improve the accuracy and efficiency of traditional Chinese medicine acupoint location theory;

[0033] 2. Targeted localization methods are adopted for different categories of acupoints throughout the body. In addition to considering key point information, contour information and other relevant information are further taken into account. Furthermore, this method divides all acupoints into multiple categories and proposes a localization method for each category, rather than proposing a different localization method for each acupoint. This approach ensures localization accuracy while reducing the workload of system development.

[0034] 3. Augmented reality technology is also used to overlay acupoint information in real-world scenarios, making it more intuitive and convenient, reducing the time required for manual acupoint location and improving work efficiency. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram illustrating the implementation steps of the positioning method provided in the embodiments of this application;

[0037] Figure 2 This is a schematic diagram of the HRNet model based on the residual ladder network improvement provided in the embodiments of this application;

[0038] Figure 3 (a) to (d) are schematic diagrams of key points of the human body provided in the embodiments of this application;

[0039] Figure 4 This is a schematic diagram of human body part segmentation provided in an embodiment of this application;

[0040] Figure 5 This is a structural framework diagram of the electronic device provided in the embodiments of this application;

[0041] Figure 6 This is a hardware structure diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0044] The acupoint location method based on whole-body key point recognition and instance segmentation provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0045] Please see Figure 1 This application provides an embodiment of an acupoint location method based on whole-body key point recognition and instance segmentation, comprising:

[0046] Step 1: Read the video stream from the AR device, extract the image frames from the video stream, and have the part detection module detect the part categories of the image frames and output the determined part categories.

[0047] Step 2: Based on the characteristic types of acupoints in different parts of the human body, proceed to Step 3 for acupoints with significant characteristics and Step 4 for acupoints with insignificant characteristics.

[0048] Step 3: The key point detection module performs key point detection processing on the acupoints with significant features to obtain the key point coordinates. After accumulating the corresponding acupoint offset, the acupoint coordinates are obtained. Finally, the acupoint location and name are output.

[0049] Step 4: The instance segmentation module outputs the outline of the part, then finds features based on the proportion of the outline, accumulates the corresponding acupoint offset to obtain the acupoint coordinates, and finally outputs the acupoint location and name.

[0050] Step 5: Use augmented reality technology to overlay the acupoint locations and names output in Steps 3 and 4 onto a real human body image to achieve accurate acupoint positioning.

[0051] In step one, the body part detection module includes an arm processing component, a face processing component, a torso processing component, and a leg processing component. In this way, the body part detection module can send the pictures of different human body parts in the identified current video frame into different processing flows, specifically including:

[0052] The acupoints of the Lung Meridian of Hand-Taiyin, Large Intestine Meridian of Hand-Yangming, Heart Meridian of Hand-Shaoyin, Small Intestine Meridian of Hand-Taiyang, Pericardium Meridian of Hand-Jueyin, and Sanjiao Meridian of Hand-Shaoyang are located by the arm processing component. The acupoints on the Stomach Meridian of Foot-Yangming, Spleen Meridian of Foot-Taiyin, Bladder Meridian of Foot-Taiyang, Kidney Meridian of Foot-Shaoyin, Gallbladder Meridian of Foot-Shaoyang, and Liver Meridian of Foot-Jueyin are located by the face processing component, torso processing component, and leg processing component. The acupoints of the Conception Vessel and Governor Vessel are located by the torso processing component.

[0053] The body part detection module is trained through the following steps: constructing a target detection data set according to the images of each human body part and the body part names; using the constructed target detection data set to train the body part detection module.

[0054] The body part detection module is implemented based on Faster R-CNN. It uses a shared convolutional layer to extract features for the images of each human body part, obtains a body part feature map, and sends the obtained body part feature map into the Region Proposal Network. The Region Proposal Network generates detection boxes to be detected, that is, specifies the positions of regions of interest, and makes a first correction to the bounding boxes of the regions of interest. The Region of Interest Pooling layer selects the features corresponding to each region of interest on the body part feature map according to the output of the Region Proposal Network and sets the dimension to a fixed value. A fully connected layer is used to classify the bounding boxes and make a second correction to the target bounding boxes. After that, the detection and recognition of body part categories can be achieved.

[0055] In step three, the key point detection module processes the acupoints with significant features, specifically including:

[0056] Chengqi acupoint is located on the face, directly below the pupil, between the eyeball and the edge of the lower eye socket. Among them, the feature of the pupil is significant; Taiyi acupoint is located in the upper abdomen of the human body, 2 cun above the umbilicus and 2 cun away from the anterior midline. Among them, the feature of the umbilicus is significant; Zhongchong acupoint is located at the center of the tip of the terminal phalanx of the middle finger of the human body. Among them, the feature of the tip of the terminal phalanx of the middle finger is significant.

[0057] In step four, for the acupoints with insignificant features, the instance segmentation module is used, specifically including:

[0058] Chize acupoint is located in the cubital crease, on the radial side of the biceps brachii tendon depression. Among them, the cubital crease and the radial side of the biceps brachii tendon depression features are not significant; Ximen acupoint is located on the palmar side of the forearm of the human body, on the line connecting Quze acupoint and Daling acupoint, 5 cun above the wrist crease. Among them, the wrist crease feature is not significant. Riyue acupoint is located in the upper abdomen, directly below the nipple, in the 7th intercostal space, 4 cun away from the anterior midline. Among them, the feature of the nipple is significant but the rib feature is not significant.

[0059] Furthermore, in step three, combined with Figure 2 As shown, the keypoint detection module incorporates a residual ladder network concept into its backbone network, fusing intra-layer features to obtain a refined local representation. Since the size of human keypoints varies greatly, an architecture with a wider receptive field is more suitable for extracting features related to different keypoints, as detailed below:

[0060] The residual staircase network replaces the downsampling convolutional layers in the backbone network. The residual staircase network consists of residual staircase blocks, which are connected together by dense element-wise summation. The residual staircase blocks first divide the features into four branches, and then process each branch using a 1×1 convolution. The output of the 1×1 convolution of each branch is further processed by a 3×3 convolution. Finally, the output features of the four branches are concatenated, and then the features are fused by a 1×1 convolution and connected using identity.

[0061] The keypoint detection module is trained through the following steps: a keypoint localization dataset is constructed based on human keypoint images and keypoint names; the keypoint detection module is trained using the constructed keypoint localization dataset.

[0062] The keypoint detection module is implemented using HRNet, a modified residual ladder network, and employs a multi-resolution architecture. It extracts keypoint features from human keypoint images at different resolutions using multiple branch networks, fuses these features using a fusion layer to achieve high-resolution keypoint feature extraction, predicts keypoint locations using a keypoint regression layer to obtain keypoint coordinates, and then accumulates the offsets based on the positional relationship between keypoints and acupoints to obtain the specific acupoint coordinates. (Human body keypoint reference...) Figure 3 As shown.

[0063] Taking Chengqi acupoint as an example. Chengqi acupoint is located on the face, directly below the pupil, between the eyeball and the lower eye socket edge. The coordinates of Chengqi acupoint can be determined based on key points 2, 70, and 71. Let the coordinates of key point 2 be (x2, y2), and the coordinates of key point 70 be (x...). 70 y 70 The coordinates of the 71st key point are (x 71 y 71 The coordinates of Chengqi acupoint are calculated as follows:

[0064]

[0065] Furthermore, in step four, the instance segmentation module is trained through the following steps: constructing an instance segmentation dataset based on human keypoint images, pixel-by-pixel annotations, and instance-level identifiers; and training the instance segmentation module using the constructed instance segmentation dataset.

[0066] The instance segmentation module uses keypoint detection information as the alignment method for instance segmentation, namely, an alignment method based on human skeletal posture. By utilizing the information inherent in human skeletal posture, this alignment method straightens out various human movements and separates overlapping individuals. The specific process is as follows:

[0067] First, K-means is used to optimize the following formula to cluster all human poses in the training set, and the cluster center of each class is taken as a pose template to form a pose template library.

[0068] The types of human postures involved in this application include: front of head, front of torso, front of torso, palm side of arm, outer side of arm, inner side of leg, and outer side of leg.

[0069] To align the pose of each human instance in the image, we need to estimate their affine transformation matrix with respect to a pose template. This task can be accomplished by solving the following optimization problem:

[0070]

[0071] Among them, X i Represents the pose of a human instance in an image; X t H represents the pose template; H represents the affine transformation matrix, which has 5 variables: rotation, scaling factor, x-axis translation, y-axis translation, and whether to perform left and right flips; λ is a regularization parameter.

[0072] Finally, the poses of the human instances in the image are affinely transformed using an alignment matrix to align them to the same size and orientation.

[0073] The instance segmentation module is implemented using a Mask R-CNN model based on human skeleton pose alignment. First, a feature extractor extracts features from images of various human body parts. Then, a candidate region generator generates candidate target regions. Next, a classifier classifies the candidate target regions, a regressor locates them, and finally, a segmenter segments them to obtain the human body part contours. Then, based on the contour path search and combined with keypoint coordinates, acupoint coordinates are obtained. Human body part instance segmentation is referenced... Figure 4 As shown.

[0074] Take the Chize acupoint as an example. The Chize acupoint is located in the depression on the radial side of the biceps brachii tendon, in the middle of the elbow crease. The elbow joint is located midway between the upper arm and forearm. The coordinates of the Chize acupoint can be determined based on the 4th, 5th, and 9th key points. Starting from the 9th key point, search inwards along the contours of the 4th and 5th points until the rate of change of the contour path reaches a preset threshold. The coordinates of this position are the coordinates of the Chize acupoint.

[0075] In step five, the real-world image captured by the camera is input into the computer, combined with the acupoint coordinate information output by the key point detection module and the instance segmentation module, and output to the screen display. The user can see the final enhanced scene image on the screen.

[0076] like Figure 5 As shown, this application embodiment also provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described acupoint location method embodiment based on whole-body key point recognition and instance segmentation, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0077] It should be noted that the first electronic device in the embodiments of this application includes the mobile electronic device and the non-mobile electronic device described above.

[0078] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0079] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0080] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0081] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understandable that the aforementioned modem processor may not be integrated into the processor 710.

[0082] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described acupoint location method based on whole-body key point recognition and instance segmentation, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0083] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0084] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiment of the acupoint location method based on whole-body key point recognition and instance segmentation, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0085] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0086] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0088] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for acupoint positioning based on whole-body key point recognition and instance segmentation, characterized in that, The application relates to an acupoint positioning method based on augmented reality (AR) technology. Step one: reading a video stream from an AR device, extracting an image frame in the video stream, performing part category detection on the image frame by a part detection module, and outputting a determined part category; Step two: according to the feature type of an acupoint in each part of a human body, performing step three on an acupoint with a significant feature, and performing step four on an acupoint with an insignificant feature; Step three: performing key point detection processing on the acupoint with the significant feature by a key point detection module to obtain key point coordinates, accumulating acupoint offset to obtain acupoint coordinates, and finally outputting acupoint positions and names; the residual ladder network thought is introduced into the main network of the key point detection module, and the features in a layer are fused to obtain a fine local representation, and the specific steps are as follows: The residual ladder network is used to replace a down-sampling convolution layer in the main network, the residual ladder network is composed of residual ladder blocks, the residual ladder blocks are connected together through dense element-level summation, the residual ladder blocks first divide features into four branches, then 1*1 convolution is used to process the branches, the output of the 1*1 convolution of each branch is further processed through 3*3 convolution, finally the output features of the four branches are connected in series, then the features are fused through 1*1 convolution, and an identity connection is used; Step four: outputting a part contour by an instance segmentation module, then searching for a feature according to a part contour ratio, accumulating acupoint offset to obtain acupoint coordinates, and finally outputting acupoint positions and names; Step five: superimposing the acupoint positions and names output in steps three and four on a real human body image by using augmented reality technology to realize accurate acupoint positioning.

2. The method of claim 1, wherein: The part detection module comprises an arm processing assembly, a face processing assembly, a trunk processing assembly and a leg processing assembly.

3. The method of claim 1, wherein: The part detection module is trained through the following steps: According to human body part images and part names, a target detection data set is constructed; The part detection module is trained by using the constructed target detection data set.

4. The method of claim 3, wherein: The part detection module is realized based on Faster R-CNN, shared convolution layers are used to extract features of human body part images to obtain part feature maps, the obtained part feature maps are sent to a region proposal network, the region proposal network generates a detection frame, i.e. the position of a specified interest region, and performs a first correction on the surrounding frame of the interest region, an interest region pooling layer selects the features corresponding to each interest region on the part feature map according to the output of the region proposal network, and the dimension is set to a fixed value, a full connection layer is used to classify the surrounding frame, and a second correction of the target surrounding frame is performed.

5. The method of claim 1, wherein: The key point detection module is trained through the following steps: According to human body key point images and key point names, a key point positioning data set is constructed; The key point detection module is trained by using the constructed key point positioning data set.

6. The method of claim 5, wherein: The key point detection module is realized by HRNet improved based on a residual ladder network, adopts a multi-resolution structure design, extracts key point features in a human key point image at different resolutions by using multiple branch networks, fuses the key point features by using a fusion layer, realizes high-resolution key point feature extraction, predicts key point positions by using a key point regression layer, and obtains key point coordinates.

7. The method of claim 5, wherein: The instance segmentation module is trained by the following steps: An instance segmentation dataset is constructed according to a human key point image and pixel-by-pixel annotation and instance-level identification; The instance segmentation module is trained by using the constructed instance segmentation dataset.

8. The method of claim 7, wherein: The instance segmentation module takes the key point detection information as an alignment method for instance segmentation, that is, an alignment method based on a human skeleton posture, and the alignment method based on the human skeleton posture straightens various human motions and separates overlapping people by using information implied by the human skeleton posture.

9. The method of claim 8, wherein: The instance segmentation module is realized by a Mask R-CNN model based on the alignment method based on the human skeleton posture, first, a feature extractor extracts features of images of various parts of a human body, then a candidate region generator generates a candidate target region, next, a classifier classifies the candidate target region, a regressor positions the candidate target region, finally, a segmenter segments the candidate target region to obtain a human body part contour, and then, according to a contour path search, combined with key point coordinates, acupoint coordinates are obtained.

Images