A multi-task learning method and device for acupoint localization guided by human key points

Abstract

This application relates to a multi-task learning method and device for acupoint localization guided by human key points, belonging to the field of medical image processing technology. The method includes: acquiring images of local human regions and constructing a dataset labeled with acupoints and key points; constructing a multi-task acupoint localization network including a backbone network, a multi-task feature enhancement module, a multi-task feature fusion module, and a multi-scale feature prediction module; constructing a multi-task learning loss function and jointly supervising the training of the network; inputting the image to be localized into the trained network to obtain the coordinates of acupoints and key points. This invention introduces human key point localization as an auxiliary task, utilizes the multi-task feature enhancement module and fusion module to achieve information interaction between key points and acupoints, effectively utilizing prior knowledge of human anatomical structures to guide acupoint localization, thereby improving the accuracy, robustness, and model learning efficiency of acupoint localization.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a multi-task learning method and device for acupoint localization guided by human key points. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, their application in the medical field is becoming increasingly profound. Image-based automatic acupoint localization technology is becoming an important direction for the modernization of traditional Chinese medicine (TCM). Acupoint localization is a fundamental prerequisite for TCM diagnostic and therapeutic operations such as acupuncture, massage, cupping, and scraping. The accuracy of acupoint localization directly determines the effectiveness of subsequent treatments and patient safety. In emerging application scenarios such as intelligent acupuncture robots, remote TCM teaching systems, home health self-testing devices, and TCM rehabilitation auxiliary systems, there is an urgent need for an acupoint visual localization technology that can operate in real time without manual annotation by professional physicians and possesses high accuracy and robustness.

[0003] Currently, most deep learning-based acupoint localization methods employ end-to-end multi-layer neural network frameworks. These methods directly map input images to acupoint coordinates or corresponding heatmaps, attempting to drive the model to learn a non-linear mapping relationship from pixels to acupoints through a large amount of labeled data. However, these methods have significant drawbacks: they lack utilization of explicit visual features of the human body, and the entire localization process is conducted in a "black box" manner. The model can only implicitly learn the association between acupoints and images from pixel intensity distribution, and cannot obtain explicit prior guidance from the level of human anatomical structure, such as skeletal joints, body surface contours, and other geometric constraints closely related to acupoint distribution.

[0004] Specifically, existing technologies do not explicitly model the geometric correspondence between acupoints and key points on the human body, nor do they introduce explicit visual features as intermediate supervision signals. This end-to-end direct mapping method has the following problems: First, the model learning efficiency is low, requiring a large number of labeled samples and a long training time to achieve acceptable performance; second, the model is extremely sensitive to external interference factors, and the localization accuracy drops significantly when the input image contains complex situations such as pose changes, uneven lighting, body shape differences, and local occlusion; third, due to the lack of constraints at the anatomical structure level, the model is prone to producing acupoint prediction results that are inconsistent with the human body's geometry, reducing the reliability of the localization results. These problems severely restrict the application of automatic acupoint localization technology in real clinical and home environments.

[0005] Therefore, there is an urgent need in related technologies to improve the accuracy, robustness, and learning efficiency of acupoint location. Summary of the Invention

[0006] Therefore, it is necessary to provide a multi-task learning acupoint localization method and device guided by human key points, which can improve the accuracy, robustness and learning efficiency of acupoint localization and the model, in order to address the above-mentioned technical problems.

[0007] Firstly, this application provides a multi-task learning method for acupoint localization guided by key points on the human body. The method includes: Images of local areas of the human body are acquired and a dataset is constructed, in which acupoints and key points are labeled. A multi-task acupoint localization network guided by human key points is constructed. The network includes a backbone network, a multi-task feature enhancement module, a multi-task feature fusion module, and a multi-scale feature prediction module. The backbone network is used to extract global and local features of local regions of the human body in the input image. The multi-task feature enhancement module is used to enhance the global perception capability of the network based on spatial priors. The multi-task feature fusion module is used to implicitly model the mapping relationship between acupoints and image features by using key points as intermediate variables. The multi-scale feature prediction module is used to generate heatmaps at three scales, decode the heatmaps into acupoint coordinates and key point coordinates, and fuse features at the three scales. Construct a multi-task learning loss function, jointly supervise the training of the multi-task acupoint localization network, and save the parameter weights with the best performance; The image of the local area of ​​the human body to be located is input into the trained multi-task acupoint localization network to obtain the coordinates of the corresponding acupoints and key points.

[0008] Optionally, in one embodiment of this application, the multi-task acupoint localization network includes four consecutive stages of acupoint localization and key point task interaction; wherein, each of the first to third stages includes an enhancement sub-stage, a localization sub-stage, and a fusion sub-stage in sequence; the fourth stage does not include a localization sub-stage. The enhancement sub-stage is used to enhance the upstream intermediate features of the previous stage by utilizing the contour map output by the key point localization and acupoint localization tasks of the previous stage. The localization sub-stage is used to output the contour map, boundary constraint map and point map of each sub-task in parallel through the multi-scale feature prediction module, and fuse the three outputs to obtain fused features. At the same time, the three prediction results are converted into the coordinates predicted by each sub-task of the sub-stage. The fusion sub-stage is used to add the upstream intermediate features to the enhanced features, and then input them together with the key point features and acupoint features output by the localization sub-stage into the multi-task feature fusion module, which outputs the upstream intermediate features of the next stage based on the channel attention mechanism.

[0009] Optionally, in one embodiment of this application, the backbone network includes a coordinate convolutional layer, a max pooling layer, and three residual blocks; the coordinate convolutional layer is used to superimpose coordinate information on the input image to output a feature map with location markers; the max pooling layer is concatenated with the first residual block of the three residual blocks to extract global and local features; the last two residual blocks of the three residual blocks are concatenated to extract deep semantic features and output initial upstream intermediate features.

[0010] Optionally, in one embodiment of this application, the multi-task feature enhancement module includes two feature enhancement units and a residual block; each feature enhancement unit includes an Hourglass network, a convolutional layer, a logical comparison operation unit, and a logical AND operation unit; Each feature enhancement unit is used to extract global information from the acupoint contour map and the key point contour map respectively. Pixels larger than a set threshold are filtered by a logical comparison operation unit, and pixels that exist in both contour maps are retained by a logical AND operation unit. Deep features are extracted from upstream intermediate features by an Hourglass network, and then the deep features and the filtered contour map are weighted and fused in the channel dimension by a convolutional layer. The outputs of the two feature enhancement units are added together and connected to the residual block.

[0011] Optionally, in one embodiment of this application, the multi-task feature fusion module includes two fusion units; each fusion unit includes a convolutional layer, a global average pooling layer, a batch normalization layer, and a sigmoid activation function; The first fusion unit is used to sum the feature map and key point feature map representing the original image features element by element, and then pass them through a convolutional layer and a global average pooling layer to obtain local feature representation and global feature representation respectively. The local feature representation and global feature representation are added to the summed feature and then passed through a batch normalization layer and a sigmoid activation function in sequence to generate weights corresponding to the original image features and key point features, and output the weighted summed fused feature. The second fusion unit is used to fuse the output of the first fusion unit with the acupoint feature map in the same way, and output the final fused feature.

[0012] Optionally, in one embodiment of this application, the multi-scale feature prediction module includes a graph prediction module, a multi-scale feature fusion module, and a graph-to-point conversion module; the graph prediction module includes three convolutional layers for outputting a contour map, a boundary constraint map, and a point map; the multi-scale feature fusion module includes three convolutional layers for unifying and adding the channel numbers of the contour map, boundary constraint map, and point map to obtain multi-scale features; the graph-to-point conversion module includes one convolutional layer for aligning the channel number of the point map with the channel number of the contour map and boundary constraint map, and obtaining the final predicted coordinates through element-wise multiplication and accumulation operations.

[0013] Optionally, in one embodiment of this application, the multi-task learning loss function includes an adaptive wing loss function and an adaptive fuzziness reduction loss function; the adaptive wing loss function is used to calculate the loss between the predicted point map and the real point map, and the loss between the predicted boundary constraint map and the real boundary constraint map; the adaptive fuzziness reduction loss function is used to calculate the loss between the predicted contour map and the real contour map.

[0014] Optionally, in one embodiment of this application, for each sub-stage of each subtask, the loss value of the contour map loss, the boundary constraint map loss, and the point map loss are weighted and summed as the loss value of that sub-stage; the loss value of all sub-stages in each subtask is weighted and summed as the final loss value of that subtask; and the final loss values ​​of the acupoint location subtask and the key point location subtask are weighted and summed as the loss function of the model as a whole.

[0015] Secondly, this application also provides a multi-task learning acupoint positioning device guided by key points on the human body. The device includes a memory and a processor; the memory stores a computer program, and the processor executes the steps of the methods described in the various embodiments above.

[0016] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the various embodiments above.

[0017] The above-mentioned method and device for multi-task learning acupoint localization guided by key human points has the following advantages compared with existing technologies: First, the multi-task acupoint localization network guided by human key points proposed in this invention simultaneously completes two sub-tasks: key point localization and acupoint localization. Through a dual-task interaction mechanism, it enhances the acupoint localization task's ability to perceive human structure.

[0018] Second, the multi-task feature fusion module proposed in this invention enhances the information interaction between key point localization and acupoint localization tasks through feature fusion, thereby strengthening the auxiliary role of key point localization in acupoint localization.

[0019] Third, the multi-task feature enhancement module proposed in this invention uses contour heatmaps to enhance the global characteristics of upstream intermediate features, enhance the spatial perception capability of the network, and further improve the performance of acupoint localization and key point localization.

[0020] Fourth, the multi-task learning loss function proposed in this invention, through joint optimization of multi-stage, multi-task, and multi-supervision signals, enables the model to converge stably under full supervision, gradually improving the performance of acupoint localization and key point localization. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multi-task learning acupoint location method guided by key points of the human body in one embodiment. Figure 2 This is a schematic diagram of a multi-task acupoint positioning network guided by key points of the human body in one embodiment. Figure 3 This is a schematic diagram of the structure of a multi-task feature enhancement module in one embodiment; Figure 4 This is a schematic diagram of the structure of a multi-task feature fusion module in one embodiment; Figure 5 This is a schematic diagram of the structure of a multi-scale feature prediction module in one embodiment; Figure 6 This is an internal structural diagram of a multi-task learning acupoint positioning device guided by key points on the human body, as shown in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] In one embodiment, such as Figure 1 As shown, a multi-task learning method for acupoint localization guided by key points of the human body is provided, including the following steps: S101: Obtain images of local areas of the human body and construct a dataset, in which acupoints and key points are marked.

[0024] In this embodiment, a camera is used to capture high-definition, unobstructed images of local areas of the human body and acupoints and key points are labeled. The image data is then preprocessed to construct a dataset of acupoints and key points in local areas of the human body.

[0025] S102: Construct a multi-task acupoint localization network guided by human key points. The network includes a backbone network, a multi-task feature enhancement module, a multi-task feature fusion module, and a multi-scale feature prediction module. The backbone network is used to extract global and local features of local regions of the human body in the input image. The multi-task feature enhancement module is used to enhance the global perception capability of the network based on spatial priors. The multi-task feature fusion module is used to implicitly model the mapping relationship between acupoints and image features by using key points as intermediate variables. The multi-scale feature prediction module is used to generate heatmaps at three scales, decode the heatmaps into acupoint coordinates and key point coordinates, and fuse features at the three scales.

[0026] In this embodiment of the application, a construction is performed as follows: Figure 2 The diagram illustrates a multi-task acupoint localization network guided by human keypoints. This network comprises a backbone network, a multi-task feature enhancement module, a multi-task feature fusion module, and a multi-scale feature prediction module. Specifically, the backbone network extracts global and local features of local human regions from the input image; the multi-task feature enhancement module enhances the network's global perception capabilities based on spatial priors; the multi-task feature fusion module implicitly models the relationship between acupoints and image features using keypoints as an intermediate variable; and the multi-scale feature prediction module generates heatmaps at three scales, decodes them into acupoint and keypoint coordinates, and fuses the three-scale features of the image.

[0027] In one embodiment of this application, the multi-task acupoint localization network includes four consecutive stages of interaction between acupoint localization and key point tasks; wherein each of the first to third stages includes an enhancement sub-stage, a localization sub-stage, and a fusion sub-stage in sequence; the fourth stage does not include a localization sub-stage. The enhancement sub-stage is used to enhance the upstream intermediate features of the previous stage by utilizing the contour map output by the key point localization and acupoint localization tasks of the previous stage. The localization sub-stage is used to output the contour map, boundary constraint map and point map of each sub-task in parallel through the multi-scale feature prediction module, and fuse the three outputs to obtain fused features. At the same time, the three prediction results are converted into the coordinates predicted by each sub-task of the sub-stage. The fusion sub-stage is used to add the upstream intermediate features to the enhanced features, and then input them together with the key point features and acupoint features output by the localization sub-stage into the multi-task feature fusion module, which outputs the upstream intermediate features of the next stage based on the channel attention mechanism.

[0028] In one embodiment of this application, such as Figure 2 As shown, the multi-task acupoint localization network guided by human key points includes four consecutive stages of interaction between acupoint localization and key point tasks; except for the last stage which does not include a localization sub-stage, the interaction of each of the remaining stages includes an enhancement sub-stage, a localization sub-stage, and a fusion sub-stage, as follows: Figure 2 As shown, the details are as follows: In the enhancement sub-stage, the multi-task feature enhancement module uses the contour maps output from the keypoint localization and acupoint localization tasks in the previous stage. and Enhance the upstream and intermediate features of the previous stage The aim is to provide better feature input for subsequent localization sub-stages; the enhanced feature representation is as follows: This process can be described as follows:

[0029] in, Indicates the first Multi-task feature enhancement module for each stage.

[0030] In the localization sub-stage, the two sub-task networks complete the localization sub-stage in parallel and independently; the multi-scale feature prediction module outputs three prediction results for each sub-task (contour map). Boundary constraint diagram And dot plot The three outputs of each subtask in this localization sub-stage are then combined to obtain the fused features. This provides a foundation for information transfer between stages, and at the same time converts the prediction results of the three scales into the predicted points of each subtask in this sub-stage. This process can be described as follows:

[0031] in, Used to identify and distinguish acupoints and key points; Indicates the first Multi-scale feature prediction module for each stage.

[0032] Fusion sub-stage, upstream intermediate features After processing through a convolutional layer, the enhanced features are compared with those of the previous feature. Adding them together yields the original image features, which contain both global information and local details. Subsequently, the multi-task feature fusion module inputs... The features output by the localization sub-stage that are associated with the two sub-tasks. (Key points) and (Acupoints), based on the channel attention mechanism, fuses three types of features and outputs the upstream intermediate features for the next stage. This process can be represented as:

[0033] in, Indicates the first Multi-task feature fusion module for each stage; Indicates the kernel size as The convolutional layer.

[0034] In one embodiment of this application, the backbone network includes a coordinate convolutional layer, a max pooling layer, and three residual blocks; the coordinate convolutional layer is used to superimpose coordinate information on the input image to output a feature map with location markers; the max pooling layer is concatenated with the first residual block of the three residual blocks to extract global and local features; the last two residual blocks of the three residual blocks are concatenated to extract deep semantic features and output initial upstream intermediate features.

[0035] In one embodiment of this application, such as Figure 2As shown, the backbone network consists of a coordinate convolutional layer, a max pooling layer, and three residual blocks; specifically, firstly, the image of a local human body region... After processing by a coordinate convolutional layer, the output is a feature map with location markers. To enhance the model's spatial awareness; then, a structure consisting of a residual block and a max-pooling layer cascaded together is used to improve the model's spatial awareness. Extracting global and local features from local regions of the human body in an image Finally, using two cascaded residual blocks from Deep semantics that extract global and local information are processed and the initial upstream intermediate features are output. This is used to support downstream positioning tasks. This process can be represented as:

[0036] in, Represents a residual block; Indicates the max pooling layer; This represents a coordinate convolutional layer.

[0037] In one embodiment of this application, the multi-task feature enhancement module includes two feature enhancement units and a residual block; each feature enhancement unit includes an Hourglass network, a convolutional layer, a logical comparison operation unit, and a logical AND operation unit; Each feature enhancement unit is used to extract global information from the acupoint contour map and the key point contour map respectively. Pixels larger than a set threshold are filtered by a logical comparison operation unit, and pixels that exist in both contour maps are retained by a logical AND operation unit. Deep features are extracted from upstream intermediate features by an Hourglass network, and then the deep features and the filtered contour map are weighted and fused in the channel dimension by a convolutional layer. The outputs of the two feature enhancement units are added together and connected to the residual block.

[0038] In one embodiment of this application, such as Figure 3 As shown, the multi-task feature enhancement module includes two feature enhancement units and one residual block; specifically, the two parallel feature enhancement units respectively extract features from the acupoint contour map. and key point outline diagram Extracting global information of local human regions related to acupoint and key point localization tasks. , And use global information to enhance upstream intermediate features. , and These represent the number of acupoints and key points, respectively. The number of contour map channels participating in global information filtering; the outputs of the two feature enhancement units are added together and connected to a residual block to further promote the effective integration of information between subtasks, resulting in enhanced features. This process can be represented as:

[0039] in, Represents a residual block; and These represent the fusion units used for acupoint location and key point location, respectively.

[0040] As a preferred embodiment of the present invention, the feature enhancement unit includes an Hourglass network, a convolutional layer, a logical comparison operation, and a logical AND operation; specifically, a logical comparison operation is first used. Select the acupoint outlines separately and key point outline diagram Medium greater than the set threshold pixels, The magnitude reflects the filtering effect on noise and local interference information; subsequently, a logical AND operation is used. Pixels present in both subtask contour maps are retained. This operation leverages keypoint contour maps, which are more strongly correlated with local human body structures, to remove components in acupoint contour maps that are irrelevant to global information. To further enhance the representation of features highly relevant to subtasks, an Hourglass network is introduced to extract features from upstream intermediate features. Deep features are extracted to ensure the network has sufficient parameters to represent the information required for the subtask; finally, a convolutional layer is used to combine the features extracted by Hourglass with the filtered contour map. Weighted fusion along the channel dimension yields the fused features. This process can be represented as:

[0041] in, Indicates the kernel size as Convolutional layers; This indicates a splicing operation at the channel dimension.

[0042] In one embodiment of this application, the multi-task feature fusion module includes two fusion units; each fusion unit includes a convolutional layer, a global average pooling layer, a batch normalization layer, and a sigmoid activation function; The first fusion unit is used to sum the feature map and key point feature map representing the original image features element by element, and then pass them through a convolutional layer and a global average pooling layer to obtain local feature representation and global feature representation respectively. The local feature representation and global feature representation are added to the summed feature and then passed through a batch normalization layer and a sigmoid activation function in sequence to generate weights corresponding to the original image features and key point features, and output the weighted summed fused feature. The second fusion unit is used to fuse the output of the first fusion unit with the acupoint feature map in the same way, and output the final fused feature.

[0043] In one embodiment of this application, such as Figure 4 As shown, the multi-task feature fusion module includes two fusion units; specifically, firstly, the features representing the original image are fused together. and key features Input the first fusion unit to obtain the fusion features. Because the distribution of key points is visually correlated with the structure of local areas of the human body, ensuring... It can directly establish a correlation with the features of the original image; subsequently, Characteristics of acupoints Input the second fusion unit to obtain Since the distribution of acupoints and keypoints are correlated, by using keypoint features as a bridge, acupoint features can be indirectly associated with image features, thus effectively modeling the mapping relationship between acupoints and image features. Furthermore, residual connections are used to sum the results from intermediate processes to avoid the vanishing gradient phenomenon. This process can be represented as:

[0044] in, This is the final output of the multi-task feature fusion module; and These represent the first and second fusion units, respectively.

[0045] As a preferred embodiment of the present invention, the fusion unit includes a convolutional layer, a global average pooling layer, a batch normalization layer, and a sigmoid activation function; specifically, the input feature map is first processed... and Element-wise summation is performed, and then a convolutional layer is used to weight and fuse the summed features along the channel dimension to obtain a local feature representation. Simultaneously, a global average pooling layer is used to extract a global representation of the summed features. Then, , and After the three are added together, they are sequentially passed through a batch normalization layer and a Sigmoid activation function to generate the corresponding... weight and weight Final output Depend on and We obtain the result by weighted summation. This process can be represented as:

[0046] in, Indicates element-wise multiplication; Indicates the batch normalization layer; The spatial size of the output feature map is... Global average pooling layer; Indicates the kernel size as The convolutional layer.

[0047] In one embodiment of this application, the multi-scale feature prediction module includes a graph prediction module, a multi-scale feature fusion module, and a graph-to-point conversion module. The graph prediction module includes three convolutional layers for outputting a contour map, a boundary constraint map, and a point map. The multi-scale feature fusion module includes three convolutional layers for unifying and adding the channel numbers of the contour map, boundary constraint map, and point map to obtain multi-scale features. The graph-to-point conversion module includes one convolutional layer for aligning the channel number of the point map with the channel number of the contour map and boundary constraint map, and obtaining the final predicted coordinates through element-wise multiplication and accumulation operations.

[0048] In one embodiment of this application, such as Figure 5 As shown, the multi-scale feature prediction module includes a graph prediction module, a multi-scale feature fusion module, and a graph-to-point conversion module. The graph prediction module includes three convolutional layers; the multi-scale feature fusion module includes three convolutional layers; and the graph-to-point conversion module includes one convolutional layer. Specifically, the graph prediction module uses three convolutional layers to convert the enhanced features... Weighted by channel dimension, output the contour map related to the subtask. Boundary constraint diagram And dot plot , Used to identify and distinguish acupoints and key points. This indicates the number of boundary constraints for acupoints or key points; the multi-scale feature fusion module uses three convolutional layers to... , and The number of channels is unified to The multi-scale features are obtained by summing the outputs of the three convolutional layers. The graph-to-point module uses a convolutional layer to... The number of channels and and Alignment is performed, and the final predicted coordinates are obtained through element-wise multiplication and accumulation operations. This process can be described as follows:

[0049] in, Indicates the kernel size as Convolutional layers; Indicates an accumulation operation; This indicates element-wise multiplication.

[0050] S103: Construct a multi-task learning loss function, jointly supervise the training of the multi-task acupoint localization network, and save the parameter weights with the best performance.

[0051] In one embodiment of this application, the multi-task learning loss function includes an adaptive wing loss function and an adaptive fuzzy reduction loss function; the adaptive wing loss function is used to calculate the loss between the predicted point map and the true point map, and the loss between the predicted boundary constraint map and the true boundary constraint map; the adaptive fuzzy reduction loss function is used to calculate the loss between the predicted contour map and the true contour map.

[0052] In one embodiment of this application, for each sub-stage of each subtask, the loss value of the contour map loss, the boundary constraint map loss, and the point map loss are weighted and summed to obtain the loss value of that sub-stage; the loss values ​​of all sub-stages in each subtask are weighted and summed to obtain the final loss value of that subtask; and the final loss values ​​of the acupoint location subtask and the key point location subtask are weighted and summed to obtain the overall loss function of the model.

[0053] In one embodiment of this application, the multi-task learning loss function includes an adaptive wing loss function and an adaptive fuzziness reduction loss function; specifically, in the first subtask... Each sub-stage uses the adaptive wing loss function. Calculate the predicted point map separately and boundary constraint diagram With tags and The loss between them is calculated using an adaptive fuzzy reduction loss function. Calculate the predicted profile map With tags The loss between each monitoring signal; the weighted sum of the loss values ​​of each monitoring signal is used as the loss value of this sub-stage of the subtask. The weighted sum of the loss values ​​of each subtask at each stage is used as the final loss value for optimizing that subtask. The final loss functions of the two sub-tasks are further weighted and summed to form the overall loss function of the model. This process can be described as follows:

[0054] in, Used to identify and distinguish acupoints and key points; Used to control subtasks The degree of contribution to the overall loss; For subtasks The Weighting coefficients for each sub-stage; , , These are the weighting coefficients for each loss term, used to balance the contributions of different monitoring signals to the loss; This represents the error between the actual pixel value and the predicted pixel value. Used to ensure the continuity of the function at the threshold; The threshold for dividing the small error region into the large error region; Control the loss scaling in the small error region; Used to prevent division by zero errors; Control the linear slope in the large error region; and They are respectively and Unbiased weighted covariance; , , , Depend on The unbiased estimator is obtained by eigenvalue decomposition; This is the distance function.

[0055] S104: Input the image of the local area of ​​the human body to be located into the trained multi-task acupoint localization network to obtain the corresponding acupoint and key point coordinates.

[0056] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0057] In one embodiment, a multi-task learning acupoint localization device guided by human key points is provided. This device can be a terminal, and its internal structure diagram can be as follows: Figure 6 As shown, the device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a multi-task learning acupoint location method guided by key human points. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0058] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the multi-task learning acupoint positioning device guided by human key points applied thereto. A specific multi-task learning acupoint positioning device guided by human key points may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0059] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0061] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0062] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-task learning method for acupoint localization guided by key points of the human body, characterized in that, The method includes: Images of local areas of the human body are acquired and a dataset is constructed, in which acupoints and key points are labeled. A multi-task acupoint localization network guided by human key points is constructed. The network includes a backbone network, a multi-task feature enhancement module, a multi-task feature fusion module, and a multi-scale feature prediction module. The backbone network is used to extract global and local features of local regions of the human body in the input image. The multi-task feature enhancement module is used to enhance the global perception capability of the network based on spatial priors. The multi-task feature fusion module is used to implicitly model the mapping relationship between acupoints and image features by using key points as intermediate variables. The multi-scale feature prediction module is used to generate heatmaps at three scales, decode the heatmaps into acupoint coordinates and key point coordinates, and fuse features at the three scales. Construct a multi-task learning loss function, jointly supervise the training of the multi-task acupoint localization network, and save the parameter weights with the best performance; The image of the local area of ​​the human body to be located is input into the trained multi-task acupoint localization network to obtain the coordinates of the corresponding acupoints and key points.

2. The method for multi-task learning acupoint localization guided by key human points according to claim 1, characterized in that, The multi-task acupoint localization network includes four consecutive stages of interaction between acupoint localization and key point tasks; wherein each of the first to third stages includes an enhancement sub-stage, a localization sub-stage, and a fusion sub-stage in sequence; the fourth stage does not include a localization sub-stage. The enhancement sub-stage is used to enhance the upstream intermediate features of the previous stage by utilizing the contour map output by the key point localization and acupoint localization tasks of the previous stage. The localization sub-stage is used to output the contour map, boundary constraint map and point map of each sub-task in parallel through the multi-scale feature prediction module, and fuse the three outputs to obtain fused features. At the same time, the three prediction results are converted into the coordinates predicted by each sub-task of the sub-stage. The fusion sub-stage is used to add the upstream intermediate features to the enhanced features, and then input them together with the key point features and acupoint features output by the localization sub-stage into the multi-task feature fusion module, which outputs the upstream intermediate features of the next stage based on the channel attention mechanism.

3. The method for multi-task learning acupoint localization guided by key human points according to claim 2, characterized in that, The backbone network includes a coordinate convolutional layer, a max pooling layer, and three residual blocks. The coordinate convolutional layer is used to superimpose coordinate information on the input image to output a feature map with location markers. The max pooling layer is concatenated with the first residual block of the three residual blocks to extract global and local features. The last two residual blocks of the three residual blocks are concatenated to extract deep semantic features and output initial upstream intermediate features.

4. The method for multi-task learning acupoint localization guided by key human points according to claim 2, characterized in that, The multi-task feature enhancement module includes two feature enhancement units and a residual block; each feature enhancement unit includes an Hourglass network, a convolutional layer, a logical comparison operation unit, and a logical AND operation unit. Each feature enhancement unit is used to extract global information from the acupoint contour map and the key point contour map respectively. Pixels larger than a set threshold are filtered by a logical comparison operation unit, and pixels that exist in both contour maps are retained by a logical AND operation unit. Deep features are extracted from upstream intermediate features by an Hourglass network, and then the deep features and the filtered contour map are weighted and fused in the channel dimension by a convolutional layer. The outputs of the two feature enhancement units are added together and connected to the residual block.

5. The multi-task learning acupoint location method guided by key human points according to claim 2, characterized in that, The multi-task feature fusion module includes two fusion units; each fusion unit includes a convolutional layer, a global average pooling layer, a batch normalization layer, and a sigmoid activation function; The first fusion unit is used to sum the feature map and key point feature map representing the original image features element by element, and then pass them through a convolutional layer and a global average pooling layer to obtain local feature representation and global feature representation respectively. The local feature representation and global feature representation are added to the summed feature and then passed through a batch normalization layer and a sigmoid activation function in sequence to generate weights corresponding to the original image features and key point features, and output the weighted summed fused feature. The second fusion unit is used to fuse the output of the first fusion unit with the acupoint feature map in the same way, and output the final fused feature.

6. The method for multi-task learning acupoint localization guided by key human points according to claim 2, characterized in that, The multi-scale feature prediction module includes a graph prediction module, a multi-scale feature fusion module, and a graph-to-point conversion module. The graph prediction module includes three convolutional layers to output a contour map, a boundary constraint map, and a point map. The multi-scale feature fusion module includes three convolutional layers to unify the number of channels in the contour map, boundary constraint map, and point map and then add them together to obtain multi-scale features. The graph-to-point conversion module includes one convolutional layer to align the number of channels in the point map with the number of channels in the contour map and boundary constraint map, and obtains the final predicted coordinates through element-wise multiplication and accumulation operations.

7. The method for multi-task learning acupoint localization guided by key human points according to claim 1, characterized in that, The multi-task learning loss function includes an adaptive wing loss function and an adaptive fuzzy reduction loss function; the adaptive wing loss function is used to calculate the loss between the predicted point map and the true point map, as well as the loss between the predicted boundary constraint map and the true boundary constraint map; the adaptive fuzzy reduction loss function is used to calculate the loss between the predicted contour map and the true contour map.

8. The method for multi-task learning acupoint localization guided by key human points according to claim 7, characterized in that, For each sub-stage of each subtask, the loss of the contour map, the loss of the boundary constraint map, and the loss of the point map are weighted and summed to obtain the loss value of that sub-stage; the loss values ​​of all sub-stages in each subtask are weighted and summed to obtain the final loss value of that subtask. The weighted sum of the final loss values ​​of the acupoint location subtask and the key point location subtask is used as the overall loss function of the model.

9. A multi-task learning acupoint positioning device guided by key human points, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images