A method for eye acupoint localization based on modular feature enhanced GAN

Abstract

The present invention discloses an eye acupoint localization method based on a modular feature-enhanced GAN, which belongs to the technical field of eye acupoint localization. The specific technical solution is as follows: constructing a training data set, including collecting facial images of human faces to form an image data set without acupoint calibration, and marking eye acupoints on the facial images to form an image data set with acupoint calibration; adopting a supervised training strategy to construct a cyclic generative adversarial model, designing a feature enhancement module in the modular feature-enhanced GAN to enhance the acupoint localization capability of the generator; designing a combination of multiple loss functions such as adversarial loss, L1 supervised loss, and KL loss to constrain the generator and the discriminator; the present invention can automatically locate eye acupoints according to the above system and method, which reduces the difficulty of acupoint localization on the one hand, and improves the acupoint localization capability of the GAN network on the other hand.

Description

Technical Field

[0001] The present invention belongs to the technical field of eye acupoint positioning, and in particular relates to an eye acupoint positioning method based on modular feature enhanced GAN. Background Art

[0002] The accelerated pace of life and increased work pressure in today's society have led to a high incidence of chronic diseases and sub-health issues. To minimize work productivity, people tend to seek quick help from Western medicine. However, for chronic conditions, acupuncture is undoubtedly more effective in Traditional Chinese Medicine (TCM). Acupuncture primarily relies on acupuncture to stimulate the circulation of Qi and blood, which is then transported to specific areas of the body through the meridians to achieve therapeutic effects. Acupuncture is a vital component of my country's healthcare sector and a representative of humanity's intangible cultural heritage. It is widely used in areas such as disease treatment, daily health care, and beauty massage. However, the key to acupuncture lies in the rapid and accurate identification of acupoints on the human body, which requires costly training and extensive experience. Individual patient differences and the subjectivity of commonly used acupoint location methods in clinical practice can lead to inaccuracies in acupoint location, which can prevent acupuncture treatment from achieving its intended effect and even cause harm to the body. Therefore, the development of a simple and efficient automatic acupoint location technology is particularly significant. Due to the complexity of acupoint positioning and the limitations of technical means, as well as the need to consider various human factors in automatic acupoint positioning, existing automatic acupoint positioning methods have weak generalization capabilities and cannot adapt to the acupoint positioning needs of different populations and different environments. At the same time, automatic acupoint positioning methods are complex to operate and require professional knowledge and skills, which limits their promotion and application in clinical applications. In response to these problems and to achieve the modernization of acupoint positioning, more and more scholars have begun to apply modern information technology, big data, artificial intelligence, etc. to acupoint positioning to improve the shortcomings of existing automatic acupoint positioning methods.

[0003] In recent years, deep learning (DL) has experienced rapid development and has been widely applied in many fields, including computer vision, object detection, and image generation, achieving promising results. However, these models suffer from poor user control capabilities, and the image resolution and quality need to be improved. For example, Yang Xiangping et al. designed a model for predicting relative coordinates of acupoints based on a particle swarm optimization-based neural network (PSO-BP) algorithm and combined it with an ARM processor to create a system for human acupoint localization. However, due to the impact of floating-point numbers on precision and the cumulative loss caused by the Taylor expansion of the transfer function, the accuracy of acupoint localization is low. Zhang Huakai et al. proposed new acupoint localization methods based on fast correlation grayscale normalization image matching and SFT feature matching, respectively, using manual labeling. They also conducted experimental analysis on real images. Although these methods can design acupoint localization methods, the output is relatively slow. Chang Menglong used the ASM algorithm to locate facial key points, refined the collected information through edge extraction algorithms and feature triangulation, and proposed a method for calculating facial bone scale and acupoint localization in facial images. A neural network was then used to locate acupoints without obvious graphical features. However, these algorithms impose strict sample restrictions, hindering research, widespread application, and limited optimization of the BP network algorithm. Therefore, better methods are needed to achieve both accurate and convenient acupoint localization. Currently, a model that produces higher-quality images and is more concise is the deep learning-based generative adversarial network (GAN) proposed by Goodfellow et al. Building on GANs, Isola et al. proposed Pix2pix. Because it is a one-to-one mapping, where the output for a given input is always the same, when noise is added to the generator input, the network ignores the noise and the output image remains unchanged, hindering the ability to label acupoint locations on different faces. To address this issue, we need to find a method that forces the generator to not ignore the noise but instead uses it to generate diverse images. This is the BicycleGAN proposed by JYZhu et al. This model utilizes the location of facial acupoints in Traditional Chinese Medicine (TCM) and the positions of the face and facial organs to design the generator structure. The generator first generates images of acupoint locations, which are then judged by the discriminator to determine if the images are real or fake. This adversarial process continuously updates the weight parameters in the generator, ultimately achieving accurate and rapid acupoint localization. This method not only improves the accuracy and convenience of acupoint positioning, but also provides strong support for the digital transformation and modernization development of traditional Chinese medicine.

[0004] In summary, the denoising network based on the GAN framework still has problems such as unstable network training process, poor generalization ability and incomplete feature extraction. In the training data set, the accuracy of acupoint positioning is often affected by the standardization of the collected facial images. In order to better test the acupoint positioning network designed by us, the research of this invention aims to use existing camera equipment (such as mobile phones, computers, tablets, etc.) to analyze and detect facial images, identify the facial features of the target person, and then automatically locate the acupoints; during the training process, a variety of loss functions are used to impose targeted constraints on the output results of each sub-network to ensure the stability of network training; in the framework optimization, an eye acupoint positioning network based on modular feature enhancement generative adversarial network is proposed, in which the dual-loop generation strategy is more capable of utilizing the adversarial advantages of the generator and discriminator. Summary of the Invention

[0005] The purpose of this invention is to provide an eye acupoint localization method based on modular feature enhanced GAN, which automatically calibrates acupoints according to eye images, reduces the difficulty of acupoint calibration, and improves the accuracy of acupoint calibration.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a method for eye acupoint localization based on modular feature enhanced GAN, which is carried out according to the following steps:

[0007] Step 1) Select a supervised training strategy and construct a paired eye acupoint image dataset;

[0008] Step 2) Input the paired image data into a modular feature-enhanced generative adversarial network, and train the network model by inputting training data to reach a convergence state. The modular feature-enhanced generative adversarial network includes two branches: the I2I-GAN model and the Z2Z-GAN model. The I2I-GAN model consists of a generator, an encoder, and a discriminator D1, and the Z2Z-GAN model consists of a generator, an encoder, and a discriminator D2. The two branches correspond to different training methods;

[0009] The I2I-GAN model first encodes the noise z1 from the image B with acupoint calibration, then inputs the noise z1 and the image A without acupoint calibration into the generator with the feature enhancement module to locate the acupoints, and finally outputs the image with acupoints.

[0010] The Z2Z-GAN model inputs the image A without acupoint calibration and random noise N(z) into the generator with feature enhancement module to locate the acupoints and outputs an image with acupoints. Finally, from the image The noise z2 is encoded in

[0011] Discriminator D1 determines the image with acupuncture points output by the I2I-GAN model generator True or false, the discriminator D2 determines the image with acupuncture points output by the Z2Z-GAN model generator true and false;

[0012] Finally, the gradient optimization algorithm is used for iterative optimization;

[0013] Step 3) Automatic acupoint positioning and output: The input image without acupoint calibration is converted into the corresponding image with acupoint calibration through the modular feature enhancement generative adversarial network model after iterative training, and then the image is output.

[0014] Furthermore, the specific function of iterative optimization in step 2) is:

[0015] The loss function of the discriminator D1 is:

[0016]

[0017] The loss function of the discriminator D2 is:

[0018]

[0019] Where A represents the image without acupoint calibration, It represents the image with acupoint calibration output by the I2I-GAN model. represents the acupoint-calibrated image output by the Z2Z-GAN model, z1 is the noise output by the encoder in the I2I-GAN model, z2 represents the noise output by the encoder in the Z2Z-GAN model, and N(z) represents random noise;

[0020] Image output by the generator in the I2I-GAN model There is a standard acupoint calibration image B for constraint, which is expressed as:

[0021]

[0022] The noise z1 output by the encoder in the I2I-GAN model must obey the Gaussian distribution and is constrained by the KL loss, which is expressed as:

[0023] L KL (E)=E B～p(B) [D KL (E(B))||N(z)]

[0024] In the Z2Z-GAN model, the random noise N(z) and the noise z2 of the encoded output are constrained by L1 loss, which is expressed as:

[0025]

[0026] The total loss is:

[0027]

[0028] L GAN =L GAN1 +L GAN2

[0029] Where λ is the loss term weight.

[0030] Furthermore, the generator is a U-Net network with a 16-layer encoder-decoder structure, a convolution kernel of 4, a stride of 2, and a zero padding of 1. Cross-layer connections are added to input the features of the encoding layer into the decoding layer. At the same time, the channel enhancement module and the spatial enhancement module are combined into a feature enhancement module, which is embedded between the encoder and decoder in the generator.

[0031] Furthermore, the channel enhancement module performs average pooling and maximum pooling on the image features, passes the two pooled features through a multi-layer neural network, and finally adds the two features together to obtain the feature weight through an activation function and multiplies it with the input feature map.

[0032] Furthermore, the spatial enhancement module sequentially processes the input features through average pooling and maximum pooling, performs convolution operations, and finally uses the activation function to obtain the feature weights and multiply them with the input feature map.

[0033] Furthermore, the specific process of constructing the paired eye acupoint image dataset in step 1) is: capturing a human facial image through a camera, and manually marking acupoints on the human facial image to form an acupoint calibration image.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. Input facial image information, convert facial images without acupoint calibration into facial images with acupoint calibration by using a modular feature-enhanced generative adversarial network model, and then output the image; by adopting the generative adversarial training method of supervised learning, the accuracy and efficiency of acupoint positioning are improved.

[0036] 2. The feature enhancement module designed in the modular feature enhancement generative adversarial network generator, including the channel enhancement module and the spatial enhancement module, improves the network's grasp of image features and improves the accuracy of facial acupoint positioning.

[0037] 3. Design multiple loss functions for modular feature-enhanced generative adversarial networks, including adversarial loss L GAN , L1 loss, and KL loss. The combination of multiple loss functions constrains the entire network to ensure the smooth progress of supervised training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the processing flow of the modular feature-enhanced generative adversarial network of the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of the automatic eye acupoint localization method based on modular feature enhanced generative adversarial network;

[0040] Figure 3 This is a schematic diagram of the generator network;

[0041] Figure 4 Schematic diagram of the discriminator network;

[0042] Figure 5 Provide visual effects for acupuncture point positioning;

[0043] Figure 6 This is a summary diagram of acupoint positioning indicators for different algorithms;

[0044] Figure 7 Line chart and curve chart for feature enhancement module comparison;

[0045] Figure 8 To analyze the results of network acupoint localization with different learning rates;

[0046] Figure 9 This is a comparison chart of the experimental results of the L1 loss coefficient λ; DETAILED DESCRIPTION

[0047] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] Before introducing the specific automatic eye acupoint positioning method of the present invention, the overall design concept of the present invention is briefly described.

[0049] 1. Selection of acupoint positioning model

[0050] The powerful capabilities of deep learning rely on large amounts of training data. This is especially true for image processing, where training networks with paired data (clean images of the same face and images with acupuncture point calibration) significantly improves performance. The BicycleGAN network used in this paper is a supervised deep learning network. However, the original BicycleGAN suffers from unstable training and poor acupuncture point localization. To address this, we improved upon it by adding a feature enhancement module and enhancing the generator, ultimately proposing a modular feature-enhanced generative adversarial network.

[0051] 2. Selection of feature enhancement module:

[0052] The feature enhancement module adopts a combination of a channel enhancement module and a spatial enhancement module. On the one hand, considering that in the process of the generator encoding the image, the random convolution operation is performed, the number of channels increases accordingly, and at the same time, the deep features of the image are also increasing, so we choose to use the channel enhancement module to enhance the image features; on the second hand, considering that the key issue of acupoint positioning is accuracy, the position of the acupoints is closely related to the features in the image space, so the spatial enhancement module is combined with the channel enhancement module to form a feature enhancement module; on the third hand, considering the position of the feature enhancement module in the generator, since the image will gradually generate acupoints at the decoding end, the present invention inserts the feature enhancement module after the encoding end and before the decoding end of the generator, which is helpful for the generation of acupoints.

[0053] 3. Design of loss function:

[0054] According to the structural characteristics of the network, a loss function system based on adversarial loss is designed, which is composed of adversarial loss L GAN , L1 loss and KL loss, L1 loss ensures the quality of output image and acupoint positioning, KL loss ensures the consistency of noise, and adversarial loss ensures the adversarial training of generator and discriminator and the parameter update of both, thereby constraining the entire network to ensure the smooth progress of training.

[0055] Based on the above design ideas, the specific positioning method of the present invention is described in detail below. Figure 1 As shown in the figure, a method for eye acupoint localization based on modular feature enhancement GAN includes: dataset construction, image input module, acupoint calibration network, and image output module. The specific steps are as follows:

[0056] Step 1) generating a training data set, including collecting human facial image data, and marking acupoints on the human facial image to form an acupoint marking image data set.

[0057] Step 2) Based on the modular feature enhancement generative adversarial network, the network model is trained by inputting training data to reach a convergence state.

[0058] Step 3) The input image without acupoint calibration is converted into the corresponding image with acupoint calibration through the modular feature enhancement generative adversarial network model after iterative training, and then the image is output.

[0059] Among them, such as Figure 2As shown in the figure, the modular feature-enhanced generative adversarial network includes two generators G, two encoders E, and discriminators D1 and D2. The generator is responsible for generating images with acupoint calibration by adding noise information and images without acupoint calibration through the feature enhancement module, while the encoder is responsible for extracting noise information from the image. The modular feature-enhanced generative adversarial network includes two branches: the I2I-GAN model and the Z2Z-GAN model. The I2I-GAN model consists of a generator G, an encoder E, and a discriminator D1, while the Z2Z-GAN model consists of a generator G, an encoder E, and a discriminator D2. The two branches correspond to different training methods.

[0060] The convergence process includes the following steps:

[0061] The I2I-GAN model first encodes the noise z1 from the image B with acupoint calibration, then inputs the noise z1 and the image A without acupoint calibration into the generator with the feature enhancement module to locate the acupoints, and finally outputs the image with acupoints. The Z2Z-GAN model inputs the image A without acupoint calibration and random noise N(z) into the generator with feature enhancement to locate the acupoints and outputs an image with acupoints. Finally, from the image The noise z2 is encoded in .

[0062] like Figure 3 As shown, the generator is a 16-layer U-Net network with a codec structure, 4 convolution kernels, a stride of 2, and zero padding of 1. Cross-layer connections are also implemented to feed features from the encoding layer into the decoding layer, preventing excessive information loss during upsampling. A feature enhancement module is added after the encoding layer to help the generator better grasp image features, resulting in higher-quality and more accurate acupuncture point locations.

[0063] like Figure 4 As shown in Figure 1, both the discriminant networks D1 and D2 use the patchD structure. PatchD is a 6-layer fully convolutional network. In its first four convolutional layers, the convolution kernel size is 4×4 and the stride is 2; in its last two convolutional layers, the convolution kernel size is 4×4 and the stride is 1.

[0064] like Figure 2 As shown in the figure, in order to ensure the overall performance of the eye acupoint positioning network, multiple loss functions are used to impose targeted constraints on different parts of the network.

[0065] Specifically, the discriminator loss is used to distinguish the authenticity of the image output by the generator, and the discriminator D1 determines whether the image output by the I2I-GAN model generator has acupuncture points. True or false, the discriminator D2 determines the image with acupuncture points output by the Z2Z-GAN model generator true and false.

[0066] The loss function of the discriminator D1 is:

[0067]

[0068] The loss function of the discriminator D2 is:

[0069]

[0070] Where A represents the image without acupoint calibration, It represents the image with acupoint calibration output by the I2I-GAN model. represents the acupoint-calibrated image output by the Z2Z-GAN model, z1 represents the noise output by the encoder in the I2I-GAN model, and z2 represents the noise output by the encoder in the Z2Z-GAN model;

[0071] Image output by the generator in the I2I-GAN model There is a standard acupoint calibration image B for constraint, which is expressed as:

[0072]

[0073] The noise z1 output by the encoder in the I2I-GAN model must obey the Gaussian distribution and is constrained by the KL loss, which is expressed as:

[0074] L KL (E)=E B～p(B) [D KL (E(B))||N(0,I)]

[0075] In the Z2Z-GAN model, the random noise N(z) and the noise z2 of the encoded output are constrained by L1 loss, which is expressed as:

[0076]

[0077] The total loss is:

[0078]

[0079] L GAN =L GAN1 +L GAN2

[0080] Where λ is the loss term weight.

[0081] Figure 5 This is a diagram showing the results of eye acupoint localization based on the modular feature-enhanced generative adversarial network described in the present invention;

[0082] Figure 6The index values ​​of acupoint calibration for different algorithms are given. CNN-UNet represents a CNN network with UNet as the generator and no discriminator. Pix2pix represents a supervised GAN network with a UNet generator and PatchD as the discriminator. CycleGAN represents a cyclic generative unsupervised network with two generators and two discriminators, and the generator and discriminator structures are the same as Pix2pix. BicycleGAN is the original model described in the present invention. As can be seen from the figure, the OAR value obtained by the algorithm used in the present invention is as high as 78% among the five algorithms, which is better than the OAR obtained by the other four algorithms. ACE is analyzed and the coordinate error of the algorithm of the present invention is the lowest among the five algorithms. Therefore, the acupoint localization effect of the modular feature-enhanced generative adversarial network described in the present invention is the best.

[0083] Figure 7 The comparative experimental results of adding different feature enhancement modules to the original BicycleGAN are shown. The horizontal axis represents the network with different feature enhancement modules. Original Bicycle represents the original BicycleGAN network; Space represents the original BicycleGAN network with only the space enhancement module added; Channel represents the original BicycleGAN network with only the channel enhancement module added; The method in this paper represents the modular feature enhancement-based generative adversarial network described in this invention; The left vertical axis is the OAR index, and the right vertical axis is the ACE index. Figure 7 It can be seen intuitively that the OAR value obtained by the feature enhancement module of the present invention reaches the highest of 78.89% compared with the other three algorithms, and the coordinate error ACE is the lowest at 5.04. The OAR values ​​obtained by the other three algorithms are ranked as follows: channel < original Bicycle < space, and the ACE values ​​obtained are ranked as follows: space < original Bicycle < channel. It can be concluded that the acupoint positioning effect obtained by using the feature enhancement module added to the modular feature enhancement generative adversarial network described in the present invention is the best.

[0084] Figure 8 is the performance index of the modular feature-enhanced generative adversarial network according to the present invention under different learning rates. When the number of iterations is fixed at 1000, Figure 8 It can be seen that when the learning rate is 0.0002, the OAR value and ACE value are optimal, which are 61.67% and 6.42 respectively.

[0085] Figure 9This figure summarizes the network output metrics for different values ​​of the L1 loss function coefficient λ. The horizontal axis shows different values ​​of λ, the left-hand vertical axis shows the OAR metric, and the right-hand vertical axis shows the ACE metric. Multiple experiments were conducted by varying the L1 coefficient λ to achieve the best results. As shown in the figure, when the L1 loss function coefficient is 10, the optimal OAR and ACE values ​​are achieved, reaching 61.67% and 6.42, respectively.

[0086] In summary, the eye acupoint localization network based on modular feature-enhanced generative adversarial network of the present invention utilizes existing camera equipment (such as mobile phones, computers, tablets, etc.) to identify the facial features of the target person through analysis and detection of facial images, and then automatically locates the acupoints; during the training process, a variety of loss functions are used to impose targeted constraints on the output results of each sub-network to ensure the stability of network training; in the framework optimization, the dual-loop generation strategy is more capable of utilizing the adversarial advantages of the generator and the discriminator.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims

1. A method for eye acupoint localization based on modular feature enhanced GAN, characterized in that: Follow these steps: Step 1) Select a supervised training strategy and construct a paired eye acupoint image dataset; Step 2) Input the paired image data into a modular feature-enhanced generative adversarial network, and train the network model by inputting training data to reach a convergence state. The modular feature-enhanced generative adversarial network includes two branches: the I2I-GAN model and the Z2Z-GAN model. The I2I-GAN model consists of a generator, an encoder, and a discriminator D1, and the Z2Z-GAN model consists of a generator, an encoder, and a discriminator D2. The two branches correspond to different training methods; The I2I-GAN model first encodes the noise z1 from the image B with acupoint calibration, then inputs the noise z1 and the image A without acupoint calibration into the generator with the feature enhancement module to locate the acupoints, and finally outputs the image with acupoints. The Z2Z-GAN model inputs the image A without acupoint calibration and random noise N(z) into the generator with feature enhancement module to locate the acupoints and outputs an image with acupoints. Finally, from the image The noise z2 is encoded in Discriminator D1 determines the image with acupuncture points output by the I2I-GAN model generator True or false, the discriminator D2 determines the image with acupuncture points output by the Z2Z-GAN model generator true and false; Finally, the gradient optimization algorithm is used for iterative optimization; Step 3) Automatic acupoint positioning and output: The input image without acupoint calibration is converted into the corresponding image with acupoint calibration through the iteratively trained modular feature enhancement generative adversarial network model, and then the image is output; The specific function of iterative optimization in step 2) is: The loss function of the discriminator D1 is: The loss function of the discriminator D2 is: Where A represents the image without acupoint calibration, It represents the image with acupoint calibration output by the I2I-GAN model. represents the acupoint-calibrated image output by the Z2Z-GAN model, z1 is the noise output by the encoder in the I2I-GAN model, z2 represents the noise output by the encoder in the Z2Z-GAN model, and N(z) represents random noise; Image output by the generator in the I2I-GAN model There is a standard acupoint calibration image B for constraint, which is expressed as: The noise z1 output by the encoder in the I2I-GAN model must obey the Gaussian distribution and is constrained by the KL loss, which is expressed as: L KL (E)=E B～p(B) [D KL (E(B))||N(z)] In the Z2Z-GAN model, the random noise N(z) and the noise z2 of the encoded output are constrained by L1 loss, which is expressed as: The total loss is: L GAN =L GAN1 + LGAN2 Where λ is the loss term weight.

2. The eye acupoint localization method based on modular feature-enhanced GAN according to claim 1, characterized in that: The generator is a U-Net network with a 16-layer encoder-decoder structure, 4 convolution kernels, a stride of 2, and zero padding of 1. Cross-layer connections are added to input the features of the encoding layer into the decoding layer. At the same time, the channel enhancement module is combined with the spatial enhancement module and embedded between the encoder and decoder in the generator.

3. The method for eye acupoint localization based on modular feature-enhanced GAN according to claim 2, characterized in that: The channel enhancement module performs average pooling and maximum pooling on the image features respectively, passes the two pooled features through a multi-layer neural network, and finally adds the two features together to obtain the feature weight through the activation function and multiply it with the input feature map.

4. The eye acupoint localization method based on modular feature-enhanced GAN according to claim 3, characterized in that: The spatial enhancement module sequentially processes the input features through average pooling and maximum pooling, then performs convolution operations, and finally uses the activation function to obtain the feature weights and multiply them with the input feature map.

5. The eye acupoint localization method based on modular feature-enhanced GAN according to claim 1, characterized in that: The specific process of constructing the paired eye acupoint image dataset in step 1) is: capturing a human facial image through a camera, and manually marking acupoints on the facial image to form an acupoint calibration image.

Images