Model training method, and periorbital acupoint auxiliary positioning method and device

Abstract

The invention provides a model training method and a periorbital acupoint auxiliary positioning method and device, belongs to the technical field of deep learning, and aims to solve the problems that current periorbital acupoint positioning depends on physician experience and is low in positioning accuracy, periorbital images of different periorbital models can be used, and the positioning accuracy of the periorbital acupoints is improved through two stages of model training processes. Training the first two-way convolutional neural network model to obtain a target periocular acupoint auxiliary positioning model, and performing auxiliary positioning on a target periocular color image of a target patient through the target periocular acupoint auxiliary positioning model, so that the difference of individual periocular models can be fully considered, excessive dependence on subjective experience of a physician is avoided, and the accuracy of the target periocular acupoint auxiliary positioning model is improved. And the accuracy of positioning the periorbital acupoints of different patients can be improved to a certain extent.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a model training method, an eye periorbital acupoint assisted positioning method and device. Background Technology

[0002] Acupoints around the eyes are an important part of traditional Chinese medicine acupuncture. Stimulating these acupoints can regulate local blood and qi circulation, improve visual function, and treat various eye and systemic diseases. Stimulating acupoints around the eyes can effectively treat corresponding eye conditions. For example, for acute styes, pricking the Yuyao acupoint combined with cupping can significantly improve the therapeutic effect compared to medication alone. However, if the stimulation points are not accurately located, the treatment will be ineffective. Therefore, accurately locating acupoints around the eyes has become a worthy area of ​​research.

[0003] Currently, the location of acupoints around the eyes mainly relies on the physician's experience, using a general acupoint model for actual positioning. This method does not take into account the discrepancy between the general model and the actual patient's periorbital model, and depends primarily on the physician's own positioning experience, resulting in low accuracy in locating acupoints around the eyes. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a model training method, an eye periorbital acupoint assisted positioning method, a device, an electronic device, and a readable storage medium to overcome or at least partially solve the above problems.

[0005] In a first aspect, embodiments of this application provide a model training method, the method comprising: Obtain sample true-color periorbital images and sample true-color periorbital annotated images from sample patients; wherein, the sample true-color periorbital annotated images are obtained by experts annotating the sample true-color periorbital images; Generate sample grayscale periorbital images of the sample true-color periorbital images and sample grayscale periorbital annotation images of the sample true-color periorbital images, respectively. Based on the sample grayscale periorbital image and the sample grayscale periorbital labeled image, the first dual-path convolutional neural network model is trained in the first stage to obtain the second dual-path convolutional neural network model. Based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image, the second dual-path convolutional neural network model is trained in the second stage to obtain the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is used to output the target periorbital acupoint labeled image based on the target patient's target periorbital color image.

[0006] Optionally, the step of performing a first-stage model training on the first dual-path convolutional neural network model based on the sample grayscale peri-eye image and the sample grayscale peri-eye labeled image to obtain a second dual-path convolutional neural network model includes: The sample grayscale periorbital image is input into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model; Based on the first periorbital acupoint location prediction image and the sample grayscale periorbital labeled image, the first model loss value of the first dual-path convolutional neural network model is determined. Based on the first model loss value, the first network module parameters of the first convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model are adjusted to obtain a second dual-path convolutional neural network model; wherein, the first convolutional neural network module includes the first input terminal, and the first output terminal of the first convolutional neural network module is connected to the first input terminal of the fully connected network module.

[0007] Optionally, the first convolutional neural network module includes a gray-level co-occurrence matrix generation submodule, a fusion submodule, and a convolutional neural network submodule. The step of inputting the sample gray-level periorbital image into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model includes: The sample grayscale periorbital image is input into the grayscale co-occurrence matrix generation submodule through the first input terminal to obtain the 0° grayscale co-occurrence matrix, 45° grayscale co-occurrence matrix, 90° grayscale co-occurrence matrix and 135° grayscale co-occurrence matrix output by the grayscale co-occurrence matrix generation submodule; The 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix, and the 135° gray-level co-occurrence matrix are input into the fusion submodule for fusion and splicing to obtain the first splicing matrix; The first stitching matrix and the sample grayscale periorbital image are input into the convolutional neural network submodule to obtain the first periorbital acupoint localization prediction image output by the fully connected network module connected to the convolutional neural network submodule.

[0008] Optionally, determining the first model loss value of the first dual-path convolutional neural network model based on the first periorbital acupoint location prediction image and the sample grayscale periorbital labeled image includes: The first periorbital acupoint location prediction image and the sample grayscale periorbital labeled image are subtracted in the same direction to obtain the first difference image; For the gray-level markers in the first difference image, the gray-level markers are matched with the minimum distance to obtain multiple gray-level marker pairs; Calculate the first distance between the grayscale markers contained in each grayscale marker pair and sum the first distances to obtain the first sum value; The first sum is used as the independent variable to input the sigmoid function for calculation, thereby obtaining the first model loss value of the first dual-path convolutional neural network model.

[0009] Optionally, the second stage of training the second dual-path convolutional neural network model based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image to obtain the target periorbital acupoint auxiliary localization model includes: The sample true-color periorbital image is input into the second dual-path convolutional neural network model through the second input terminal, and the sample grayscale periorbital image is input into the second dual-path convolutional neural network model through the first input terminal to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model; wherein, the sample true-color periorbital image is obtained by marking some acupoints on the original periorbital image of the sample patient; Based on the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image, the second model loss value and the third model loss value of the second dual-path convolutional neural network model are determined; wherein, the calculation methods of the second model loss value and the third model loss value are different; Based on the second model loss value and the third model loss value, the parameters of the third network module of the second convolutional neural network module and the parameters of the second network module of the fully connected network module of the second dual-path convolutional neural network model are adjusted to obtain the target periorbital acupoint assisted positioning model.

[0010] Optionally, determining the second and third model loss values ​​of the second dual-path convolutional neural network model based on the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image includes: The second periorbital acupoint location prediction image and the sample true-color periorbital labeled image are subtracted to obtain the second difference image; The second difference image is clustered by pixel points to obtain multiple clusters; Using straight line segments, connect the cluster centers of each cluster in a clockwise direction to obtain a closed connecting line; The first length of the closed connection line is calculated, and the first length is determined as the second model loss value of the second dual-path convolutional neural network model. Perform Z-code transformation on the second difference image to obtain the first difference Z-code; The first difference Z-encoded value is input into the sigmoid function for calculation to obtain the third model loss value of the second dual-path convolutional neural network model.

[0011] Optionally, the step of adjusting the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model based on the second model loss value and the third model loss value to obtain the target periorbital acupoint assisted localization model includes: Based on the second model loss value and the third model loss value, the parameters of the third network module of the second convolutional neural network module and the parameters of the second network module of the fully connected network module of the second dual-path convolutional neural network model are adjusted once to obtain the parameter-adjusted second dual-path convolutional neural network model. The process of inputting the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously inputting the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model, is repeated to begin the next training. The number of consecutive occurrences of the second model's loss value being less than the first threshold and the number of consecutive occurrences of the third model's loss value being less than the second threshold are counted. If the first number of times is greater than or equal to the first preset number of times, and the second number of times is greater than or equal to the first preset number of times, stop adjusting the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model, and obtain the target periorbital acupoint auxiliary positioning model.

[0012] Secondly, embodiments of this application provide a method for assisted location of acupoints around the eyes, the method comprising: Acquire target periorbital color images of the target patient; Generate a grayscale image of the target periorbital area corresponding to the target periorbital color image; The target periorbital color image and the target periorbital grayscale image are input into the target periorbital acupoint auxiliary localization model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is obtained based on any of the model training methods described above.

[0013] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising: The first acquisition module is used to acquire a sample true-color periorbital image and a sample true-color periorbital annotated image of the sample patient; wherein, the sample true-color periorbital annotated image is obtained by an expert marking the sample true-color periorbital image; The first generation module is used to generate a sample grayscale periorbital image of the sample true-color periorbital image and a sample grayscale periorbital annotation image of the sample true-color periorbital image. The first training module is used to perform a first-stage model training on the first dual-path convolutional neural network model based on the sample grayscale peri-eye image and the sample grayscale peri-eye labeled image, so as to obtain a second dual-path convolutional neural network model. The second training module is used to perform a second-stage model training on the second dual-path convolutional neural network model based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image, to obtain a target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is used to output a target periorbital acupoint labeled image based on the target patient's target periorbital color image.

[0014] Optionally, the first training module includes: The first input / output submodule is used to input the sample grayscale periorbital image into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint positioning prediction image output by the first dual-path convolutional neural network model. The first determining submodule is used to determine the first model loss value of the first dual-path convolutional neural network model based on the first periorbital acupoint positioning prediction image and the sample grayscale periorbital labeled image. The first adjustment submodule is used to adjust the first network module parameters of the first path convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model based on the first model loss value, so as to obtain a second dual-path convolutional neural network model; wherein, the first path convolutional neural network module includes the first input terminal, and the first output terminal of the first path convolutional neural network module is connected to the first input terminal of the fully connected network module.

[0015] Optionally, the first convolutional neural network module includes a gray-level co-occurrence matrix generation submodule, a fusion submodule, and a convolutional neural network submodule, and the first input-output submodule includes: The first input / output unit is used to input the sample grayscale periorbital image into the grayscale co-occurrence matrix generation submodule through the first input terminal to obtain the 0° grayscale co-occurrence matrix, 45° grayscale co-occurrence matrix, 90° grayscale co-occurrence matrix and 135° grayscale co-occurrence matrix output by the grayscale co-occurrence matrix generation submodule; The fusion unit is used to input the 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix and the 135° gray-level co-occurrence matrix into the fusion submodule for fusion and splicing to obtain the first splicing matrix; The second input / output unit is used to input the first splicing matrix and the sample grayscale periorbital image into the convolutional neural network submodule to obtain the first periorbital acupoint localization prediction image output by the fully connected network module connected to the convolutional neural network submodule.

[0016] Optionally, the first determining submodule includes: The first image subtraction unit is used to subtract the first periorbital acupoint positioning prediction image and the sample grayscale periorbital annotation image in the same direction to obtain the first difference image. The matching unit is used to match each gray-level marker point in the first difference image with the minimum distance to obtain multiple gray-level marker point pairs; The first calculation unit is used to calculate the first distance between the grayscale marker points contained in each grayscale marker point pair and sum the first distances to obtain a first sum value; The second calculation unit is used to input the first sum as an independent variable into the sigmoid function to calculate the first model loss value of the first dual-path convolutional neural network model.

[0017] Optionally, the second training module includes: The second input / output submodule is used to input the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously input the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model; wherein, the sample true-color periorbital image is obtained by marking some acupoints on the original periorbital image of the sample patient; The second determining submodule is used to determine the second model loss value and the third model loss value of the second dual-path convolutional neural network model based on the second periorbital acupoint positioning prediction image and the sample true-color periorbital labeled image; wherein the calculation methods of the second model loss value and the third model loss value are different; The second adjustment submodule is used to adjust the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model based on the second model loss value and the third model loss value, so as to obtain the target periorbital acupoint auxiliary positioning model.

[0018] Optionally, the second determining submodule includes: The second image subtraction unit is used to subtract the second periorbital acupoint location prediction image from the sample true-color periorbital annotation image to obtain the second difference image. A clustering unit is used to cluster the second difference image according to pixels to obtain multiple clusters; A connecting unit is used to connect the cluster centers of each cluster in a clockwise direction using straight line segments to obtain a closed connecting line; A statistical determination unit is used to calculate the first length of the closed connection line and determine the first length as the second model loss value of the second dual-path convolutional neural network model. Z-coding transformation unit is used to perform Z-coding transformation on the second difference image to obtain the first difference Z-code; The third calculation unit is used to input the first difference Z-encoded input into the sigmoid function for calculation to obtain the third model loss value of the second dual-path convolutional neural network model.

[0019] Optionally, the second adjustment submodule includes: The adjustment unit is used to adjust the parameters of the third network module of the second convolutional neural network module and the parameters of the second network module of the fully connected network module of the second dual-path convolutional neural network model once based on the second model loss value and the third model loss value, so as to obtain the parameter-adjusted second dual-path convolutional neural network model. The re-execution unit is used to re-execute the step of inputting the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously inputting the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model, so as to start the next training. The statistical unit is used to count the first number of times the loss value of the second model is less than the first threshold consecutively and the second number of times the loss value of the third model is less than the second threshold consecutively. The stop adjustment unit is used to stop adjusting the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model when the first number of times is greater than or equal to the first preset number of times and the second number of times is greater than or equal to the first preset number of times, so as to obtain the target periorbital acupoint auxiliary positioning model.

[0020] Fourthly, embodiments of this application provide an auxiliary positioning device for acupoints around the eyes, the device comprising: The second acquisition module is used to acquire a color image of the target periocular area of ​​the target patient. The second generation module is used to generate a target peri-eye grayscale image corresponding to the target peri-eye color image; The input / output module is used to input the target periorbital color image and the target periorbital grayscale image into the target periorbital acupoint auxiliary localization model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is obtained based on any of the model training methods described above.

[0021] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the model training method as described above, or the periorbital acupoint assisted positioning method.

[0022] Sixthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the model training method described above, or the periorbital acupoint assisted positioning method.

[0023] The specific beneficial effects are as follows: This application embodiment acquires sample true-color periocular images and sample true-color periocular annotated images from sample patients. The sample true-color periocular annotated images are obtained by experts annotating the sample true-color periocular images. Sample grayscale periocular images of the sample true-color periocular images and sample grayscale periocular annotated images of the sample true-color periocular images are generated respectively. Based on the sample grayscale periocular images and sample grayscale periocular annotated images, a first-stage model training is performed on a first dual-path convolutional neural network model to obtain a second dual-path convolutional neural network model. Based on the sample true-color periocular images, sample grayscale periocular images, and sample true-color periocular annotated images, the second dual-path convolutional neural network model is further trained... The second stage of model training is performed to obtain the target periorbital acupoint auxiliary localization model. This model is used to output target periorbital acupoint marker images based on the target patient's target periorbital color image. Periorbital images from different periorbital models can be used. Through the two-stage model training process, the first dual-path convolutional neural network model is trained to obtain the target periorbital acupoint auxiliary localization model. This model can be used to assist in the localization of the target patient's target periorbital color image. It can fully consider the differences in individual periorbital models, avoid over-reliance on the physician's subjective experience, and improve the accuracy of periorbital acupoint localization for different patients to a certain extent. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of a model training method provided in an embodiment of this application; Figure 2 A flowchart illustrating another model training method provided in an embodiment of this application; Figure 3 A flowchart illustrating an auxiliary method for locating acupoints around the eyes, provided in an embodiment of this application; Figure 4 This is a schematic diagram of a true-color periorbital image of a sample provided in an embodiment of this application; Figure 5 This is a comparative schematic diagram of the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image provided in the embodiments of this application; Figure 6 This is a template diagram of the standard location of acupoints provided in an embodiment of this application; Figure 7 This is a logic block diagram of a model training device provided in an embodiment of this application; Figure 8 This is a logic block diagram of an eye acupoint auxiliary positioning device provided in an embodiment of this application; Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0027] Reference Figure 1 , Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of this application. The method may include: Step 101: Obtain a sample true-color periorbital image and a sample true-color periorbital annotated image from the sample patient; wherein the sample true-color periorbital annotated image is obtained by an expert annotating the sample true-color periorbital image.

[0028] In the embodiments of this application, images of the periocular area can be screened from those obtained from ophthalmology hospitals. After desensitization processing to remove patients' personal privacy information, patient numbers are added to obtain sample true-color images of the periocular area. Then, the sample patients are distinguished by these numbers (e.g., number 1, 2, 3, etc.). Next, acupoints are annotated on the sample true-color images of the periocular area by a traditional Chinese medicine ophthalmologist to add acupoint markers, resulting in an annotated true-color image of the periocular area.

[0029] Step 102: Generate a sample grayscale periorbital image of the sample true-color periorbital image and a sample grayscale periorbital annotation image of the sample true-color periorbital image.

[0030] In the embodiments of this application, the sample true-color periorbital image and the sample true-color periorbital annotated image can be grayscaled respectively to obtain the sample grayscale periorbital image of the sample true-color periorbital image and the sample grayscale periorbital annotated image of the sample true-color periorbital image.

[0031] Step 103: Based on the sample grayscale periorbital image and the sample grayscale periorbital labeled image, perform the first stage of model training on the first dual-path convolutional neural network model to obtain the second dual-path convolutional neural network model.

[0032] In the embodiments of this application, sample grayscale periorbital annotated images can be used as image labels for sample grayscale periorbital images. Based on the sample grayscale periorbital images and sample grayscale periorbital annotated images, a first-stage model training is performed on a first dual-path convolutional neural network model to obtain a second dual-path convolutional neural network model. The first dual-path convolutional neural network model can include two convolutional neural networks for data input and a fully connected network for data fusion, with the outputs of both convolutional neural networks connected to the input of the fully connected network. In the first stage, only the network parameters of the first convolutional neural network and the fully connected network of the first dual-path convolutional neural network model used for inputting the sample grayscale periorbital images are adjusted. The learning rate can be set to 0.001, and the epoch can be set to 1000.

[0033] Step 104: Based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image, perform a second-stage model training on the second dual-path convolutional neural network model to obtain a target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is used to output a target periorbital acupoint labeled image based on the target patient's target periorbital color image.

[0034] In the embodiments of this application, a second-stage model training can be performed on a second dual-path convolutional neural network model based on sample true-color periorbital images, sample grayscale periorbital images, and sample true-color periorbital labeled images, thereby obtaining a target periorbital acupoint auxiliary localization model. The sample grayscale periorbital images can maintain the input path of the first-stage training unchanged, while the sample true-color periorbital images can be input from the other path of the second dual-path convolutional neural network model. Finally, the periorbital acupoint localization prediction image is output from the output layer of the fully connected network (continuously inputting sample grayscale periorbital images during this process enhances the learning effect of the second-stage model network). In the second-stage model training, the model loss value can be calculated using the sample true-color periorbital images and sample true-color periorbital labeled images. Simultaneously, based on this model loss value, the network parameters of the first path convolutional neural network, the second path convolutional neural network, and the fully connected network can be adjusted. The learning rate for the first path convolutional neural network can be set to 0.00001, the learning rate for the second path convolutional neural network and the fully connected network can be set to 0.0001, and epoch=10000 is set. This allows for a lower degree of adjustment to the network parameters of the first convolutional neural network during the second stage of training, keeping it in a fine-tuning state. This can, to some extent, avoid the phenomenon of gradient explosion in model parameters and improve the efficiency of model training.

[0035] In the embodiments of this application, sample true-color periocular images and sample true-color periocular annotated images are obtained from sample patients. The sample true-color periocular annotated images are obtained by experts annotating the sample true-color periocular images. Sample grayscale periocular images of the sample true-color periocular images and sample grayscale periocular annotated images of the sample true-color periocular images are generated respectively. Based on the sample grayscale periocular images and sample grayscale periocular annotated images, a first-stage model training is performed on a first dual-path convolutional neural network model to obtain a second dual-path convolutional neural network model. Based on the sample true-color periocular images, sample grayscale periocular images, and sample true-color periocular annotated images, the second dual-path convolutional neural network model is further trained. The model undergoes a second-stage training process to obtain a target periorbital acupoint auxiliary localization model. This model outputs acupoint marker images based on the target patient's periorbital color image. Different periorbital images can be used. Through a two-stage model training process, the first dual-path convolutional neural network model is trained to obtain the target periorbital acupoint auxiliary localization model. This model can then be used to assist in the localization of the target patient's periorbital color image. It fully considers the differences in individual periorbital models, avoids over-reliance on the physician's subjective experience, and can improve the accuracy of periorbital acupoint localization for different patients to a certain extent.

[0036] Reference Figure 2 ,Figure 2 This is a flowchart illustrating another model training method provided in an embodiment of this application. The method may include: Step 201: Obtain the sample true-color periorbital image and the sample true-color periorbital annotated image of the sample patient.

[0037] Step 202: Generate a sample grayscale periorbital image of the sample true-color periorbital image and a sample grayscale periorbital annotation image of the sample true-color periorbital image.

[0038] In the embodiments of this application, the implementation of steps 201 and 202 can be referred to the embodiments of steps 101 and 102, and will not be repeated here.

[0039] Step 203: Input the sample grayscale periorbital image into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model.

[0040] In embodiments of this application, a sample grayscale periorbital image can be input into a first dual-path convolutional neural network model via a first input terminal, thereby obtaining a first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model. The first input terminal can be the input layer of the first path of the convolutional neural network in the first dual-path convolutional neural network model.

[0041] Optionally, step 203 may include the following sub-steps: Sub-step 2031: Input the sample grayscale periorbital image into the grayscale co-occurrence matrix generation sub-module through the first input terminal to obtain the 0° grayscale co-occurrence matrix, 45° grayscale co-occurrence matrix, 90° grayscale co-occurrence matrix and 135° grayscale co-occurrence matrix output by the grayscale co-occurrence matrix generation sub-module.

[0042] In the embodiments of this application, considering that the core of acupoint localization is based on the relative position of the acupoint in the eye image, specifically the distance between the acupoint pixel and the near and far pixels, a gray-level co-occurrence matrix can be used to enhance image representation. Therefore, a gray-level co-occurrence matrix generation submodule can be added before the basic convolutional neural network submodule to output 0°, 45°, 90°, and 135° gray-level co-occurrence matrices after inputting a gray-level sample periorbital image.

[0043] Sub-step 2032: Input the 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix and the 135° gray-level co-occurrence matrix into the fusion sub-module for fusion and splicing to obtain the first splicing matrix.

[0044] In the embodiments of this application, to improve the feature correlation between gray-level co-occurrence matrices at different angles, a fusion submodule can be used to fuse and stitch together the 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix, and the 135° gray-level co-occurrence matrix to obtain a first stitched matrix. The number of rows in the first stitched matrix is ​​the same as the number of rows in any of the gray-level co-occurrence matrices, and the number of columns in the first stitched matrix can be equal to the sum of the number of columns in the 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix, and the 135° gray-level co-occurrence matrix.

[0045] Sub-step 2033: Input the first splicing matrix and the sample grayscale periorbital image into the convolutional neural network sub-module to obtain the first periorbital acupoint localization prediction image output by the fully connected network module connected to the convolutional neural network sub-module.

[0046] In the embodiments of this application, the first stitching matrix and the sample grayscale periorbital image are of different data types. Therefore, the first stitching matrix and the sample grayscale periorbital image can be input together into the convolutional neural network submodule to obtain the first periorbital acupoint localization prediction image output by the fully connected network module connected to the convolutional neural network submodule. The first periorbital acupoint localization prediction image is a grayscale image, but its acupoint localization prediction marker is red (pre-set).

[0047] Step 204: Based on the first periorbital acupoint localization prediction image and the sample grayscale periorbital labeled image, determine the first model loss value of the first dual-path convolutional neural network model.

[0048] In embodiments of this application, a first model loss value for the first dual-path convolutional neural network model can be calculated based on the first periorbital acupoint location prediction image and the sample grayscale periorbital annotation image. For example, the image similarity between the first periorbital acupoint location prediction image and the sample grayscale periorbital annotation image can be used as the first model loss value.

[0049] Optionally, step 204 may include the following sub-steps: Sub-step 2041 involves subtracting the first periorbital acupoint location prediction image from the sample grayscale periorbital annotation image in the same direction to obtain the first difference image.

[0050] Sub-step 2042: For grayscale markers in the first difference image, match each grayscale marker with the minimum distance to obtain multiple grayscale marker pairs.

[0051] Sub-step 2043: Calculate the first distance between the grayscale markers contained in each grayscale marker pair and sum the first distances to obtain the first sum value.

[0052] Sub-step 2044: The first sum is used as an independent variable to input the sigmoid function for calculation, thereby obtaining the first model loss value of the first dual-path convolutional neural network model.

[0053] In the embodiments of this application, considering that the accuracy of image similarity as the first model loss value is not high, a novel method for calculating the first model loss value can be designed according to the actual situation of this application. Since acupoint location prediction is a multi-point prediction, the location prediction deviation of multiple points can be considered. Through the process of sub-steps 2041 to 2044, the sum of the location prediction deviations of each periorbital acupoint can be calculated, and the first model loss value can be obtained by encoding and conversion through the sigmoid function. By subtracting the first periorbital acupoint location prediction image and the sample grayscale periorbital annotation image in the same direction, a first difference image is obtained. For the grayscale markers in the first difference image, each grayscale marker is matched with the minimum distance to obtain multiple grayscale marker pairs. The first distance between the grayscale markers contained in each grayscale marker pair is calculated and the first distance is summed to obtain a first sum value. The first sum value is used as the independent variable and input into the sigmoid function for calculation to obtain the first model loss value of the first dual-path convolutional neural network model. This can improve the accuracy of the first model loss value and improve the training efficiency of the first stage of model training.

[0054] Step 205: Based on the first model loss value, adjust the first network module parameters of the first convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model to obtain a second dual-path convolutional neural network model; wherein, the first convolutional neural network module includes the first input terminal, and the first output terminal of the first convolutional neural network module is connected to the first input terminal of the fully connected network module.

[0055] In the embodiments of this application, the first-path convolutional neural network module may include a first input terminal, a gray-level co-occurrence matrix generation submodule, a fusion submodule, and a convolutional neural network submodule, and the first output terminal of the first-path convolutional neural network module may be connected to the first input terminal of the fully connected network module. Based on this, the first network module parameters of the first-path convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model can be adjusted according to the first model loss value, thereby obtaining a second dual-path convolutional neural network model. The adjustment direction of the first and second network module parameters can be the direction that reduces the first model loss value. During adjustment, the first and second network module parameters can be adjusted multiple times. After each adjustment, step 203 can be re-executed to start the next training. When the number of training iterations exceeds the set number of epochs, the first stage of model training can be terminated, thereby obtaining the second dual-path convolutional neural network model.

[0056] In the embodiments of this application, by inputting sample grayscale periorbital images into a first dual-path convolutional neural network model through a first input terminal, a first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model is obtained. Based on the first periorbital acupoint localization prediction image and the sample grayscale periorbital labeled image, a first model loss value of the first dual-path convolutional neural network model is determined. Based on the first model loss value, the first network module parameters of the first path convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model are adjusted to obtain a second dual-path convolutional neural network model. The first path convolutional neural network module includes a first input terminal, and the first output terminal of the first path convolutional neural network module is connected to the first input terminal of the fully connected network module. The first model loss value can be used to perform the first stage of model training on the first dual-path convolutional neural network model, which can improve the training efficiency and reliability of the first stage of model training to a certain extent.

[0057] Step 206: Input the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously input the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model; wherein, the sample true-color periorbital image is obtained by marking some acupoints on the original periorbital image of the sample patient.

[0058] In the embodiments of this application, partial acupoint marking can be performed on the original periorbital images of sample patients to obtain sample true-color periorbital images. Simultaneously, the original periorbital image itself can also serve as a sample true-color periorbital image. The sample true-color periorbital image can be input into a second dual-path convolutional neural network model through a second input terminal, while the sample grayscale periorbital image is input into the second dual-path convolutional neural network model through a first input terminal, resulting in the output of the second dual-path convolutional neural network model: a second periorbital acupoint localization prediction image. This input process does not require ensuring that the simultaneously input sample true-color periorbital images and sample grayscale periorbital images correspond. The selection of the sample grayscale periorbital images can be random. Furthermore, the input sample true-color periorbital images can be adjusted according to the training progress. At the beginning of the second stage of training, sample true-color periorbital images with more markings (6 to 7) can be input. As the training progresses, images with fewer acupoint markings (3 to 4) can be input, until images without acupoint markings are input.

[0059] Step 207: Based on the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image, determine the second model loss value and the third model loss value of the second dual-path convolutional neural network model; wherein the calculation methods of the second model loss value and the third model loss value are different.

[0060] In the embodiments of this application, the second model loss value and the third model loss value of the second dual-path convolutional neural network model can be determined from two perspectives and using two calculation methods, based on the second periorbital acupoint positioning prediction image and the sample true-color periorbital labeled image.

[0061] Optionally, step 207 may include the following sub-steps: Sub-step 2071 involves subtracting the second periorbital acupoint location prediction image from the sample true-color periorbital labeled image to obtain the second difference image.

[0062] Sub-step 2072: Cluster the second difference image according to pixel points to obtain multiple clusters.

[0063] Sub-step 2073: Using straight line segments, connect the cluster centers of each cluster in a clockwise direction to obtain a closed connecting line.

[0064] Sub-step 2074: Calculate the first length of the closed connection line and determine the first length as the second model loss value of the second dual-path convolutional neural network model.

[0065] In the embodiments of this application, an image subtraction method, combined with clustering and mathematical graphing methods, can be used to obtain a closed connection line. The first length of this closed connection line is then used as the second model loss value of the second dual-path convolutional neural network model. It is understood that the smaller the positioning deviation between the positioning markers of the periorbital acupoints in the second periorbital acupoint localization prediction image and the sample true-color periorbital annotation image, the smaller the first length of the closed connection line, which in the extreme case is 0. Therefore, the first length of this closed connection line can be used as the second model loss value. Through sub-steps 2072 to 2074, the second model loss value can be calculated, which can improve the accuracy and reliability of the second model loss value to a certain extent.

[0066] Sub-step 2075: Perform Z-code transformation on the second difference image to obtain the first difference Z-code.

[0067] Sub-step 2076: Input the first difference Z-encoded value into the sigmoid function for calculation to obtain the third model loss value of the second dual-path convolutional neural network model.

[0068] In the embodiments of this application, the second difference image obtained after image subtraction can also undergo Z-coding transformation. Because the smaller the positioning deviation between the positioning markers of the periorbital acupoints in the second periorbital acupoint localization prediction image and the sample true-color periorbital annotation image, the more zeros are present in the first difference Z-code obtained after the second difference image undergoes Z-coding transformation. The result calculated by the sigmoid function will also be smaller. Therefore, the third model loss value obtained by the above method is also effective for model training. Through sub-steps 2075 and 2076, the third model loss value can be calculated in another way, which can improve the accuracy and reliability of the third model loss value to a certain extent.

[0069] Step 208: Based on the second model loss value and the third model loss value, adjust the third network module parameters of the second convolutional neural network module of the second dual-path convolutional neural network model and the second network module parameters of the fully connected network module to obtain the target periorbital acupoint assisted positioning model.

[0070] In the embodiments of this application, the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model can be adjusted collaboratively based on the second model loss value and the third model loss value, thereby obtaining the target periorbital acupoint assisted positioning model.

[0071] Optionally, step 208 may include the following sub-steps: Sub-step 2081: Based on the second model loss value and the third model loss value, adjust the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model once to obtain the parameter-adjusted second dual-path convolutional neural network model.

[0072] Sub-step 2082 involves re-executing the step of inputting the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously inputting the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model, in order to begin the next training.

[0073] Sub-step 2083: Count the number of times the loss value of the second model is less than the first threshold consecutively and the number of times the loss value of the third model is less than the second threshold consecutively.

[0074] Sub-step 2084: When the first number of times is greater than or equal to the first preset number of times and the second number of times is greater than or equal to the first preset number of times, stop adjusting the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model to obtain the target periorbital acupoint auxiliary positioning model.

[0075] In the embodiments of this application, the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model can be adjusted using the second model loss value and the third model loss value. The adjustment direction can be to reduce the second model loss value and the third model loss value. After adjustment, step 206 can be re-executed to start the next training. During the second stage of model training, the first number of consecutive occurrences of the second model loss value being less than the first threshold and the second number of consecutive occurrences of the third model loss value being less than the second threshold can be counted, where there is no correlation between the first threshold and the second threshold. When the second model loss value is greater than or equal to the first threshold, or when the third model loss value is greater than or equal to the second threshold, the first number and the second number can be immediately set to zero. If the first number is greater than or equal to the first preset number, and the second number is also greater than or equal to the first preset number, the adjustment of the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model can be stopped, thus obtaining the target periorbital acupoint auxiliary positioning model. Through sub-steps 2081 to 2084, the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model can be adjusted by coordinating the second and third model loss values. Training stops when both the second and third model loss values ​​meet the convergence condition, thus obtaining the target periorbital acupoint auxiliary positioning model. This can improve the training efficiency and accuracy of the second-stage model training to a certain extent.

[0076] In the embodiments of this application, a second periorbital acupoint localization prediction image is obtained by inputting a sample true-color periorbital image into a second dual-path convolutional neural network model through a second input terminal, and simultaneously inputting a sample grayscale periorbital image into the second dual-path convolutional neural network model through a first input terminal. The sample true-color periorbital image is obtained by partially marking acupoints on the original periorbital image of the sample patient. Based on the second periorbital acupoint localization prediction image and the sample true-color periorbital marked image, a second model loss value and a third model loss value are determined for the second dual-path convolutional neural network model. The calculation methods for the second and third model loss values ​​are different. Based on the second and third model loss values, the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model are adjusted to obtain a target periorbital acupoint auxiliary localization model. The second-stage model training of the second dual-path convolutional neural network model can be performed using the second and third model loss values, which can improve the acupoint prediction accuracy of the final target periorbital acupoint auxiliary localization model to a certain extent.

[0077] Reference Figure 3 , Figure 3 This is a flowchart illustrating an auxiliary method for locating acupoints around the eyes, provided in an embodiment of this application. The method may include: Step 301: Obtain a color image of the target periocular area of ​​the target patient.

[0078] In the embodiments of this application, a color image of the target periocular area of ​​the target patient can be obtained by capturing the image using a camera device.

[0079] Step 302: Generate a grayscale image of the target peri-eye corresponding to the target peri-eye color image.

[0080] In the embodiments of this application, the target peri-eye color image can be grayscale processed to obtain the target peri-eye grayscale image corresponding to the target peri-eye color image.

[0081] Step 303: Input the target periorbital color image and the target periorbital grayscale image into the target periorbital acupoint auxiliary localization model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is obtained based on any of the model training methods described above.

[0082] In embodiments of this application, a target periorbital color image and a target periorbital grayscale image can be input into a target periorbital acupoint auxiliary localization model to obtain a target periorbital acupoint marker image output by the target periorbital acupoint auxiliary localization model. During input, the target periorbital grayscale image is input through a first convolutional neural network, and the target periorbital color image is input through a second convolutional neural network. The target periorbital acupoint auxiliary localization model can be obtained based on any of the model training methods described above.

[0083] In the embodiments of this application, a target periorbital color image of the target patient is acquired, and a target periorbital grayscale image corresponding to the target periorbital color image is generated. The target periorbital color image and the target periorbital grayscale image are input into the target periorbital acupoint auxiliary positioning model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary positioning model. The target periorbital acupoint auxiliary positioning model is obtained based on any of the model training methods described above, which can quickly perform periorbital acupoint auxiliary positioning for the target patient. To a certain extent, it can avoid the uncertainty caused by the excessive reliance on the doctor's subjective experience in acupoint positioning, thereby improving the accuracy of periorbital acupoint auxiliary positioning to a certain extent.

[0084] refer to Figure 4 , Figure 5 and Figure 6 , Figure 4 This is a schematic diagram of a sample true-color periorbital image provided in an embodiment of this application.Figure 5 This is a comparative diagram of the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image provided in the embodiments of this application. Figure 6 This is a template diagram of the standard location of acupoints provided in an embodiment of this application. Figure 4 The top left sub-image is the original periorbital image obtained by taking the picture, while the rest are sample true-color periorbital images obtained by partially marking the acupoints around the eyes. Figure 4 Due to the shooting angle and hair obscuring the view, the temples are not marked. Figure 5 The upper part of the image is the predicted image of the second periorbital acupoints output during model training, and the lower part is the sample true-color periorbital labeled image. Figure 4 and Figure 5 In the text, the color of the acupoint marker is set to red (color value FF0000), and the marker size is 7px×7px. Figure 6 A template diagram showing the standard locations of acupoints is presented.

[0085] refer to Figure 7 , Figure 7 This is a logic block diagram of a model training device provided in an embodiment of this application. The model training device 700 may include: The first acquisition module 701 is used to acquire a sample true-color periorbital image and a sample true-color periorbital annotated image of the sample patient; wherein, the sample true-color periorbital annotated image is obtained by an expert marking the sample true-color periorbital image; The first generation module 702 is used to generate a sample grayscale periorbital image of the sample true-color periorbital image and a sample grayscale periorbital annotation image of the sample true-color periorbital image. The first training module 703 is used to perform a first-stage model training on the first dual-path convolutional neural network model based on the sample grayscale peri-eye image and the sample grayscale peri-eye labeled image, so as to obtain a second dual-path convolutional neural network model. The second training module 704 is used to perform a second-stage model training on the second dual-path convolutional neural network model based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image, to obtain a target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is used to output a target periorbital acupoint labeled image based on the target patient's target periorbital color image.

[0086] Optionally, the first training module 703 includes: The first input / output submodule is used to input the sample grayscale periorbital image into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint positioning prediction image output by the first dual-path convolutional neural network model. The first determining submodule is used to determine the first model loss value of the first dual-path convolutional neural network model based on the first periorbital acupoint positioning prediction image and the sample grayscale periorbital labeled image. The first adjustment submodule is used to adjust the first network module parameters of the first path convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model based on the first model loss value, so as to obtain a second dual-path convolutional neural network model; wherein, the first path convolutional neural network module includes the first input terminal, and the first output terminal of the first path convolutional neural network module is connected to the first input terminal of the fully connected network module.

[0087] Optionally, the first convolutional neural network module includes a gray-level co-occurrence matrix generation submodule, a fusion submodule, and a convolutional neural network submodule, and the first input-output submodule includes: The first input / output unit is used to input the sample grayscale periorbital image into the grayscale co-occurrence matrix generation submodule through the first input terminal to obtain the 0° grayscale co-occurrence matrix, 45° grayscale co-occurrence matrix, 90° grayscale co-occurrence matrix and 135° grayscale co-occurrence matrix output by the grayscale co-occurrence matrix generation submodule; The fusion unit is used to input the 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix and the 135° gray-level co-occurrence matrix into the fusion submodule for fusion and splicing to obtain the first splicing matrix; The second input / output unit is used to input the first splicing matrix and the sample grayscale periorbital image into the convolutional neural network submodule to obtain the first periorbital acupoint localization prediction image output by the fully connected network module connected to the convolutional neural network submodule.

[0088] Optionally, the first determining submodule includes: The first image subtraction unit is used to subtract the first periorbital acupoint positioning prediction image and the sample grayscale periorbital annotation image in the same direction to obtain the first difference image. The matching unit is used to match each gray-level marker point in the first difference image with the minimum distance to obtain multiple gray-level marker point pairs; The first calculation unit is used to calculate the first distance between the grayscale marker points contained in each grayscale marker point pair and sum the first distances to obtain a first sum value; The second calculation unit is used to input the first sum as an independent variable into the sigmoid function to calculate the first model loss value of the first dual-path convolutional neural network model.

[0089] Optionally, the second training module 704 includes: The second input / output submodule is used to input the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously input the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model; wherein, the sample true-color periorbital image is obtained by marking some acupoints on the original periorbital image of the sample patient; The second determining submodule is used to determine the second model loss value and the third model loss value of the second dual-path convolutional neural network model based on the second periorbital acupoint positioning prediction image and the sample true-color periorbital labeled image; wherein the calculation methods of the second model loss value and the third model loss value are different; The second adjustment submodule is used to adjust the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model based on the second model loss value and the third model loss value, so as to obtain the target periorbital acupoint auxiliary positioning model.

[0090] Optionally, the second determining submodule includes: The second image subtraction unit is used to subtract the second periorbital acupoint location prediction image from the sample true-color periorbital annotation image to obtain the second difference image. A clustering unit is used to cluster the second difference image according to pixels to obtain multiple clusters; A connecting unit is used to connect the cluster centers of each cluster in a clockwise direction using straight line segments to obtain a closed connecting line; A statistical determination unit is used to calculate the first length of the closed connection line and determine the first length as the second model loss value of the second dual-path convolutional neural network model. Z-coding transformation unit is used to perform Z-coding transformation on the second difference image to obtain the first difference Z-code; The third calculation unit is used to input the first difference Z-encoded input into the sigmoid function for calculation to obtain the third model loss value of the second dual-path convolutional neural network model.

[0091] Optionally, the second adjustment submodule includes: The adjustment unit is used to adjust the parameters of the third network module of the second convolutional neural network module and the parameters of the second network module of the fully connected network module of the second dual-path convolutional neural network model once based on the second model loss value and the third model loss value, so as to obtain the parameter-adjusted second dual-path convolutional neural network model. The re-execution unit is used to re-execute the step of inputting the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously inputting the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model, so as to start the next training. The statistical unit is used to count the first number of times the loss value of the second model is less than the first threshold consecutively and the second number of times the loss value of the third model is less than the second threshold consecutively. The stop adjustment unit is used to stop adjusting the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model when the first number of times is greater than or equal to the first preset number of times and the second number of times is greater than or equal to the first preset number of times, so as to obtain the target periorbital acupoint auxiliary positioning model.

[0092] The model training device in this application embodiment can be integrated into an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices besides a terminal. For example, the electronic device can be a GPU box, a mobile phone, a robot, an ultra-mobile personal computer (UMPC), or a personal digital assistant (PDA), etc., and can also be a server, a personal computer (PC), etc. This application embodiment does not specifically limit the specific implementation.

[0093] The model training device in this application embodiment can be a device with an operating system. This operating system can be Android, Linux, Windows, or other possible operating systems; this application embodiment does not specifically limit it.

[0094] The model training device provided in this application embodiment can achieve... Figure 1 and Figure 2 The various processes implemented in the method embodiments will not be described again here to avoid repetition. Reference Figure 8 , Figure 8 This application provides a logic block diagram of an eye area acupoint auxiliary positioning device 800, which may include: The second acquisition module 801 is used to acquire a target periorbital color image of the target patient; The second generation module 802 is used to generate a target peri-eye grayscale image corresponding to the target peri-eye color image; The input / output module 803 is used to input the target periorbital color image and the target periorbital grayscale image into the target periorbital acupoint auxiliary positioning model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary positioning model; wherein, the target periorbital acupoint auxiliary positioning model is obtained based on any of the model training methods described above.

[0095] The periorbital acupoint auxiliary positioning device in this application embodiment can be integrated into an electronic device, or a component of an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices besides a terminal. For example, the electronic device can be a GPU BOX, mobile phone, tablet computer, laptop computer, PDA, robot, ultra-mobile personal computer (UMPC), or personal digital assistant (PDA), etc., and can also be a server, personal computer (PC), etc. This application embodiment does not specifically limit the specific device.

[0096] The periorbital acupoint auxiliary positioning device in this application embodiment can be a device with an operating system. This operating system can be Android, Linux, Windows, or other possible operating systems; this application embodiment does not specifically limit it.

[0097] The periorbital acupoint auxiliary positioning device provided in this application embodiment can achieve… Figure 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0098] This application provides an electronic device, see [link to relevant documentation] Figure 9 The electronic device 90 includes: a processor 901, a memory 902, and a computer program 9021 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the program, it implements the model training method of the aforementioned embodiments, or the acupoint-assisted positioning method around the eyes.

[0099] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the model training method or the periorbital acupoint assisted positioning method disclosed in this application.

[0100] This application also provides a computer program product that, when run on an electronic device, enables the processor to implement the steps of the model training method or the periorbital acupoint assisted positioning method disclosed in this application.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0102] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0106] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0107] The above provides a detailed description of the model training method, the method and device for assisting in the localization of acupoints around the eyes provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain sample true-color periorbital images and sample true-color periorbital annotated images from sample patients; wherein, the sample true-color periorbital annotated images are obtained by experts annotating the sample true-color periorbital images; Generate sample grayscale periorbital images of the sample true-color periorbital images and sample grayscale periorbital annotation images of the sample true-color periorbital images, respectively. Based on the sample grayscale periorbital image and the sample grayscale periorbital labeled image, the first dual-path convolutional neural network model is trained in the first stage to obtain the second dual-path convolutional neural network model. Based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image, the second dual-path convolutional neural network model is trained in the second stage to obtain the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is used to output the target periorbital acupoint labeled image based on the target patient's target periorbital color image.

2. The method according to claim 1, characterized in that, The process of training a first-stage model of a first dual-path convolutional neural network model based on the sample grayscale peri-eye image and the sample grayscale peri-eye labeled image to obtain a second dual-path convolutional neural network model includes: The sample grayscale periorbital image is input into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model; Based on the first periorbital acupoint location prediction image and the sample grayscale periorbital labeled image, the first model loss value of the first dual-path convolutional neural network model is determined. Based on the first model loss value, the first network module parameters of the first convolutional neural network module and the second network module parameters of the fully connected network module of the first dual-path convolutional neural network model are adjusted to obtain a second dual-path convolutional neural network model; wherein, the first convolutional neural network module includes the first input terminal, and the first output terminal of the first convolutional neural network module is connected to the first input terminal of the fully connected network module.

3. The method according to claim 2, characterized in that, The first convolutional neural network module includes a gray-level co-occurrence matrix generation submodule, a fusion submodule, and a convolutional neural network submodule. The step of inputting the sample gray-level periorbital image into the first dual-path convolutional neural network model through the first input terminal to obtain the first periorbital acupoint localization prediction image output by the first dual-path convolutional neural network model includes: The sample grayscale periorbital image is input into the grayscale co-occurrence matrix generation submodule through the first input terminal to obtain the 0° grayscale co-occurrence matrix, 45° grayscale co-occurrence matrix, 90° grayscale co-occurrence matrix and 135° grayscale co-occurrence matrix output by the grayscale co-occurrence matrix generation submodule; The 0° gray-level co-occurrence matrix, the 45° gray-level co-occurrence matrix, the 90° gray-level co-occurrence matrix, and the 135° gray-level co-occurrence matrix are input into the fusion submodule for fusion and splicing to obtain the first splicing matrix; The first stitching matrix and the sample grayscale periorbital image are input into the convolutional neural network submodule to obtain the first periorbital acupoint localization prediction image output by the fully connected network module connected to the convolutional neural network submodule.

4. The method according to claim 2, characterized in that, The step of determining the first model loss value of the first dual-path convolutional neural network model based on the first periorbital acupoint location prediction image and the sample grayscale periorbital labeled image includes: The first periorbital acupoint location prediction image and the sample grayscale periorbital labeled image are subtracted in the same direction to obtain the first difference image; For the gray-level markers in the first difference image, the gray-level markers are matched with the minimum distance to obtain multiple gray-level marker pairs; Calculate the first distance between the grayscale markers contained in each grayscale marker pair and sum the first distances to obtain the first sum value; The first sum is used as the independent variable to input the sigmoid function for calculation, thereby obtaining the first model loss value of the first dual-path convolutional neural network model.

5. The method according to claim 1, characterized in that, The second dual-path convolutional neural network model is trained in the second stage based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image to obtain a target periorbital acupoint auxiliary localization model, including: The sample true-color periorbital image is input into the second dual-path convolutional neural network model through the second input terminal, and the sample grayscale periorbital image is input into the second dual-path convolutional neural network model through the first input terminal to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model; wherein, the sample true-color periorbital image is obtained by marking some acupoints on the original periorbital image of the sample patient; Based on the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image, the second model loss value and the third model loss value of the second dual-path convolutional neural network model are determined; wherein, the calculation methods of the second model loss value and the third model loss value are different; Based on the second model loss value and the third model loss value, the parameters of the third network module of the second convolutional neural network module and the parameters of the second network module of the fully connected network module of the second dual-path convolutional neural network model are adjusted to obtain the target periorbital acupoint assisted positioning model.

6. The method according to claim 5, characterized in that, The step of determining the second and third model loss values ​​of the second dual-path convolutional neural network model based on the second periorbital acupoint location prediction image and the sample true-color periorbital labeled image includes: The second periorbital acupoint location prediction image and the sample true-color periorbital labeled image are subtracted to obtain the second difference image; The second difference image is clustered by pixel points to obtain multiple clusters; Using straight line segments, connect the cluster centers of each cluster in a clockwise direction to obtain a closed connecting line; The first length of the closed connection line is calculated, and the first length is determined as the second model loss value of the second dual-path convolutional neural network model. Perform Z-code transformation on the second difference image to obtain the first difference Z-code; The first difference Z-encoded value is input into the sigmoid function for calculation to obtain the third model loss value of the second dual-path convolutional neural network model.

7. The method according to claim 5, characterized in that, The step of adjusting the parameters of the third network module of the second convolutional neural network module and the second network module of the fully connected network module of the second dual-path convolutional neural network model based on the second model loss value and the third model loss value, to obtain the target periorbital acupoint assisted localization model, includes: Based on the second model loss value and the third model loss value, the parameters of the third network module of the second convolutional neural network module and the parameters of the second network module of the fully connected network module of the second dual-path convolutional neural network model are adjusted once to obtain the parameter-adjusted second dual-path convolutional neural network model. The process of inputting the sample true-color periorbital image into the second dual-path convolutional neural network model through the second input terminal, and simultaneously inputting the sample grayscale periorbital image into the second dual-path convolutional neural network model through the first input terminal, to obtain the second periorbital acupoint localization prediction image output by the second dual-path convolutional neural network model, is repeated to begin the next training. The number of consecutive occurrences of the second model's loss value being less than the first threshold and the number of consecutive occurrences of the third model's loss value being less than the second threshold are counted. If the first number of times is greater than or equal to the first preset number of times, and the second number of times is greater than or equal to the first preset number of times, stop adjusting the third network module parameters of the second convolutional neural network module and the second network module parameters of the fully connected network module of the second dual-path convolutional neural network model, and obtain the target periorbital acupoint auxiliary positioning model.

8. A method for assisted positioning of acupoints around the eyes, characterized in that, The method includes: Acquire target periorbital color images of the target patient; Generate a grayscale image of the target periorbital area corresponding to the target periorbital color image; The target periorbital color image and the target periorbital grayscale image are input into the target periorbital acupoint auxiliary localization model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is obtained based on the model training method described in any one of claims 1 to 7.

9. A model training device, characterized in that, The device includes: The first acquisition module is used to acquire a sample true-color periorbital image and a sample true-color periorbital annotated image of the sample patient; wherein, the sample true-color periorbital annotated image is obtained by an expert marking the sample true-color periorbital image; The first generation module is used to generate a sample grayscale periorbital image of the sample true-color periorbital image and a sample grayscale periorbital annotation image of the sample true-color periorbital image. The first training module is used to perform a first-stage model training on the first dual-path convolutional neural network model based on the sample grayscale peri-eye image and the sample grayscale peri-eye labeled image, so as to obtain a second dual-path convolutional neural network model. The second training module is used to perform a second-stage model training on the second dual-path convolutional neural network model based on the sample true-color periorbital image, the sample grayscale periorbital image, and the sample true-color periorbital labeled image, to obtain a target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is used to output a target periorbital acupoint labeled image based on the target patient's target periorbital color image.

10. An auxiliary positioning device for acupoints around the eyes, characterized in that, The device includes: The second acquisition module is used to acquire a color image of the target periocular area of ​​the target patient. The second generation module is used to generate a target peri-eye grayscale image corresponding to the target peri-eye color image; The input / output module is used to input the target periorbital color image and the target periorbital grayscale image into the target periorbital acupoint auxiliary localization model to obtain the target periorbital acupoint marker image output by the target periorbital acupoint auxiliary localization model; wherein, the target periorbital acupoint auxiliary localization model is obtained based on the model training method according to any one of claims 1 to 7.

Images