Acne grading method, system and equipment based on multi-modal fusion and medium

Abstract

The invention discloses an acne grading method, system and equipment based on multi-modal fusion and a medium, belongs to facial acne severity grading in the technical field of artificial intelligence, and aims to solve the technical problem of poor facial acne severity grading effect in the prior art. A constructed facial acne grading model comprises a VGG feature extraction module, a multi-modal feature fusion module, an acne feature sensing module and a comprehensive grading module, the VGG feature extraction module extracts facial features in different modals, and the multi-modal feature fusion module models the relation between the facial features in different modals through cross attention for fusion. The acne feature sensing module enhances the sensing of acne features through a cross attention mechanism, and the comprehensive grading module makes a comprehensive and accurate acne severity grading result. According to the method, comprehensive grading of the severity of the acnes is realized by combining the multi-modal face pictures of the patient, and the grading effect of the severity of the facial acnes is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to the severity grading of facial acne, and in particular to an acne grading method, system, device and medium based on multimodal fusion. Background Art

[0002] Acne is one of the most common skin diseases. According to surveys, the impact of hormone levels during puberty, as well as unhealthy eating habits and work and rest habits, can easily lead to skin acne. In the daily treatment of acne, doctors will grade the severity of acne, which can help doctors develop more targeted treatment plans for patients and also make it easier for patients to understand their condition in a timely manner.

[0003] Acne severity grading usually has different standards in different regions, but most of them are based on the doctor's experience and the number of skin lesions on the patient's face. For example, some grading methods will classify the patient's face into five levels based on the overall condition of the patient: Very Mild (very mild or no need for drug treatment), Mild (mild), Moderate (moderate), Severe (severe) and Very Severe (very severe) indicating urgent need for professional treatment. Of course, there are also 4 levels of facial acne severity.

[0004] When grading the severity of acne, it is mostly done manually by doctors, which greatly increases the workload and work intensity of doctors, and the grading efficiency is low. With the development of artificial intelligence and deep neural network technology, it is very necessary to use computers to automatically grade the severity of acne for patients to understand their own condition and assist doctors in formulating treatment plans.

[0005] The invention patent application with application number 202310795157.7 discloses an acne grading method, device, equipment and medium, which includes: obtaining a facial image of a target user, wherein the facial image includes an image corresponding to acne; detecting the facial image through a pre-trained detection model to obtain at least one acne type and the number of acnes corresponding to the acne type; grading the acne type and the number of acnes corresponding to the acne type through a pre-trained grading model to obtain the acne level of the target user. This solution implements the analysis of the facial image of the target user and obtains the acne level of the target user, so that the acne level does not depend on the doctor's experience, and the classification of acne levels is more objective and accurate.

[0006] The invention patent application with application number 202410122717.7 discloses an acne grading method, apparatus, equipment and storage medium, which inputs an unlabeled pre-trained sample image set into the first feature extraction network and the second feature extraction network of the initial model; determines the target similarity matrix according to the first pre-trained sample feature set output by the first feature extraction network and the second pre-trained sample feature set output by the second feature extraction network, and the target pre-trained sample feature set obtained from the database; adjusts the network parameters in the initial model according to the loss function value calculated according to the target similarity matrix to obtain an optimized initial model; selects one of the feature extraction networks in the optimized initial model to construct a pre-trained model; trains the pre-trained model with a training sample image set with labeled data to obtain a target acne grading model, thereby enhancing the feature learning ability of the model and improving the accuracy of acne grading.

[0007] Like the acne grading method in the above invention patent, the existing technology only focuses on the single-modal picture of the patient's face when using computer-assisted acne grading, which is different from the doctor's focus on the overall condition of the patient's face during the actual diagnosis process; and the existing multimodal method of natural images is not suitable for acne image tasks because it does not consider the impact on the subsequent perception of acne lesion areas when fusing multimodal features. As a result, the existing technology has a low perception ability of the network for facial acne, poor feature extraction effect for facial acne, and poor grading effect of facial acne severity. Summary of the invention

[0008] The purpose of the present invention is to provide an acne grading method, system, device and medium based on multimodal fusion in order to solve the technical problem of poor grading effect of facial acne severity in the prior art.

[0009] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: An acne grading method based on multimodal fusion comprises the following steps: Step S1, obtaining acne image sample data; Obtain facial image samples under different modalities, and annotate the facial image samples to obtain label data; Step S2, constructing a facial acne grading model; Construct a facial acne grading model, which includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module, and a comprehensive grading module. The VGG feature extraction module extracts facial features in different modalities. The multimodal feature fusion module fuses the relationship between facial features in different modalities through cross-attention modeling. The acne feature perception module enhances the perception of acne features through a cross-attention mechanism. The comprehensive grading module makes a comprehensive and accurate acne severity grading result. Step S3, training a facial acne grading model; The facial image samples and label data obtained in step S1 are used to train the facial acne grading model constructed in step S2 to obtain a mature facial acne grading model; Step S4, facial acne grading; Acquire facial acne images to be graded in different modes, and input them into step S3 to obtain a mature facial acne grading model, and obtain acne severity grading results.

[0010] Furthermore, in step S1, the facial image sample is preprocessed, and the specific method is as follows: Step S1.1, a 3D format is uniformly used to represent facial image samples, and a bilinear interpolation method is used to uniformly scale the facial image samples to a size of 3*520*520; Step S1.2, crop the center of the facial image sample to 3*512*512, and perform data enhancement by channel normalization, random flipping, and rotation in sequence; Step S1.3, the severity level of each acne in the facial image sample is digitally numbered and processed into a one-hot label form.

[0011] Furthermore, in step S2, when the multimodal feature fusion module performs feature fusion, specifically: Step S2.2.1, first use 1*1 convolution operation to adjust the multi-view features extracted by the VGG feature extraction module , , To dimension L, and then expand it in the spatial dimension to obtain new features representing the three modes of the left, middle and right sides respectively , , ,

[0012] in, , , represents 1*1 convolution, Indicates expansion in the spatial dimension; In step S2.2.2, cross-attention calculations are performed between the features of the left modality and the features of the middle modality, and between the features of the right modality and the features of the middle modality, respectively, to perform interaction between different modalities.

[0013] Furthermore, in step S2.2.2, when performing cross attention calculation between the features of the left modality and the features of the middle modality, specifically: Step S2.2.2.1, use the first fully connected layer to transform the middle modal features Perform a linear transformation and use it as the query matrix Q in the attention calculation. Use the second fully connected layer to transform the left modality features Perform linear transformation and use it as the key matrix K and value matrix V in attention calculation;

[0014] in, represents a linear transformation, , , All represent the parameters of linear transformation; Step S2.2.2.2, calculate the product between the query matrix Q and the key matrix K transpose through matrix algorithm to obtain the cross attention weight matrix :

[0015] in, represents the number of features, Represents feature dimension; Step S2.2.2.3, cross attention weight matrix Perform normalization:

[0016] in, represents the scaling factor, Represents the cross attention weight matrix Each row in is normalized; Step S2.2.2.4, the attention weight matrix Multiply it with the value matrix V, use the relationship between the features of the left mode and the features of the middle mode to update the features of the left mode, and get the updated features of the left auxiliary mode ;

[0017]

[0018] in, , represents the learnable parameters of the first fully connected layer, , represents the learnable parameters of the second fully connected layer, , Represents the attention weight matrix Multiplication operation with the value matrix V, represents the left modal feature before updating, Represents the updated left modal features.

[0019] Furthermore, in step S2, when the acne feature perception module perceives the acne area, specifically: Step S2.3.1, initialize a learnable perception matrix , ; Perception Matrix Each row vector in It's a sensor. ; Step S2.3.2, the perception matrix As the query matrix in the attention calculation, the fused features are used as the key and value matrix; the perception of acne features is enhanced through the cross attention mechanism, and the multiplication of the query matrix and the key matrix is ​​calculated to obtain the perception weight matrix , the perception weight matrix Multiply with the value matrix to extract features centered on the lesion area ; in, , , N represents the number of perceptron vectors, L represents the dimension of the perceptron vector, and Y represents the perception results of different perceptrons.

[0020] Furthermore, in step S2, the comprehensive classification module includes x fully connected layers, each of which performs feature The corresponding eigenvector in To make a prediction, the output of the nth fully connected layer is expressed as:

[0021] in, Represents the learnable parameters of the nth fully connected layer; Final acne severity prediction results The prediction results of each feature are weighted and summed according to their contribution, specifically:

[0022] in, represents the contribution of the i-th eigenvector extracted by the perceptron, Represents the prediction result of the extracted i-th feature vector.

[0023] Furthermore, in step S3, when training the facial acne grading model, the loss function is:

[0024] Where C represents the total number of levels, G represents the total number of data used for training, and j represents the data of the jth patient. represents the probability that the severity prediction level of the jth patient is z, represents the probability of the jth patient's label at level z.

[0025] An acne grading system based on multimodal fusion, including: A facial image sample acquisition module is used to acquire facial image samples under different modalities and annotate the facial image samples to obtain label data; Facial acne grading model construction module, used to build a facial acne grading model. The facial acne grading model includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module and a comprehensive grading module. The VGG feature extraction module extracts features from images under different modalities. The multimodal feature fusion module fuses the relationship between features in images under different modalities through cross-attention modeling. The acne feature perception module enhances the perception of acne features through a cross-attention mechanism. The comprehensive grading module makes a comprehensive and accurate acne severity grading result. A facial acne grading model training module is used to train the facial acne grading model constructed by the facial acne grading model construction module using the facial image samples and label data obtained by the facial image sample acquisition module to obtain a mature facial acne grading model; The facial acne grading module is used to obtain acne images of the face to be graded in different modes, and input them into the facial acne grading model training module to obtain a mature facial acne grading model to obtain the acne severity grading result.

[0026] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0027] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0028] The beneficial effects of the present invention are as follows: 1. In the present invention, the constructed facial acne grading model includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module and a comprehensive grading module. The VGG feature extraction module extracts facial features under different modalities, the multimodal feature fusion module fuses the relationship between facial features under different modalities through cross-attention modeling, the acne feature perception module enhances the perception of acne features through the cross-attention mechanism, and the comprehensive grading module makes a comprehensive and accurate acne severity grading result; by combining multimodal patient facial images, a comprehensive grading of acne severity is achieved, and the grading effect of facial acne severity is improved.

[0029] 2. In the present invention, the modal fusion method adopted by the multimodal feature fusion module effectively reduces the redundant information between different modalities, enhances the consistency of the modalities, and takes into account the impact of modal fusion on subsequent acne grading; the perceptron mechanism is utilized in the multimodal feature fusion module, which greatly enhances the model's attention to acne features, improves the effect of feature fusion, and ultimately improves the grading effect of facial acne severity.

[0030] 3. In the present invention, multimodal views of the face are taken into consideration. In the acne grading task, doctors usually make a comprehensive judgment on the patient's facial condition. However, previous methods all diagnose based on a single-modal image of the patient's face, which does not meet the actual clinical needs and also affects the accuracy of the diagnosis due to the lack of diagnostic basis. The proposed method integrates images of different modalities of the patient's face to achieve comprehensive diagnostic grading.

[0031] 4. In the present invention, the proposed classification method utilizes a perceptron to extract acne features, and the perceptron's attention can display acne features more finely, which can be visualized as a basis for interpreting the diagnosis results.

[0032] 5. In the present invention, the grading method takes into account the impact of acne characteristics on acne diagnosis and grading. Similar to the diagnosis process of dermatologists in clinical practice, it is necessary to pay attention to the acne characteristics on the patient's face and use this as a basis for judging the severity of acne. Previous methods either lack attention to acne characteristics or require additional labeled data, which seriously affects the effect and versatility of the model. The proposed perceptron mechanism strengthens the focus on acne characteristics without additional labeled information.

[0033] 6. In the present invention, the classification method has strong generalization. First, in the case of modality loss, the original classification diagnosis effect can be enhanced by modality transformation. On the other hand, since no additional annotation information such as acne location and acne number is used, it can be easily migrated to different diagnostic standards. The human and material costs of data collection are reduced.

[0034] 7. In the present invention, the classification method has a certain diagnostic interpretability. The visualization result of the perceptron's perceptual attention matrix of acne features can be regarded as the diagnostic basis of the model. This provides a more reliable explanation for the model's diagnostic results. The previous model is a completely black river model, which is a big problem for auxiliary medical diagnosis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a schematic diagram of the structure of the facial acne grading model of the present invention; Figure 3 Schematic diagram of feature fusion of the multimodal feature fusion module in the present invention; Figure 4 It is a schematic diagram of the structure of residual connection and multi-layer perception in the multimodal feature fusion module of the present invention; Figure 5 It is a structural schematic diagram of the comprehensive grading module in the present invention; Figure 6 is a schematic diagram of acne perception visualization in the present invention; DETAILED DESCRIPTION In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0036] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0037] Example 1 This embodiment provides an acne grading method based on multimodal fusion, such as Figure 1 As shown, the following steps are included: Step S1, obtaining acne image sample data; Acquire facial image samples under different modalities, and annotate the facial image samples to obtain label data.

[0038] The facial image samples include two parts: the first part is from the public facial acne severity grading dataset ACNE04, which contains single-modal images from each patient, with a total of 1,457 cases and 1,457 patient images. Images of other modalities are generated through data enhancement; the second part is the diagnostic dataset HX-MMACNE of patients in different modalities taken by right VISIA, collected in cooperation with dermatologists. This dataset contains 652 cases and 1,956 images in total.

[0039] The facial image samples in the above two parts both contain label data, which are annotated by professional doctors.

[0040] During training, facial images need to be uniformly preprocessed and enhanced so that the image size and format of the facial images are suitable for computer processing, and the generalization of the network is improved through data enhancement.

[0041] Preprocess the facial image samples. The specific method is as follows: Step S1.1, a 3D format is uniformly used to represent facial image samples, and a bilinear interpolation method is used to uniformly scale the facial image samples to a size of 3*520*520; Step S1.2, crop the center of the facial image sample to 3*512*512, and perform data enhancement by channel normalization, random flipping, and rotation in sequence; Step S1.3, the severity level of each acne in the facial image sample is digitally numbered and processed into a one-hot label form.

[0042] The above image samples are divided into training set and test set in a ratio of 4:1, which are used for training and tuning the model and the final model test evaluation. In order to enhance the stability of the model results, both data sets are divided in a five-fold cross-validation manner, and the average of the five results is taken as the final evaluation result.

[0043] Step S2, constructing a facial acne grading model; Construct a facial acne grading model, which includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module, and a comprehensive grading module. The VGG feature extraction module extracts facial features in different modalities. The multimodal feature fusion module fuses the relationship between facial features in different modalities through cross-attention modeling. The acne feature perception module enhances the perception of acne features through the cross-attention mechanism. The comprehensive grading module makes a comprehensive and accurate acne severity grading result, such as Figure 2 shown.

[0044] The VGG feature extraction module uses the existing conventional VGG network without any creative work. The facial image is input into the VGG feature extraction module for feature extraction, and facial features under different modalities are extracted. ,in Represent the height, width, and channel dimensions of the feature respectively.

[0045] Then, these multimodal features are sent to the multimodal feature fusion module, which reduces the redundancy between different modal features through interaction between features, enhances the consistency between features, and performs multimodal feature fusion.

[0046] In the prior art, the severity of acne is usually graded on a single modality image of the patient's face. However, a single modality image usually only includes a portion of the face, ignoring acne lesions in other parts of the face, resulting in inaccurate diagnosis results that cannot fully reflect the patient's condition. Therefore, this embodiment uses images of three different modalities to obtain complete information about the patient's face, and designs a multimodal feature fusion module. By interacting between features of different modalities, redundancy between features is reduced, consistency information is enhanced, and channel dimension splicing and fusion operations are used to avoid interference with subsequent acne grading. When the multimodal feature fusion module performs feature fusion, such as Figure 3 As shown, specifically: Step S2.2.1, first use 1*1 convolution operation to adjust the multi-view features extracted by the VGG feature extraction module , , To dimension L, and then expand it in the spatial dimension to obtain new features representing the three modes of the left, middle and right sides respectively , , ,

[0047] in, , , represents 1*1 convolution, Indicates expansion in the spatial dimension; In step S2.2.2, cross-attention calculations are performed between the features of the left modality and the features of the middle modality, and between the features of the right modality and the features of the middle modality, respectively, to perform interaction between different modalities.

[0048] When performing cross attention calculation, the cross attention calculation between the features of the left modality and the features of the middle modality is taken as an example for explanation. When performing cross attention calculation between the features of the left modality and the features of the middle modality, specifically: Step S2.2.2.1, use the first fully connected layer to transform the middle modal features Perform a linear transformation and use it as the query matrix Q in the attention calculation. Use the second fully connected layer to transform the left modality features Perform linear transformation and use it as the key matrix K and value matrix V in attention calculation;

[0049] in, represents a linear transformation, , , All represent the parameters of linear transformation; Step S2.2.2.2, after obtaining the query matrix Q and the key matrix K, the product of the query matrix Q and the transpose of the key matrix K is calculated by matrix algorithm to obtain the cross attention weight matrix :

[0050] in, represents the number of features, Represents feature dimension; Because matrix multiplication can be regarded as vector multiplication, as shown in the above formula, each row vector in the query matrix Q is multiplied by each column vector in the key matrix K through matrix multiplication, and the result is the similarity relationship between the two vectors. The vectors here are actually the feature vectors in the feature graphs of the two modes, so the result can represent the relationship between the two modes.

[0051] Step S2.2.2.3, in order to eliminate the influence of vector dimension, the cross attention weight matrix Perform normalization:

[0052] in, represents the scaling factor, Represents the cross attention weight matrix Each row in is normalized so that the sum of the probabilities of each row is 1.

[0053] Step S2.2.2.4, with the attention weight matrix Taking the output of the nth element of the i-th row in as an example, the output of the Softmax operation can be expressed as:

[0054] Then the attention weight matrix Multiply it with the value matrix V, and use the relationship between the features of the left mode and the features of the middle mode to update the features of the left mode to obtain the updated features of the left mode At the same time, in order to accelerate the training and convergence process of the model, the residual connection mechanism is used in it, and in order to make the features of the updated left modality richer, a multi-layer perceptron is also used to improve the learning effect of the model, such as Figure 4 The specific principle of the residual connection is to sum the features of the left modal after the update with the features of the left modal before the update in an element-by-element manner to accelerate the convergence of the model, which is specifically expressed as:

[0055] The multi-layer perceptron is implemented through two fully connected layers with ReLU activation function, which is defined as:

[0056] Therefore, the final interactive left auxiliary modal feature The calculation method is:

[0057] in, , represents the learnable parameters of the first fully connected layer, , represents the learnable parameters of the second fully connected layer, , Represents the attention weight matrix Multiplication operation with the value matrix V, represents the left modal feature before updating, Represents the updated left modal features.

[0058] The features of the right modality and the features of the middle modality are calculated in the same way to obtain the updated right feature map by cross attention. For the features of the middle modality that contain more lesion features, the attention is continued to be calculated to obtain the features of the new middle modality , i.e., the query, key and value in the attention are both linear transformations of the same data.

[0059] Through the cross-attention operation, the redundant information between the feature maps of different modalities is reduced, the consistency between different modalities is improved, and the interference of background information is suppressed. Then, the feature connection method is used for fusion to avoid affecting the subsequent perception of the acne lesion area, and finally the fusion feature is obtained. .

[0060] Similar to how doctors judge a patient's condition by observing the acne lesions on the patient's face, the acne lesion area also plays a vital role in the process of automated diagnosis. Therefore, the model needs to be able to focus on these areas and extract the features of these lesion areas. Previous deep learning methods have ignored this point, resulting in poor results. At the same time, in order to improve the versatility of the proposed method, it is necessary to improve the network structure and perform acne perception using only image-level label supervision. To solve this problem, this embodiment adopts an attention-based acne feature perception module, and sets a sensor to interact with the fused features to perceive the acne area and extract features centered on the acne area; at the same time, due to the variable size and type of the acne area, multiple sensors are set to improve the perception ability. When the acne feature perception module perceives the acne area, it specifically: Step S2.3.1, initialize a learnable perception matrix , ; Perception Matrix Each row vector in It's a sensor. ; Step S2.3.2, the perception matrix As the query matrix in the attention calculation, the fused features are used as the key and value matrix; the perception of acne features is enhanced through the cross attention mechanism, and the multiplication of the query matrix and the key matrix is ​​calculated to obtain the perception weight matrix , the perception weight matrix Multiply with the value matrix to extract features centered on the lesion area ; in, , , N represents the number of perceptron vectors, L represents the dimension of the perceptron vector, and Y represents the perception results of different perceptrons.

[0061] After obtaining the feature centered on the acne area Finally, these characteristics are used to predict the severity of the patient , the row vector in the matrix The dimension is C, and Y represents the feature vector extracted by the nth perceptron prediction results.

[0062] The comprehensive classification module includes x fully connected layers, each of which performs feature The corresponding eigenvector in To make a prediction, the output of the nth fully connected layer is expressed as:

[0063] in, Represents the learnable parameters of the nth fully connected layer; At the same time, it should be noted that the extracted feature matrix , where each eigenvector The contribution to the final prediction is different, so each feature vector needs to be evaluated Specifically, the feature vector set Z is input into the fully connected layer with Sigmoid activation function to predict the final contribution , where the Sigmoid activation function is defined as follows, mapping the calculated contribution to the range of 0-1:

[0064] Therefore, the final contribution prediction formula is as follows:

[0065] Final acne severity prediction results The prediction results of each feature are weighted and summed according to their contribution, specifically:

[0066] in, represents the contribution of the i-th eigenvector extracted by the perceptron, Represents the prediction result of the extracted i-th feature vector, such as Figure 5 shown.

[0067] Step S3, training a facial acne grading model; The facial image samples and label data obtained in step S1 are used to train the facial acne grading model constructed in step S2 to obtain a mature facial acne grading model.

[0068] First of all, the input data part, each patient contains three modal (left, middle, right) pictures, and the RGB three-channel encoding method is used to convert the pictures of each modality into corresponding values. At the same time, in order to balance the grading effect, the consumption of computing resources and the grading speed, the size of each picture is processed to 512×512 pixels. For the severity level labels annotated in natural language, digital numbers are used for correspondence. The collected MVACNE dataset is divided into five levels, and the corresponding codes are 0-4. The public dataset ACNE04 is divided into four severity levels, and the corresponding codes are 0-3. In addition, in order to enhance the generalization of the model, each picture is enhanced by random cropping, horizontal flipping, and random rotation to form a form different from the original image. And normalization is performed in the channel dimension to accelerate the convergence process of the model.

[0069] The loss function is the goal of model optimization. Since the acne severity grading task is similar to the image classification task, the multi-classification cross entropy loss function is selected as the optimization goal. According to what was mentioned above, for each patient, the prediction result is: , where C represents the total number of severity levels, so each component Indicates the possibility that the model believes that this image belongs to this category, and then the output of the model is passed through , converted into a probability distribution through the Softmax function ,in The calculation formula is as follows:

[0070] In the previous article, the severity level is mapped to a digital number. Now we need to convert the digital number into a one-hot code. Taking the five types of labels in the HX-MMACNE dataset as an example, if the label is "1", initialize a vector of length 5 and change the position 1 (starting from 0) to 1, that is, the label , which means that the label considers the probability at level 1 to be 1 and the probability at other levels to be 0. After transformation, the label vector is obtained: . Then the final loss function is:

[0071] Where C represents the total number of levels, G represents the total number of data used for training, and j represents the data of the jth patient. represents the probability that the severity prediction level of the jth patient is z, Represents the probability (0 or 1) of the jth patient's label at level z.

[0072] The loss function represents the gap between the model's results and the actual labels. The gradient update parameters are calculated through the back propagation of the loss function. Specifically, the gradient of the loss function with respect to the model parameters is obtained by derivation of the loss function, and then the learning rate is used to update the parameters at a certain ratio in the gradient direction. By continuously adjusting this process, the model reaches an optimal parameter. In this paper, the initial learning rate is set to 0.001, and the stochastic gradient optimizer (SGD) is used to optimize the parameters of the model. The weight decay is set to 0.0001, which is actually a regularization term used to prevent overfitting of the model and improve the generalization of the model. During training, the batch size is set to 16, that is, there are 16 cases in each batch, and the order is shuffled in each round of training. A total of 120 rounds of training are conducted, and the learning rate is decayed by 50% every 20 rounds to prevent the model from reaching a local optimum.

[0073] After the model training is completed, the test set is used to test the effect of the training model. Only when the indicators meet certain requirements can the model be considered to have achieved the effect. When testing the model in this embodiment, the classification accuracy (Accuracy), precision (Precision), sensitivity (Sensitivity), specificity (Specificity) and Youden Index (YI) are mainly used for comprehensive evaluation. The higher these indicators, the better. This test was conducted on two data sets, namely HX-MMACNE collected in cooperation with the hospital, and the public data set ACNE04. The test results on the collected HX-MMACNE data set are shown in the following table:

[0074] The black bold font in the above table indicates the best result on this indicator. It can be seen that the classification method of this application achieves the best effect on the collected test set.

[0075] In addition, in order to verify the performance of the proposed method when modality is missing and its adaptability to the distribution of other datasets, the public dataset ACNE04 was tested. Because each case in the ACNE04 dataset only contains data of a single modality of the patient's face, this embodiment uses image flipping transformation to generate images as other auxiliary modalities for input, which is also a measure taken by the method of this application to deal with modality missing. The test results are as follows:

[0076] It can also be seen that the method of the present application can achieve state-of-the-art performance even in the absence of data in other modalities in public datasets, which demonstrates the superiority of the method of the present application and its robustness in the absence of modalities.

[0077] In addition, in order to demonstrate the degree of attention paid by the method of the present application to acne characteristics, this embodiment also visualizes the acne attention matrix in the model. It can be seen that the proposed method can pay attention to the acne locations in a refined manner. These locations are the evidence for doctors to diagnose patients, which also increases the credibility of the diagnostic grading results and improves the interpretability of the model. The previous prediction model is a complete black box model, and the credibility of its effect in the field of medical auxiliary diagnosis is not high. Figure 6 shown.

[0078] Step S4, facial acne grading; Acquire facial acne images to be graded in different modes, and input them into step S3 to obtain a mature facial acne grading model, and obtain acne severity grading results.

[0079] Example 2 This embodiment provides an acne grading system based on multimodal fusion, including: The facial image sample acquisition module is used to acquire facial image samples under different modalities and annotate the facial image samples to obtain label data.

[0080] The facial image samples include two parts: the first part is from the public facial acne severity grading dataset ACNE04, which contains single-modal images from each patient, with a total of 1,457 cases and 1,457 patient images. Images of other modalities are generated through data enhancement; the second part is the diagnostic dataset HX-MMACNE of patients in different modalities taken by right VISIA, collected in cooperation with dermatologists. This dataset contains 652 cases and 1,956 images in total.

[0081] The facial image samples in the above two parts both contain label data, which are annotated by professional doctors.

[0082] During training, facial images need to be uniformly preprocessed and enhanced so that the image size and format of the facial images are suitable for computer processing, and the generalization of the network is improved through data enhancement.

[0083] Preprocess the facial image samples. The specific method is as follows: Step S1.1, a 3D format is uniformly used to represent facial image samples, and a bilinear interpolation method is used to uniformly scale the facial image samples to a size of 3*520*520; Step S1.2, crop the center of the facial image sample to 3*512*512, and perform data enhancement by channel normalization, random flipping, and rotation in sequence; Step S1.3, the severity level of each acne in the facial image sample is digitally numbered and processed into a one-hot label form.

[0084] The above image samples are divided into training set and test set in a ratio of 4:1, which are used for training and tuning the model and the final model test evaluation. In order to enhance the stability of the model results, both data sets are divided in a five-fold cross-validation manner, and the average of the five results is taken as the final evaluation result.

[0085] The facial image sample acquisition module is used to build a facial acne grading model. The facial acne grading model includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module, and a comprehensive grading module. The VGG feature extraction module extracts facial features under different modalities. The multimodal feature fusion module fuses the relationship between facial features under different modalities through cross-attention modeling. The acne feature perception module enhances the perception of acne features through the cross-attention mechanism. The comprehensive grading module makes a comprehensive and accurate acne severity grading result, such as Figure 2 shown.

[0086] The VGG feature extraction module uses the existing conventional VGG network without any creative work. The facial image is input into the VGG feature extraction module for feature extraction, and facial features under different modalities are extracted. ,in Represent the height, width, and channel dimensions of the feature respectively.

[0087] Then, these multimodal features are sent to the multimodal feature fusion module, which reduces the redundancy between different modal features through interaction between features, enhances the consistency between features, and performs multimodal feature fusion.

[0088] In the prior art, the severity of acne is usually graded on a single modality image of the patient's face. However, a single modality image usually only includes a portion of the face, ignoring acne lesions in other parts of the face, resulting in inaccurate diagnosis results that cannot fully reflect the patient's condition. Therefore, this embodiment uses images of three different modalities to obtain complete information about the patient's face, and designs a multimodal feature fusion module. By interacting between features of different modalities, redundancy between features is reduced, consistency information is enhanced, and channel dimension splicing and fusion operations are used to avoid interference with subsequent acne grading. When the multimodal feature fusion module performs feature fusion, such as Figure 3 As shown, specifically: Step S2.2.1, first use 1*1 convolution operation to adjust the multi-view features extracted by the VGG feature extraction module , , To dimension L, and then expand it in the spatial dimension to obtain new features representing the three modes of the left, middle and right sides respectively , , ,

[0089] in, , , represents 1*1 convolution, Indicates expansion in the spatial dimension; In step S2.2.2, cross-attention calculations are performed between the features of the left modality and the features of the middle modality, and between the features of the right modality and the features of the middle modality, respectively, to perform interaction between different modalities.

[0090] When performing cross attention calculation, the cross attention calculation between the features of the left modality and the features of the middle modality is taken as an example for explanation. When performing cross attention calculation between the features of the left modality and the features of the middle modality, specifically: Step S2.2.2.1, use the first fully connected layer to transform the middle modal features Perform a linear transformation and use it as the query matrix Q in the attention calculation. Use the second fully connected layer to transform the left modality features Perform linear transformation and use it as the key matrix K and value matrix V in attention calculation;

[0091] in, represents a linear transformation, , , All represent the parameters of linear transformation; Step S2.2.2.2, after obtaining the query matrix Q and the key matrix K, the product of the query matrix Q and the transpose of the key matrix K is calculated by matrix algorithm to obtain the cross attention weight matrix :

[0092] in, represents the number of features, Represents feature dimension; Because matrix multiplication can be regarded as vector multiplication, as shown in the above formula, each row vector in the query matrix Q is multiplied by each column vector in the key matrix K through matrix multiplication, and the result is the similarity relationship between the two vectors. The vectors here are actually the feature vectors in the feature graphs of the two modes, so the result can represent the relationship between the two modes.

[0093] Step S2.2.2.3, in order to eliminate the influence of vector dimension, the cross attention weight matrix Perform normalization:

[0094] in, represents the scaling factor, Represents the cross attention weight matrix Each row in is normalized so that the sum of the probabilities of each row is 1.

[0095] Step S2.2.2.4, with the attention weight matrix Taking the output of the nth element of the i-th row in as an example, the output of the Softmax operation can be expressed as:

[0096] Then the attention weight matrix Multiply it with the value matrix V, and use the relationship between the features of the left mode and the features of the middle mode to update the features of the left mode to obtain the updated features of the left mode At the same time, in order to accelerate the training and convergence process of the model, the residual connection mechanism is used in it, and in order to make the features of the updated left modality richer, a multi-layer perceptron is also used to improve the learning effect of the model, such as Figure 4 The specific principle of the residual connection is to sum the features of the left modal after the update with the features of the left modal before the update in an element-by-element manner to accelerate the convergence of the model, which is specifically expressed as:

[0097] The multi-layer perceptron is implemented through two fully connected layers with ReLU activation function, which is defined as:

[0098] Therefore, the final interactive left auxiliary modal feature The calculation method is:

[0099] in, , represents the learnable parameters of the first fully connected layer, , represents the learnable parameters of the second fully connected layer, , Represents the attention weight matrix Multiplication operation with the value matrix V, represents the left modal feature before updating, Represents the updated left modal features.

[0100] The features of the right modality and the features of the middle modality are calculated in the same way to obtain the updated right feature map by cross attention. For the features of the middle modality that contain more lesion features, the attention is continued to be calculated to obtain the features of the new middle modality , i.e., the query, key and value in the attention are both linear transformations of the same data.

[0101] Through the cross-attention operation, the redundant information between the feature maps of different modalities is reduced, the consistency between different modalities is improved, and the interference of background information is suppressed. Then, the feature connection method is used for fusion to avoid affecting the subsequent perception of the acne lesion area, and finally the fusion feature is obtained. .

[0102] Similar to how doctors judge a patient's condition by observing the acne lesions on the patient's face, the acne lesion area also plays a vital role in the process of automated diagnosis. Therefore, the model needs to be able to focus on these areas and extract the features of these lesion areas. Previous deep learning methods have ignored this point, resulting in poor results. At the same time, in order to improve the versatility of the proposed method, it is necessary to improve the network structure and perform acne perception using only image-level label supervision. To solve this problem, this embodiment adopts an attention-based acne feature perception module, and sets a sensor to interact with the fused features to perceive the acne area and extract features centered on the acne area; at the same time, due to the variable size and type of the acne area, multiple sensors are set to improve the perception ability. When the acne feature perception module perceives the acne area, it specifically: Step S2.3.1, initialize a learnable perception matrix , ; Perception Matrix Each row vector in It's a sensor. ; Step S2.3.2, the perception matrix As the query matrix in the attention calculation, the fused features are used as the key and value matrix; the perception of acne features is enhanced through the cross attention mechanism, and the multiplication of the query matrix and the key matrix is ​​calculated to obtain the perception weight matrix , the perception weight matrix Multiply with the value matrix to extract features centered on the lesion area ; in, , , N represents the number of perceptron vectors, L represents the dimension of the perceptron vector, and Y represents the perception results of different perceptrons.

[0103] After obtaining the feature centered on the acne area Finally, these characteristics are used to predict the severity of the patient , the row vector in the matrix The dimension is C, and Y represents the feature vector extracted by the nth perceptron prediction results.

[0104] The comprehensive classification module includes x fully connected layers, each of which performs feature The corresponding eigenvector in To make a prediction, the output of the nth fully connected layer is expressed as:

[0105] in, Represents the learnable parameters of the nth fully connected layer; At the same time, it should be noted that the extracted feature matrix , where each eigenvector The contribution to the final prediction is different, so each feature vector needs to be evaluated Specifically, the feature vector set Z is input into the fully connected layer with Sigmoid activation function to predict the final contribution , where the Sigmoid activation function is defined as follows, mapping the calculated contribution to the range of 0-1:

[0106] Therefore, the final contribution prediction formula is as follows:

[0107] Final acne severity prediction results The prediction results of each feature are weighted and summed according to their contribution, specifically:

[0108] in, represents the contribution of the i-th eigenvector extracted by the perceptron, Represents the prediction result of the extracted i-th feature vector, such as Figure 5 shown.

[0109] The facial acne grading model training module is used to train the facial acne grading model constructed by the facial acne grading model construction module using the facial image samples and label data obtained by the facial image sample acquisition module to obtain a mature facial acne grading model.

[0110] First of all, the input data part, each patient contains three modal (left, middle, right) pictures, and the RGB three-channel encoding method is used to convert the pictures of each modality into corresponding values. At the same time, in order to balance the grading effect, the consumption of computing resources and the grading speed, the size of each picture is processed to 512×512 pixels. For the severity level labels annotated in natural language, digital numbers are used for correspondence. The collected MVACNE dataset is divided into five levels, and the corresponding codes are 0-4. The public dataset ACNE04 is divided into four severity levels, and the corresponding codes are 0-3. In addition, in order to enhance the generalization of the model, each picture is enhanced by random cropping, horizontal flipping, and random rotation to form a form different from the original image. And normalization is performed in the channel dimension to accelerate the convergence process of the model.

[0111] The loss function is the goal of model optimization. Since the acne severity grading task is similar to the image classification task, the multi-classification cross entropy loss function is selected as the optimization goal. According to what was mentioned above, for each patient, the prediction result is: , where C represents the total number of severity levels, so each component Indicates the possibility that the model believes that this image belongs to this category, and then the output of the model is passed through , converted into a probability distribution through the Softmax function ,in The calculation formula is as follows:

[0112] In the previous article, the severity level is mapped to a digital number. Now we need to convert the digital number into a one-hot code. Taking the five types of labels in the HX-MMACNE dataset as an example, if the label is "1", initialize a vector of length 5 and change the position 1 (starting from 0) to 1, that is, the label , which means that the label considers the probability at level 1 to be 1 and the probability at other levels to be 0. After transformation, the label vector is obtained: . Then the final loss function is:

[0113] Where C represents the total number of levels, G represents the total number of data used for training, and j represents the data of the jth patient. represents the probability that the severity prediction level of the jth patient is z, Represents the probability (0 or 1) of the jth patient's label at level z.

[0114] The loss function represents the gap between the model's results and the actual labels. The gradient update parameters are calculated through the back propagation of the loss function. Specifically, the gradient of the loss function with respect to the model parameters is obtained by derivation of the loss function, and then the learning rate is used to update the parameters at a certain ratio in the gradient direction. By continuously adjusting this process, the model reaches an optimal parameter. In this paper, the initial learning rate is set to 0.001, and the stochastic gradient optimizer (SGD) is used to optimize the parameters of the model. The weight decay is set to 0.0001, which is actually a regularization term used to prevent overfitting of the model and improve the generalization of the model. During training, the batch size is set to 16, that is, there are 16 cases in each batch, and the order is shuffled in each round of training. A total of 120 rounds of training are conducted, and the learning rate is decayed by 50% every 20 rounds to prevent the model from reaching a local optimum.

[0115] After the model training is completed, the test set is used to test the effect of the training model. Only when the indicators meet certain requirements can the model be considered to have achieved the effect. When testing the model in this embodiment, the classification accuracy (Accuracy), precision (Precision), sensitivity (Sensitivity), specificity (Specificity) and Youden Index (YI) are mainly used for comprehensive evaluation. The higher these indicators, the better. This test was conducted on two data sets, namely HX-MMACNE collected in cooperation with the hospital, and the public data set ACNE04. The test results on the collected HX-MMACNE data set are shown in the following table:

[0116] The black bold font in the above table indicates the best result on this indicator. It can be seen that the classification method of this application achieves the best effect on the collected test set.

[0117] In addition, in order to verify the performance of the proposed method when modality is missing and its adaptability to the distribution of other datasets, the public dataset ACNE04 was tested. Because each case in the ACNE04 dataset only contains data of a single modality of the patient's face, this embodiment uses image flipping transformation to generate images as other auxiliary modalities for input, which is also a measure taken by the method of this application to deal with modality missing. The test results are as follows:

[0118] It can also be seen that the method of the present application can achieve state-of-the-art performance even in the absence of data in other modalities in public datasets, which demonstrates the superiority of the method of the present application and its robustness in the absence of modalities.

[0119] In addition, in order to demonstrate the degree of attention paid by the method of the present application to acne characteristics, this embodiment also visualizes the acne attention matrix in the model. It can be seen that the proposed method can pay attention to the acne locations in a refined manner. These locations are the evidence for doctors to diagnose patients, which also increases the credibility of the diagnostic grading results and improves the interpretability of the model. The previous prediction model is a complete black box model, and the credibility of its effect in the field of medical auxiliary diagnosis is not high. Figure 6 shown.

[0120] Step S4, facial acne grading; Acquire facial acne images to be graded in different modalities and input them into the facial acne grading model training module to obtain a mature facial acne grading model and obtain the acne severity grading result.

[0121] Example 3 A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of an acne grading method based on multimodal fusion.

[0122] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0123] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the acne grading method based on multimodal fusion, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0124] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the acne grading method based on multimodal fusion.

[0125] Example 4 A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of an acne grading method based on multimodal fusion.

[0126] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned acne grading method based on multimodal fusion.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the acne grading method based on multimodal fusion described in the embodiment of the present application.

Claims

1. An acne grading method based on multimodal fusion, characterized in that: The following steps are involved: Step S1, obtaining acne image sample data; Obtain facial image samples under different modalities, and annotate the facial image samples to obtain label data; Step S2, constructing a facial acne grading model; Construct a facial acne grading model, which includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module, and a comprehensive grading module. The VGG feature extraction module extracts facial features in different modalities. The multimodal feature fusion module fuses the relationship between facial features in different modalities through cross-attention modeling. The acne feature perception module enhances the perception of acne features through a cross-attention mechanism. The comprehensive grading module makes a comprehensive and accurate acne severity grading result. Step S3, training a facial acne grading model; The facial image samples and label data obtained in step S1 are used to train the facial acne grading model constructed in step S2 to obtain a mature facial acne grading model; Step S4, facial acne grading; Acquire facial acne images to be graded in different modes, and input them into step S3 to obtain a mature facial acne grading model to obtain an acne severity grading result.

2. The acne grading method based on multimodal fusion according to claim 1, characterized in that: In step S1, the facial image sample is preprocessed, and the specific method is as follows: Step S1.1, a 3D format is uniformly used to represent facial image samples, and a bilinear interpolation method is used to uniformly scale the facial image samples to a size of 3*520*520; Step S1.2, crop the center of the facial image sample to 3*512*512, and perform data enhancement by channel normalization, random flipping, and rotation in sequence; Step S1.3, the severity level of each acne in the facial image sample is digitally numbered and processed into a one-hot label form.

3. The acne grading method based on multimodal fusion according to claim 1, characterized in that: In step S2, when the multimodal feature fusion module performs feature fusion, specifically: Step S2.2.1, first use 1*1 convolution operation to adjust the multi-view features extracted by the VGG feature extraction module , , To dimension L, and then expand it in the spatial dimension to obtain new features representing the three modes of the left, middle and right sides respectively , , , in, , , represents 1*1 convolution, Indicates expansion in the spatial dimension; In step S2.2.2, cross-attention calculations are performed between the features of the left modality and the features of the middle modality, and between the features of the right modality and the features of the middle modality, respectively, to perform interaction between different modalities.

4. The acne grading method based on multimodal fusion according to claim 1, characterized in that: In step S2.2.2, when performing cross attention calculation between the features of the left modality and the features of the middle modality, specifically: Step S2.2.2.1, use the first fully connected layer to transform the middle modal features Perform a linear transformation and use it as the query matrix Q in the attention calculation. Use the second fully connected layer to transform the left modality features Perform linear transformation and use it as the key matrix K and value matrix V in attention calculation; in, represents a linear transformation, , , All represent the parameters of linear transformation; Step S2.2.2.2, calculate the product between the query matrix Q and the key matrix K transpose through matrix algorithm to obtain the cross attention weight matrix : in, represents the number of features, Represents feature dimension; Step S2.2.2.3, cross attention weight matrix Perform normalization: in, represents the scaling factor, Represents the cross attention weight matrix Each row in is normalized; Step S2.2.2.4, the attention weight matrix Multiply it with the value matrix V, use the relationship between the features of the left mode and the features of the middle mode to update the features of the left mode, and get the updated features of the left auxiliary mode ; in, , represents the learnable parameters of the first fully connected layer, , represents the learnable parameters of the second fully connected layer, , Represent the attention weight matrix Multiplication operation with the value matrix V, represents the left modal feature before updating, Represents the updated left modal features.

5. The acne grading method based on multimodal fusion according to claim 1, characterized in that: In step S2, when the acne feature perception module perceives the acne area, specifically: Step S2.3.1, initialize a learnable perception matrix , ; Perception Matrix Each row vector in It's a sensor. ; Step S2.3.2, the perception matrix As the query matrix in the attention calculation, the fused features are used as the key and value matrix; the perception of acne features is enhanced through the cross attention mechanism, and the multiplication of the query matrix and the key matrix is ​​calculated to obtain the perception weight matrix , the perception weight matrix Multiply with the value matrix to extract features centered on the lesion area ; in, , , N represents the number of perceptron vectors, L represents the dimension of the perceptron vector, and Y represents the perception results of different perceptrons.

6. The acne grading method based on multimodal fusion according to claim 1, characterized in that: In step S2, the comprehensive classification module includes x fully connected layers, each of which performs feature The corresponding eigenvector in To make a prediction, the output of the nth fully connected layer is expressed as: in, Represents the learnable parameters of the nth fully connected layer; Final acne severity prediction results The prediction results of each feature are weighted and summed according to their contribution, specifically: in, represents the contribution of the i-th eigenvector extracted by the perceptron, Represents the prediction result of the extracted i-th feature vector.

7. The acne grading method based on multimodal fusion according to claim 1, characterized in that: In step S3, when training the facial acne grading model, the loss function is: Where C represents the total number of levels, G represents the total number of data used for training, and j represents the data of the jth patient. represents the probability that the severity prediction level of the jth patient is z, represents the probability of the jth patient's label at level z.

8. An acne grading system based on multimodal fusion, characterized in that: include: A facial image sample acquisition module is used to acquire facial image samples under different modalities and annotate the facial image samples to obtain label data; Facial acne grading model construction module, used to build a facial acne grading model. The facial acne grading model includes a VGG feature extraction module, a multimodal feature fusion module, an acne feature perception module and a comprehensive grading module. The VGG feature extraction module extracts features from images under different modalities. The multimodal feature fusion module fuses the relationship between features in images under different modalities through cross-attention modeling. The acne feature perception module enhances the perception of acne features through a cross-attention mechanism. The comprehensive grading module makes a comprehensive and accurate acne severity grading result. A facial acne grading model training module is used to train the facial acne grading model constructed by the facial acne grading model construction module using the facial image samples and label data obtained by the facial image sample acquisition module to obtain a mature facial acne grading model; The facial acne grading module is used to obtain acne images of the face to be graded in different modes, and input them into the facial acne grading model training module to obtain a mature facial acne grading model to obtain the acne severity grading result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images