Image processing model training method, image processing method, device and storage medium

Abstract

The present application provides an image processing model training method, an image processing method, a device and a storage medium, wherein the image processing model training method comprises: obtaining multiple tear trough image samples, each tear trough image sample is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, the first annotation score represents the severity of the tear trough in the tear trough image sample, and a model is trained according to the multiple tear trough image samples to obtain an image processing model. In the present application, the image processing model is obtained by annotating the tear trough supervision data and the feature map supervision data of the multiple tear trough image samples to improve the accuracy of the tear trough evaluation.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image processing model training method, an image processing method, a device and a storage medium. Background Art

[0002] With the improvement of consumption level, more and more people begin to pay attention to skin care. The current skin detection methods include manual detection and instrument detection, which can be used for pore size detection, tear trough severity detection, wrinkle severity detection, etc.

[0003] For tear trough severity detection, manual detection mainly involves subjectively assigning tear troughs to levels of none, mild, moderate, or severe. Instrument detection mainly involves extracting image features using feature extraction methods, and then grading tear troughs into none, mild, moderate, or severe based on the image features.

[0004] However, due to the great ambiguity of the degree of tear troughs, manual detection can only give an approximate grade, which lacks scientificity and objectivity; the features extracted by instrumental detection are largely unrelated to the actual severity, and the accuracy of tear trough evaluation is low. Summary of the invention

[0005] The purpose of the present application is to provide an image processing model training method, an image processing method, a device and a storage medium to improve the accuracy of tear trough evaluation in view of the deficiencies in the above-mentioned prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides an image processing model training method, comprising:

[0008] Acquire multiple tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample;

[0009] Model training is performed according to the multiple tear trough image samples to obtain an image processing model.

[0010] Optionally, the feature map supervision data is obtained by the following steps:

[0011] Normalizing the first annotation score to obtain a tear groove mapping score;

[0012] Using a preset activation function to process the tear groove mapping score to obtain the feature map supervision data;

[0013] The tear groove supervision data is obtained by the following steps:

[0014] The feature map supervision data is subjected to global average pooling processing to obtain the tear groove supervision data.

[0015] Optionally, before normalizing the first annotation score to obtain a tear groove mapping score, the method further includes:

[0016] For each tear trough image sample, randomly determining whether to perform a grid stitching process on the tear trough image sample;

[0017] If yes, randomly obtaining a preset number of other tear trough image samples from the plurality of tear trough image samples, and splicing the other tear trough image samples and the tear trough image samples into a grid;

[0018] The step of normalizing the first labeled score to obtain a tear groove mapping score includes:

[0019] Normalizing the first annotation score to obtain a first tear groove mapping score;

[0020] Obtaining a second annotation score of the other tear trough image samples in the grid, where the second annotation score represents the severity of the tear trough in the other tear trough image samples;

[0021] The second labeling score is normalized to obtain a second tear trough mapping score, wherein the tear trough mapping score includes the first tear trough mapping score and the second tear trough mapping score.

[0022] Optionally, performing model training according to the plurality of tear trough image samples to obtain an image processing model includes:

[0023] For each tear trough image sample, the tear trough image sample is processed according to a preset neural network model to obtain feature map output data;

[0024] Performing global average pooling processing on the feature map output data to obtain tear groove output data corresponding to the feature map output data;

[0025] Obtaining a first loss according to the tear trough output data and the tear trough supervision data;

[0026] Obtaining a second loss according to the feature map output data and the feature map supervision data;

[0027] Model training is performed according to the first loss and the second loss to obtain the image processing model.

[0028] Optionally, performing model training according to the first loss and the second loss to obtain the image processing model includes:

[0029] According to a preset sorting method, sorting the plurality of tear trough supervision data annotated by the plurality of tear trough image samples, and combining them in pairs to obtain a plurality of supervision data combinations;

[0030] For each of the supervision data combinations, obtaining a combination sub-loss corresponding to the supervision data combination according to the first tear trough supervision data and the second tear trough supervision data in the supervision data combination;

[0031] Obtaining a sum of a plurality of the combined sub-losses corresponding to a plurality of the supervised data combinations, and obtaining a combined sorting loss according to the sum;

[0032] Model training is performed according to the first loss, the second loss and the combined ranking loss to obtain the image processing model.

[0033] Optionally, for each of the supervised data combinations, obtaining a combined ranking sub-loss corresponding to the supervised data combination according to the first tear trough supervised data and the second tear trough supervised data in the supervised data combination includes:

[0034] Calculating a first difference between the first tear trough supervision data and the second tear trough supervision data;

[0035] Calculating a second difference between first tear trough output data corresponding to the first tear trough supervision data and second tear trough output data corresponding to the second tear trough supervision data;

[0036] If the first difference is 0, then according to loss = -0.5*log e (1-D)-0.5*log e (D), obtaining the combined ranking sub-loss;

[0037] Otherwise, according to loss = -log e (D), obtaining the combined ranking sub-loss;

[0038] Wherein, D is the second difference, and loss is the combined sorting sub-loss.

[0039] In a second aspect, another embodiment of the present application provides an image processing method, including:

[0040] Acquire the tear trough image to be processed;

[0041] Processing the tear trough image to be processed according to a pre-trained image processing model to obtain tear trough output data corresponding to the tear trough image to be processed, wherein the image processing model is trained based on a plurality of tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample;

[0042] According to the tear trough output data corresponding to the tear trough image to be processed, a prediction score of the tear trough in the tear trough image to be processed is obtained.

[0043] In a third aspect, another embodiment of the present application provides an image processing model training device, comprising:

[0044] an acquisition module, configured to acquire a plurality of tear trough image samples, each of the tear trough image samples being annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, wherein the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, wherein the first annotation score represents a severity of the tear trough in the tear trough image sample;

[0045] The training module is used to perform model training according to the multiple tear trough image samples to obtain an image processing model.

[0046] Optionally, it also includes:

[0047] A processing module is used to normalize the first annotation score to obtain a tear trough mapping score, process the tear trough mapping score using a preset activation function to obtain the feature map supervision data; and perform global average pooling on the feature map supervision data to obtain the tear trough supervision data.

[0048] Optionally, it also includes:

[0049] A determination module, used for randomly determining, for each tear trough image sample, whether to perform a grid stitching process on the tear trough image sample;

[0050] If yes, the acquisition module is further used to randomly acquire a preset number of other tear trough image samples from the plurality of tear trough image samples, and splice the other tear trough image samples and the tear trough image samples into a grid;

[0051] The acquisition module is specifically used for:

[0052] Normalizing the first annotation score to obtain a first tear groove mapping score;

[0053] Obtaining a second annotation score of the other tear trough image samples in the grid, where the second annotation score represents the severity of the tear trough in the other tear trough image samples;

[0054] The second labeling score is normalized to obtain a second tear trough mapping score, wherein the tear trough mapping score includes the first tear trough mapping score and the second tear trough mapping score.

[0055] Optionally, the training module is specifically used to:

[0056] For each tear trough image sample, the tear trough image sample is processed according to a preset neural network model to obtain feature map output data;

[0057] Performing global average pooling processing on the feature map output data to obtain tear groove output data corresponding to the feature map output data;

[0058] Obtaining a first loss according to the tear trough output data and the tear trough supervision data;

[0059] Obtaining a second loss according to the feature map output data and the feature map supervision data;

[0060] Model training is performed according to the first loss and the second loss to obtain the image processing model.

[0061] Optionally, the training module is specifically used to:

[0062] According to a preset sorting method, sorting the plurality of tear trough supervision data annotated by the plurality of tear trough image samples, and combining them in pairs to obtain a plurality of supervision data combinations;

[0063] For each of the supervision data combinations, obtaining a combination sub-loss corresponding to the supervision data combination according to the first tear trough supervision data and the second tear trough supervision data in the supervision data combination;

[0064] Obtaining a sum of a plurality of the combined sub-losses corresponding to a plurality of the supervised data combinations, and obtaining a combined sorting loss according to the sum;

[0065] Model training is performed according to the first loss, the second loss and the combined ranking loss to obtain the image processing model.

[0066] Optionally, the training module is specifically used to:

[0067] Calculating a first difference between the first tear trough supervision data and the second tear trough supervision data;

[0068] Calculating a second difference between first tear trough output data corresponding to the first tear trough supervision data and second tear trough output data corresponding to the second tear trough supervision data;

[0069] If the first difference is 0, then according to loss = -0.5*log e (1-D)-0.5*log e (D), obtaining the combined ranking sub-loss;

[0070] Otherwise, according to loss = -log e (D), obtaining the combined ranking sub-loss;

[0071] Wherein, D is the second difference, and loss is the combined sorting sub-loss.

[0072] In a fourth aspect, another embodiment of the present application provides an image processing device, including:

[0073] An acquisition module, used for acquiring the tear trough image to be processed;

[0074] a processing module, configured to process the tear trough image to be processed according to a pre-trained image processing model, and obtain tear trough output data corresponding to the tear trough image to be processed, wherein the image processing model is trained based on a plurality of tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample;

[0075] The acquisition module is further used to acquire the prediction score of the tear trough in the tear trough image to be processed according to the tear trough output data corresponding to the tear trough image to be processed.

[0076] In the fifth aspect, another embodiment of the present application provides an image processing model training device, comprising: a processor, a memory and a bus, the memory storing a computer program executable by the processor, when the image processing model training device is running, the processor and the memory communicate through the bus, and the processor executes the computer program to execute any one of the methods described in the first aspect.

[0077] In the sixth aspect, another embodiment of the present application provides an image processing device, comprising: a processor, a memory and a bus, the memory storing a computer program executable by the processor, when the image processing device is running, the processor and the memory communicate through the bus, and the processor executes the computer program to execute the method described in the second aspect.

[0078] In the seventh aspect, another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the first aspect and the second aspect is executed.

[0079] The beneficial effects of this application are:

[0080] The present application provides an image processing model training method, an image processing method, a device and a storage medium, wherein the image processing model training method comprises: obtaining multiple tear trough image samples, each tear trough image sample is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, the first annotation score represents the severity of the tear trough in the tear trough image sample, and a model is trained according to the multiple tear trough image samples to obtain an image processing model. In the present application, the image processing model is obtained by annotating the tear trough supervision data and the feature map supervision data of the multiple tear trough image samples to improve the accuracy of the tear trough evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 1 ;

[0083] Figure 2 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 2 ;

[0084] Figure 3 A schematic diagram of a grid stitching process provided by an embodiment of the present application is shown;

[0085] Figure 4 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 3 ;

[0086] Figure 5 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 4 ;

[0087] Figure 6 The figure shows a schematic diagram of the process of the image processing method provided by the embodiment of the present application;

[0088] Figure 7 A schematic diagram of the structure of an image processing model training device provided in an embodiment of the present application is shown;

[0089] Figure 8 A schematic diagram showing the structure of an image processing device provided in an embodiment of the present application is shown;

[0090] Fig. 9 A schematic diagram of the structure of an image processing model training device provided in an embodiment of the present application is shown;

[0091] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0092] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0093] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0094] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0095] For tear trough severity detection, the existing technology provides two detection methods. One is manual detection, which mainly manually and subjectively gives the tear trough a grade of none, mild, moderate, or severe. The other is instrument detection, which mainly uses traditional methods to extract features, such as: scale-invariant feature transform (SIFT), histogram of oriented gradient (HOG), color features, texture features, shape features, etc., or uses convolutional neural networks (CNN) to extract features to classify the tear trough degree as none, mild, moderate, or severe.

[0096] However, due to the great ambiguity of the degree of tear trough, manual work can only give a rough grade, which lacks scientificity and objectivity. The severity of the tear trough extracted by features is mostly unsatisfactory. The actual severity is not related to the extracted features, and the grading of tear troughs can mostly only be given to 4 levels, and the actual score of the tear trough cannot be quantified. In summary, the existing tear trough evaluation methods have low accuracy.

[0097] Based on this, the present application provides an image processing model training method and an image processing method, by annotating tear trough supervision data and feature map supervision data of multiple tear trough image samples, performing high-quality modeling, and obtaining an image processing model. When evaluating the tear trough, the image processing model is used to process the tear trough image to be processed to obtain the predicted score of the tear trough in the tear trough image to be processed, thereby measuring the tear trough by the score, thereby improving the accuracy of the tear trough evaluation.

[0098] Combine the following Figures 1 to 4 The embodiment illustrates the image processing model training method provided in this application.

[0099] Figure 1 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 1 The execution subject of this embodiment may be an image processing model training device, for example, a terminal device, a server, or other device with data processing capabilities.

[0100] like Figure 1 As shown, the method may include:

[0101] S101, obtaining a plurality of tear trough image samples.

[0102] In practical applications, in order to evaluate the tear troughs on the face, multiple tear trough image samples can be obtained by shooting. To ensure the richness of the sample data, the multiple tear trough image samples can include image samples of different age groups and genders, and contain different types of tear troughs, for example, atrophic tear troughs, convex tear troughs, and relaxed tear troughs.

[0103] Each tear trough image sample is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample.

[0104] It should be noted that the tear trough supervision data and the feature map supervision data are representations of mapping the first annotation score to the latent space. The tear trough supervision data can be a one-dimensional representation, and the feature map supervision data corresponding to the tear trough supervision data can be a two-dimensional representation. For example, 32 images have a total of 32 tear trough supervision data (latent vector supervision data), recorded as: [32, 1], and 32 images have a total of 32 feature map supervision data (latent vector feature map supervision data), recorded as: [32, H, W], where H and W represent the horizontal and vertical coordinates of the pixels, respectively.

[0105] Among them, the first annotation score can be jointly determined by professional doctors and ordinary people. In a possible implementation, professional doctors can select typical data that can reflect the degree of tear troughs from multiple tear trough image samples obtained to determine a grading table. Taking 7 levels of grading (level 1 to level 7) as an example, the score is calculated in percentage, and the score and the grading can have a preset mapping relationship, such as level 0: 100 points, level 1: 85-99, level 2: 70-84, level 3: 55-69, level 4: 40-54, level 5: 25-39, level 6: 2-24. Ordinary people are used to annotate the scores, and the annotation is based on the grading table to estimate the scores. Usually, 3 ordinary people can annotate 1 tear trough image sample, and the average of the 3 annotation scores is taken as the first annotation score of the tear trough in the tear trough image sample, so as to obtain a relatively discrete tear trough score.

[0106] In this example, in order to measure the severity of tear troughs with high quality, data from different age groups and genders were collected, and the data were summarized, classified, graded and scored (using a percentage system). This algorithm was used to resolve the ambiguity in human labeling of tear trough severity scores, making the results more objective and ensuring the accuracy of tear trough differentiation.

[0107] For example, the first annotation score may be inversely proportional to the severity of the tear trough, the higher the first annotation score, the less severe the tear trough, and the lower the first annotation score, the more severe the tear trough. Of course, the above is an exemplary description of the first annotation score, and this embodiment is not limited thereto.

[0108] Optionally, feature map supervision data is obtained by the following steps:

[0109] The first annotation score is normalized to obtain the tear groove mapping score.

[0110] The tear trough mapping score is processed using a preset activation function to obtain feature map supervision data.

[0111] The tear groove supervision data is obtained through the following steps:

[0112] The feature map supervision data is globally averaged pooled to obtain the tear groove supervision data.

[0113] The normalization process in this step may refer to mapping the first annotation score to between [0.01, 0.99], and the mapping formula may be:

[0114]

[0115] ScoreSrc is the first annotation score, and ScoreOut is the tear groove mapping score.

[0116] Of course, the mapping formula provided above is only an example, and various mapping formulas can be expanded according to actual conditions. This embodiment does not specifically limit the mapping formula.

[0117] The activation function may be a Sigmoid function, and using a preset activation function to process the tear trough mapping score may be using an inverse process of the Sigmoid function to process the tear trough mapping score.

[0118] Among them, the Sigmoid function is defined as:

[0119]

[0120] The reverse process is:

[0121]

[0122] Among them, x is the feature map supervision data, S(x) is the tear groove mapping score, and its value range is (0, 1). In order to avoid training divergence, this scheme can limit it to [0.01, 0.99].

[0123] In this step, the tear groove mapping score can be processed by the inverse sigmoid module to obtain feature map supervision data, and then the feature map supervision data is processed by global average pooling module (GlobalAveragePool) to obtain tear groove supervision data.

[0124] S102: Perform model training according to multiple tear trough image samples to obtain an image processing model.

[0125] A plurality of tear trough image samples are input into a preset convolutional neural network, and an image processing model can be obtained by iteratively optimizing network weights, wherein the preset convolutional neural network can be a U-type network, such as a Unet network.

[0126] In summary, the image processing model training method of this embodiment includes: obtaining multiple tear trough image samples, each tear trough image sample is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, the first annotation score represents the severity of the tear trough in the tear trough image sample, and model training is performed according to the multiple tear trough image samples to obtain an image processing model. By annotating the tear trough supervision data and feature map supervision data of the multiple tear trough image samples to obtain an image processing model, the accuracy of tear trough evaluation is improved.

[0127] Exemplarily, before normalizing the first annotation score, the tear groove image sample may be preprocessed. Figure 2 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 2 ,like Figure 2 As shown, before normalizing the first annotation score to obtain the tear groove mapping score, the method may further include:

[0128] S201 . For each tear trough image sample, randomly determine whether to perform a grid stitching process on the tear trough image sample.

[0129] S202: randomly obtain a preset number of other tear trough image samples from the plurality of tear trough image samples, and splice the other tear trough image samples and the tear trough image samples into squares.

[0130] Steps S201-S202 belong to the process of image preprocessing, and whether to perform grid stitching processing on each tear trough image sample can be determined according to a certain random probability. For example, whether to perform grid stitching processing is determined with a random probability of 0.5. Grid stitching processing refers to stitching a preset number of tear trough image samples into squares. If so, execute step S201, obtain a preset number of other tear trough image samples from multiple tear trough image samples, and stitch the other tear trough image samples and the tear trough image sample into squares.

[0131] Accordingly, normalizing the first annotation score to obtain the tear groove mapping score may include:

[0132] The first annotation score is normalized to obtain a first tear groove mapping score.

[0133] The second annotation scores of other tear trough image samples in the grid are obtained, and the second annotation scores represent the severity of the tear troughs in the other tear trough image samples.

[0134] The second annotation score is normalized to obtain a second tear groove mapping score.

[0135] The tear trough mapping score includes a first tear trough mapping score and a second tear trough mapping score. The first annotation score and the second annotation score are normalized to obtain the first tear trough mapping score and the second tear trough mapping score.

[0136] In this embodiment, by mapping the real score data into the latent space and comparing the tear trough degrees in the latent space, the ambiguity and discreteness of the manually labeled tear trough degrees are solved, and the accuracy has a greater correlation and distinguishability with the actual tear trough degrees.

[0137] by Figure 3 For example, Figure 3 FIG. 4 shows a schematic diagram of a grid stitching process provided by an embodiment of the present application. Figure 3 As shown, the preset number can be 3, the first annotation score of the tear trough image sample is 11, and the second annotation scores of the other tear trough image samples are 21, 31, and 41 respectively. The four tear trough image samples are spliced ​​into a grid, and then the mapped tear trough scores are obtained according to the above formula (1), which are 0.10, 0.20, 0.30, and 0.40 respectively.

[0138] Then, the feature map supervision data corresponding to the tear groove image sample is calculated according to formula (3), which is approximately -2.20, -1.39, -0.85, and -0.41. Then, the feature map supervision data is processed by global average pooling module to obtain the tear groove supervision data. Figure 3 The approximate value in is -1.21.

[0139] Exemplarily, if it is determined not to perform grid stitching processing, the first annotation score is normalized to obtain the first tear trough mapping score. For example, the first annotation score is 26, which is normalized to 0.25. Then, the inverse Sigmoid module is used to calculate the feature map supervision data corresponding to the tear trough image sample according to formula (3), such as -1.10. Then, the global average pooling module is used to obtain the tear trough supervision data, i.e., -1.10.

[0140] Optionally, the image preprocessing may further include image enhancement processing and / or image scaling processing, and step S201 may be performed after the preprocessing.

[0141] Before step S201, the method may further include:

[0142] Perform image enhancement processing and / or image scaling processing on the tear trough image sample to obtain a processed tear trough image sample.

[0143] That is, after performing the image enhancement process and / or the image scaling process, step S201 may be performed again.

[0144] Among them, image enhancement processing may include: at least one of affine transformation, mirror mapping, compression, smoothing, noise addition and color adjustment. Image enhancement processing can improve the accuracy of image processing to reduce the problem of poor processing accuracy caused by image quality problems.

[0145] Image scaling processing refers to scaling the tear trough image samples to a fixed size, such as 256*256, so that the accuracy of image processing can be improved under the premise of unified pixels.

[0146] In summary, the image processing model training method of this embodiment also includes: randomly determining whether to perform grid splicing processing on the tear trough image sample, and if so, randomly obtaining a preset number of other tear trough image samples from multiple tear trough image samples, and splicing the other tear trough image samples and the tear trough image samples into a grid, normalizing the first annotation score to obtain the first tear trough mapping score, obtaining the second annotation score of the other tear trough image samples in the grid, the second annotation score characterizing the severity of the tear trough in the other tear trough image samples, normalizing the second annotation score to obtain the second tear trough mapping score. In this embodiment, by preprocessing the tear trough image samples to obtain high-precision data, the stability and continuity of image processing are improved.

[0147] For example, Figure 4 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 3 ,like Figure 4 As shown, step S102, performing model training according to a plurality of tear trough image samples to obtain an image processing model, may include:

[0148] S301. For each tear trough image sample, the tear trough image sample is processed according to a preset neural network model to obtain feature map output data.

[0149] S302 , performing global average pooling processing on the feature map output data to obtain tear groove output data corresponding to the feature map output data.

[0150] Among them, the preset neural network model can be a U-type network, such as a Unet network. For each tear trough image sample, the tear trough image sample is input into the preset neural network model, and the tear trough image sample is processed to obtain the feature map output data corresponding to the tear trough image sample.

[0151] Then, the feature map output data is globally average pooled through the global average pooling module to obtain the tear trough output data corresponding to the feature map output data, wherein the tear trough output data and the feature map output data are representations of the latent space, the tear trough output data is also the output latent vector, and the feature map output data is also the output latent vector feature map.

[0152] S303. Obtain a first loss according to the tear trough output data and the tear trough supervision data.

[0153] S304. Obtain a second loss according to the feature map output data and the feature map supervision data.

[0154] S305. Perform model training according to the first loss and the second loss to obtain an image processing model.

[0155] According to the tear trough output data and the tear trough supervision data, the first loss can be obtained, and according to the feature map output data and the feature map supervision data, the second loss can be obtained. Then, the model can be trained according to the first loss and the second loss, and the obtained first loss and second loss are gradient updated through the back propagation algorithm, and iterative optimization is performed until convergence to obtain an image processing model.

[0156] Optionally, the first loss and the second loss may be calculated by Smooth L1, and the formula is as follows:

[0157]

[0158] For the first loss, y is the difference between the tear trough output data and the tear trough supervision data. For the second loss, y is the difference between the feature map output data and the feature map supervision data.

[0159] The image processing model training method of this embodiment includes: for each tear trough image sample, processing the tear trough image sample according to a preset neural network model, obtaining feature map output data, performing global average pooling processing on the feature map output data, obtaining tear trough output data corresponding to the feature map output data, obtaining a first loss according to the tear trough output data and the tear trough supervision data, obtaining a second loss according to the feature map output data and the feature map supervision data, performing model training according to the first loss and the second loss, and obtaining an image processing model. In this embodiment, through the acquisition of high-precision data, high-standard classification formulation and annotation, and high-quality algorithm modeling, not only the stability and continuity of the output score are guaranteed, but also the discrimination of the tear trough accuracy is ensured.

[0160] For example, Figure 5 The following is a schematic diagram showing the process of the image processing model training method provided in the embodiment of the present application. Figure 4 ,like Figure 5As shown, step S305, performing model training according to the first loss and the second loss to obtain an image processing model may include:

[0161] S401. Sort a plurality of tear trough supervision data annotated by a plurality of tear trough image samples according to a preset sorting method, and combine them in pairs to obtain a plurality of supervision data combinations.

[0162] Among them, the preset sorting method can be a sorting method from large to small. According to the preset sorting method, multiple tear trough supervision data (latent vector supervision data) annotated by multiple tear trough image samples are sorted from large to small, and multiple supervision data combinations are obtained based on pairwise combinations.

[0163] For example, the number of tear trough image samples is N. The N tear trough supervision data are sorted from large to small and combined in pairs, and there are a total of There are supervised data combinations. If N is 16, there are more than 4,000 supervised data combinations.

[0164] S402 . For each supervision data combination, obtain a combination sub-loss corresponding to the supervision data combination according to the first tear trough supervision data and the second tear trough supervision data in the supervision data combination.

[0165] For each supervision data combination, the first tear trough supervision data and the second tear trough supervision data can be compared, and then the combination sub-loss function corresponding to the supervision data combination can be determined according to the relative size of the first tear trough supervision data and the second tear trough supervision data, and then the combination sub-loss corresponding to the supervision data combination can be calculated.

[0166] Optionally, step S402 may include:

[0167] A first difference between the first tear trough supervision data and the second tear trough supervision data is calculated.

[0168] A second difference between first tear trough output data corresponding to the first tear trough supervision data and second tear trough output data corresponding to the second tear trough supervision data is calculated.

[0169] If the first difference is 0, then according to loss = -0.5*log e (1-D)-0.5*log e (D), obtain the combined ranking sub-loss.

[0170] Otherwise, according to loss = -log e (D), obtain the combined ranking sub-loss.

[0171] Among them, D is the second difference, and loss is the combined sorting sub-loss.

[0172] For example, the currently selected first tear trough supervision data and second tear trough supervision data are label_i and label_j respectively, and label_j is greater than or equal to label_i. The difference between the first tear trough output data and the second tear trough output data is calculated according to the formula D=S(output_j-output_i), where output_j is the first tear trough output data, output_i is the second tear trough output data, and i and j represent the serial numbers of the first tear trough supervision data and the second tear trough supervision data sorted according to a preset sorting method, such as 1 and 4.

[0173] If label_j is equal to label_i, then according to loss = -0.5*log e (1-D)-0.5*log e (D) Get the combined ranking sub-loss; if label_j is not equal to label_i, according to loss = -log e (D), obtain the combined ranking sub-loss.

[0174] Calculate the first difference between the first tear trough supervision data and the second tear trough supervision data, and the second difference between the first tear trough output data corresponding to the first tear trough supervision data and the second tear trough output data corresponding to the second tear trough supervision data. If the first difference is 0, then according to loss = -0.5*log e (1-D)-0.5*log e (D) Calculate the combined sorting sub-loss; if the first difference is not equal, then according to loss_diff = -log e (D), obtain the combined ranking sub-loss.

[0175] S403: Obtain the sum of multiple combined sub-losses corresponding to multiple supervised data combinations, and obtain the combined sorting loss based on the sum.

[0176] Each supervised data combination corresponds to a combined sub-loss, so the sum of multiple combined sub-losses corresponding to multiple supervised data combinations can also be calculated, and then the combined ranking loss can be obtained based on the sum. Optionally, the combined ranking loss can be calculated according to the following formula:

[0177] loss_rank=(∑loss) / (N*(N-1) / 2)

[0178] Among them, loss_rank is the combined ranking loss, ∑loss is the sum of multiple combined sub-losses, and N is the number of multiple tear trough image samples.

[0179] S404: Perform model training according to the first loss, the second loss and the combined ranking loss to obtain an image processing model.

[0180] The first loss, the second loss and the combined ranking loss are updated with gradients through back propagation, and the optimization is iteratively optimized until convergence. The iteratively optimized weights are output and saved to obtain the image processing model.

[0181] In this embodiment, based on the combined sorting method, high-precision data classification and grading, data of the same individual at different time periods, and high-quality image data, the algorithm has a high stability for the tear trough scores of consecutive photos of the same individual, with a coefficient of variation of up to 0.03.

[0182] In summary, the image processing model training method of this embodiment sorts multiple tear trough supervision data annotated by multiple tear trough image samples according to a preset sorting method, and combines them in pairs to obtain multiple supervision data combinations. For each supervision data combination, according to the first tear trough supervision data and the second tear trough supervision data in the supervision data combination, obtain the combined sub-loss corresponding to the supervision data combination, obtain the sum of multiple combined sub-losses corresponding to the multiple supervision data combinations, and obtain the combined sorting loss according to the sum, and perform model training according to the first loss, the second loss and the combined sorting loss to obtain the image processing model. In this embodiment, by fully sorting the tear trough supervision data, mapping the real score data into the latent space, and comparing the tear trough degree in the latent space, the ambiguity and discreteness of the manually labeled tear trough degree are solved, and the accuracy has a greater correlation and distinguishability with the actual tear trough degree.

[0183] Combine the following Figure 6 The embodiment illustrates the image processing method provided by the present application.

[0184] Figure 6 The flowchart of the image processing method provided in the embodiment of the present application is shown. The execution subject of this embodiment may be an image processing device, for example, a terminal device, a server, or other device with data processing capabilities.

[0185] like Figure 6 As shown, the method may include:

[0186] S501, obtaining a tear trough image to be processed.

[0187] S502: Process the tear trough image to be processed according to the pre-trained image processing model to obtain tear trough output data corresponding to the tear trough image to be processed.

[0188] Among them, the image processing model is trained based on multiple tear trough image samples, each tear trough image sample is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate the first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample.

[0189] The tear trough image to be processed is processed according to the pre-trained image processing model to obtain tear trough output data corresponding to the tear trough image to be processed. Optionally, the tear trough image to be processed may be pre-processed and normalized before image processing, and the pre-processing includes: image enhancement processing and / or image scaling processing to improve the accuracy of image processing.

[0190] Optionally, step S502 specifically includes:

[0191] The tear trough image to be processed is input into the image processing model to obtain feature map output data corresponding to the tear trough image to be processed.

[0192] The preset activation function is used to process the feature map output data to obtain the activation output.

[0193] Perform global average pooling on the activation output to obtain the tear groove output data.

[0194] The tear trough image to be processed is input into a pre-trained image processing model, and the network inference output is executed to obtain the feature map output data (output latent vector feature map) corresponding to the tear trough image to be processed. Then, the feature map output data is processed by a preset activation function to obtain the activation output, wherein the preset activation function can be Sigmoid, see formula (2), the feature map output data is taken as x, and S(x) is output, i.e., the activation output. Then, the activation output is globally averaged through a global average pooling processing module to obtain the tear trough output data (output latent vector).

[0195] It should be noted that for the specific description of the image processing model, please refer to the relevant content of the image processing model training method provided in the above embodiment, which will not be repeated here.

[0196] S503 . Obtain a prediction score of the tear trough in the tear trough image to be processed according to the tear trough output data corresponding to the tear trough image to be processed.

[0197] The tear trough output data corresponding to the tear trough image to be processed is a representation of the latent space. Therefore, the tear trough output data corresponding to the tear trough image to be processed can be used as ScoreOut and substituted into formula (1) to obtain the prediction score of the tear trough in the tear trough image to be processed, that is, ScoreSrc in formula (1).

[0198] It should be noted that, according to the pre-obtained grading table and the predicted score of the tear trough in the tear trough image to be processed, the tear trough grading can also be output. For details about the grading table, please refer to Figure 1 Related description in the embodiments.

[0199] In summary, the image processing method of this embodiment obtains a tear trough image to be processed, processes the tear trough image to be processed according to a pre-trained image processing model, obtains tear trough output data corresponding to the tear trough image to be processed, and obtains the predicted score of the tear trough in the tear trough image to be processed according to the tear trough output data corresponding to the tear trough image to be processed. In this embodiment, the tear trough image to be processed is processed through high-quality algorithm modeling, which not only ensures the stability and continuity of the output score, but also ensures the differentiation of the tear trough accuracy, thereby improving the accuracy of the tear trough evaluation.

[0200] Figure 7 FIG. 1 shows a schematic diagram of the structure of an image processing model training device provided in an embodiment of the present application, and the image processing model training device can be integrated in an image processing model training device. Figure 7 As shown, the image processing model training device 60 includes:

[0201] An acquisition module 601 is used to acquire a plurality of tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample;

[0202] The training module 602 is used to perform model training according to the plurality of tear trough image samples to obtain an image processing model.

[0203] Optionally, it also includes:

[0204] The processing module 603 is used to normalize the first annotation score to obtain a tear trough mapping score, process the tear trough mapping score using a preset activation function to obtain the feature map supervision data; and perform global average pooling on the feature map supervision data to obtain the tear trough supervision data.

[0205] Optionally, it also includes:

[0206] A determination module 604 is used to randomly determine, for each tear trough image sample, whether to perform a grid stitching process on the tear trough image sample;

[0207] If yes, the acquisition module 601 is further used to randomly acquire a preset number of other tear trough image samples from the plurality of tear trough image samples, and splice the other tear trough image samples and the tear trough image samples into a grid;

[0208] The acquisition module 601 is specifically used for:

[0209] Normalizing the first annotation score to obtain a first tear groove mapping score;

[0210] Obtaining a second annotation score of the other tear trough image samples in the grid, where the second annotation score represents the severity of the tear trough in the other tear trough image samples;

[0211] The second labeling score is normalized to obtain a second tear trough mapping score, wherein the tear trough mapping score includes the first tear trough mapping score and the second tear trough mapping score.

[0212] Optionally, the training module 602 is specifically configured to:

[0213] For each tear trough image sample, the tear trough image sample is processed according to a preset neural network model to obtain feature map output data;

[0214] Performing global average pooling processing on the feature map output data to obtain tear groove output data corresponding to the feature map output data;

[0215] Obtaining a first loss according to the tear trough output data and the tear trough supervision data;

[0216] Obtaining a second loss according to the feature map output data and the feature map supervision data;

[0217] Model training is performed according to the first loss and the second loss to obtain the image processing model.

[0218] Optionally, the training module 602 is specifically configured to:

[0219] According to a preset sorting method, sorting the plurality of tear trough supervision data annotated by the plurality of tear trough image samples, and combining them in pairs to obtain a plurality of supervision data combinations;

[0220] For each of the supervision data combinations, obtaining a combination sub-loss corresponding to the supervision data combination according to the first tear trough supervision data and the second tear trough supervision data in the supervision data combination;

[0221] Obtaining a sum of a plurality of the combined sub-losses corresponding to a plurality of the supervised data combinations, and obtaining a combined sorting loss according to the sum;

[0222] Model training is performed according to the first loss, the second loss and the combined ranking loss to obtain the image processing model.

[0223] Optionally, the training module 602 is specifically configured to:

[0224] Calculating a first difference between the first tear trough supervision data and the second tear trough supervision data;

[0225] Calculating a second difference between first tear trough output data corresponding to the first tear trough supervision data and second tear trough output data corresponding to the second tear trough supervision data;

[0226] If the first difference is 0, then according to loss = -0.5*log e (1-D)-0.5*log e (D), obtaining the combined ranking sub-loss;

[0227] Otherwise, according to loss = -log e (D), obtaining the combined ranking sub-loss;

[0228] Wherein, D is the second difference, and loss is the combined sorting sub-loss.

[0229] The image processing model training device provided in this embodiment has an implementation process and an implementation principle similar to the image processing model training method provided in the above embodiment, and will not be repeated here.

[0230] Figure 8 FIG. 1 is a schematic diagram showing the structure of an image processing apparatus provided in an embodiment of the present application, and the image processing apparatus can be integrated in an image processing device. Figure 8 As shown, the image processing device 70 includes:

[0231] An acquisition module 701 is used to acquire a tear groove image to be processed;

[0232] A processing module 702 is used to process the tear trough image to be processed according to a pre-trained image processing model to obtain tear trough output data corresponding to the tear trough image to be processed, wherein the image processing model is trained based on a plurality of tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample;

[0233] The acquisition module 701 is further used to acquire the prediction score of the tear trough in the tear trough image to be processed according to the tear trough output data corresponding to the tear trough image to be processed.

[0234] The implementation process and implementation principle of the image processing device provided in this embodiment are similar to the image processing method provided in the above method embodiment, and will not be repeated here.

[0235] Fig. 9 The structure diagram of the image processing model training device provided in the embodiment of the present application is shown as follows: Fig. 9As shown, the image processing model training device 80 includes: a processor 801, a memory 802 and a bus 803. The memory 802 stores a computer program executable by the processor 801. When the image processing model training device 80 is running, the processor 801 communicates with the memory 802 through the bus 803, and the processor 801 executes the computer program to perform the image processing model training method.

[0236] Fig.10 The structure diagram of the image processing device provided by the embodiment of the present application is shown as follows: Fig.10 As shown, the image processing device 90 includes: a processor 901, a memory 902 and a bus 903. The memory 902 stores a computer program executable by the processor 901. When the image processing device 90 is running, the processor 901 communicates with the memory 902 through the bus, and the processor 901 executes the computer program to perform the image processing method.

[0237] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method embodiment is executed.

[0238] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0239] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0240] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for training an image processing model, characterized in that: include: Acquire multiple tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, and the first annotation score represents the severity of the tear trough in the tear trough image sample; Performing model training according to the plurality of tear trough image samples to obtain an image processing model; The feature map supervision data is obtained by the following steps: Normalizing the first annotation score to obtain a tear groove mapping score; Using a preset activation function to process the tear groove mapping score to obtain the feature map supervision data; The tear groove supervision data is obtained by the following steps: The feature map supervision data is subjected to global average pooling processing to obtain the tear groove supervision data.

2. The method according to claim 1, characterized in that Before normalizing the first annotation score to obtain the tear groove mapping score, the method further includes: For each tear trough image sample, randomly determining whether to perform a grid stitching process on the tear trough image sample; If yes, randomly obtaining a preset number of other tear trough image samples from the plurality of tear trough image samples, and splicing the other tear trough image samples and the tear trough image samples into a grid; The step of normalizing the first labeled score to obtain a tear groove mapping score includes: Normalizing the first annotation score to obtain a first tear groove mapping score; Obtaining a second annotation score of the other tear trough image samples in the grid, where the second annotation score represents the severity of the tear trough in the other tear trough image samples; The second labeling score is normalized to obtain a second tear trough mapping score, wherein the tear trough mapping score includes the first tear trough mapping score and the second tear trough mapping score.

3. The method according to claim 1, characterized in that: The performing model training according to the plurality of tear trough image samples to obtain an image processing model comprises: For each tear trough image sample, the tear trough image sample is processed according to a preset neural network model to obtain feature map output data; Performing global average pooling processing on the feature map output data to obtain tear groove output data corresponding to the feature map output data; Obtaining a first loss according to the tear trough output data and the tear trough supervision data; Obtaining a second loss according to the feature map output data and the feature map supervision data; Model training is performed according to the first loss and the second loss to obtain the image processing model.

4. The method according to claim 3, characterized in that The performing model training according to the first loss and the second loss to obtain the image processing model includes: According to a preset sorting method, sorting the plurality of tear trough supervision data annotated by the plurality of tear trough image samples, and combining them in pairs to obtain a plurality of supervision data combinations; For each of the supervision data combinations, obtaining a combination sub-loss corresponding to the supervision data combination according to the first tear trough supervision data and the second tear trough supervision data in the supervision data combination; Obtaining a sum of a plurality of the combined sub-losses corresponding to a plurality of the supervised data combinations, and obtaining a combined sorting loss according to the sum; Model training is performed according to the first loss, the second loss and the combined ranking loss to obtain the image processing model.

5. The method according to claim 4, characterized in that For each of the supervised data combinations, obtaining the combined ranking sub-loss corresponding to the supervised data combination according to the first tear trough supervised data and the second tear trough supervised data in the supervised data combination includes: Calculating a first difference between the first tear trough supervision data and the second tear trough supervision data; Calculating a second difference between first tear trough output data corresponding to the first tear trough supervision data and second tear trough output data corresponding to the second tear trough supervision data; If the first difference is 0, then according to loss = -0.5*log e (1-D)-0.5*log e (D), obtaining the combined ranking sub-loss; Otherwise, according to loss = -log e (D), obtaining the combined ranking sub-loss; Wherein, D is the second difference, and loss is the combined sorting sub-loss.

6. An image processing method, characterized in that: include: Acquire the tear trough image to be processed; Processing the tear trough image to be processed according to a pre-trained image processing model to obtain tear trough output data corresponding to the tear trough image to be processed, wherein the image processing model is trained based on a plurality of tear trough image samples, each of the tear trough image samples is annotated with tear trough supervision data and feature map supervision data corresponding to the tear trough supervision data, the tear trough supervision data is used to indicate a first annotation score of the tear trough in the tear trough image sample, the first annotation score represents the severity of the tear trough in the tear trough image sample, the feature map supervision data is obtained by the following steps: normalizing the first annotation score to obtain a tear trough mapping score, processing the tear trough mapping score with a preset activation function to obtain the feature map supervision data, the tear trough supervision data is obtained by the following steps: performing global average pooling on the feature map supervision data to obtain the tear trough supervision data; According to the tear trough output data corresponding to the tear trough image to be processed, a prediction score of the tear trough in the tear trough image to be processed is obtained.

7. An image processing model training device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores a computer program executable by the processor, and when the image processing model training device is running, the processor and the memory communicate through the bus, and the processor executes the computer program to perform the method described in any one of claims 1 to 5.

8. An image processing device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores a computer program executable by the processor, and when the image processing device is running, the processor communicates with the memory via the bus, and the processor executes the computer program to perform the method according to claim 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images