Training method and device of scar grade evaluation model, equipment and medium
By acquiring and processing multimodal information and using attention mechanism to train scar level evaluation models, the problem of lack of intelligence in scar assessment in the existing technology is solved, and a more scientific and accurate scar level evaluation is achieved.
Patent Information
- Application Number
- CN202411355803.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, scar assessment mainly relies on subjective evaluation by doctors, and the lack of intelligent evaluation methods, resulting in inadequate scientific and accurate assessment.
By obtaining multimodal information of multiple human scars, including scar images, scale scores and blood flow data, preprocessing, feature extraction and weighted fusion, the scar level evaluation model is trained using attention mechanism to achieve intelligent evaluation.
Intelligent and diversified assessment of scar levels has been achieved, the scientificity and accuracy of the assessment has been improved, and human error has been reduced.
Smart Images
Figure CN119230111A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of scar grade assessment, and in particular to a training method, device, equipment and medium for a scar grade assessment model. Background Art
[0002] Scar is the inevitable product of tissue trauma repair, and its formation mechanism is still unclear. It is currently believed that it is a post-traumatic tissue fibrous abnormal proliferative disease caused by various reasons that lead to abnormal proliferation of fibroblasts and massive formation of collagen, resulting in excessive deposition of collagen in the extracellular matrix.
[0003] Currently, scar assessment is generally done by doctors based on their clinical experience and theoretical knowledge, and there are few intelligent assessment methods. Summary of the invention
[0004] The present invention provides a scar grade assessment model training method, device, equipment and medium, which are used to solve the defect of fewer intelligent assessment methods in related technologies and can diversify intelligent assessment methods.
[0005] In a first aspect, the present invention provides a training method for a scar grade assessment model, comprising: Acquire multimodal information of a plurality of human scars; wherein each multimodal information of the human scar includes N single-modal information; N is an integer not less than 2; Preprocessing each of the N unimodal information in the multimodal information to obtain corresponding multiple processed multimodal information; wherein each of the processed multimodal information includes N preprocessed unimodal information; For any of the processed multimodal information, feature extraction is performed in the N pre-processed unimodal information of the processed multimodal information to obtain corresponding N unimodal feature data, weight coefficients of the N unimodal feature data are determined based on the attention mechanism, and weighted fusion is performed on the N unimodal feature data according to the weight coefficients of the N unimodal feature data to obtain feature fusion data corresponding to the processed multimodal information; The model to be trained is trained based on the feature fusion data corresponding to each of the processed multimodal information to obtain a trained scar grade assessment model.
[0006] Optionally, the N single-modal information include at least two of scar images, scar scale scores and blood flow data.
[0007] Optionally, when the N single-modal information include scar images, scar scale scores and blood flow data, the preprocessing of the N single-modal information in each of the multimodal information to obtain corresponding multiple processed multimodal information includes: For any of the multimodal information, if it is determined that the scar image, scar scale score or blood flow data in the multimodal information is invalid information, then the multimodal information is determined to be invalid multimodal information; Deleting each of the invalid multimodal information from the multimodal information of the plurality of human scars to obtain a plurality of valid multimodal information; Each of the valid multimodal information is standardized and normalized respectively to obtain processed multimodal information corresponding to each of the valid multimodal information.
[0008] Optionally, each of the processed multimodal information includes a pre-processed scar image, a pre-processed scale score and pre-processed blood flow data; The respectively standardizing and normalizing each of the valid multimodal information to obtain processed multimodal information corresponding to each of the valid multimodal information includes: Normalizing each scar image in the valid multimodal information to obtain a preprocessed scar image corresponding to each valid multimodal information; Normalizing the scar scale score in each of the valid multimodal information to a first preset interval to obtain a pre-processed scale score corresponding to each of the valid multimodal information; and normalizing the blood flow data in each of the valid multimodal information to a second preset interval to obtain the pre-processed blood flow data corresponding to each of the valid multimodal information; The preprocessed scar image, preprocessed scale score and preprocessed blood flow data corresponding to the same effective multimodal information are taken as a processed multimodal information to obtain the processed multimodal information corresponding to each effective multimodal information.
[0009] Optionally, the first preset interval and the second preset interval are the same interval; the blood flow data includes at least one of the root mean square value, average value, amplitude, peak value, maximum peak difference value and signal period of the blood flow.
[0010] Optionally, determining the weight coefficients of the N unimodal feature data respectively based on the attention mechanism includes: Inputting the N unimodal feature data into the created attention mechanism model respectively, so that the attention mechanism model respectively evaluates the contribution value of the N unimodal feature data to scar grade assessment; For any one of the N unimodal feature data, calculate the square value of the contribution value of the unimodal feature data to the scar level assessment, use the calculated square value as the initial weight coefficient of the unimodal feature data, judge whether the contribution value of the unimodal feature data to the scar level assessment is greater than a set threshold, and obtain a corresponding judgment result, and determine the weight coefficient of the unimodal feature data according to the judgment result and the initial weight coefficient.
[0011] Optionally, determining the weight coefficient of the unimodal feature data according to the judgment result and the initial weight coefficient includes: When the judgment result is that the contribution value of the single-modal feature data to the scar grade assessment is greater than the set threshold, the single-modal feature data is determined to be the dominant feature data, and the initial weight coefficient of the single-modal feature data is multiplied by the set enhancement coefficient to obtain the corresponding product as the weight coefficient of the single-modal feature data; wherein the enhancement coefficient is greater than 1; When the judgment result is that the contribution value of the unimodal feature data to scar grade assessment is not greater than the set threshold, the unimodal feature data is determined to be non-dominant feature data, and the initial weight coefficient of the unimodal feature data is directly used as the weight coefficient of the unimodal feature data.
[0012] In a second aspect, the present invention provides a training device for a scar grade assessment model, comprising: An information acquisition unit, used for acquiring multimodal information of a plurality of human scars; wherein each multimodal information of the human scar includes N single-modal information; N is an integer not less than 2; A preprocessing unit, configured to preprocess the N unimodal information in each of the multimodal information to obtain a corresponding plurality of processed multimodal information; wherein each of the processed multimodal information includes N preprocessed unimodal information; A feature extraction unit, configured to extract features from the N pre-processed unimodal information in any of the processed multimodal information, respectively, to obtain corresponding N unimodal feature data; A weight determination unit, used to determine the weight coefficients of the N unimodal feature data respectively based on an attention mechanism; A feature fusion unit, used to determine the weight coefficients of the N unimodal feature data respectively based on an attention mechanism; The model training unit is used to train the to-be-trained model based on the feature fusion data corresponding to each of the processed multimodal information to obtain a trained scar grade assessment model.
[0013] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the training method of the scar grade assessment model of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the scar grade assessment model training method of the first aspect or any corresponding embodiment thereof.
[0015] The training method, device, equipment and medium of the scar grade assessment model provided by the present invention can obtain multimodal information of multiple human scars; wherein, the multimodal information of each human scar includes N single-modal information; N is an integer not less than 2; the N single-modal information in each multimodal information is preprocessed to obtain corresponding multiple processed multimodal information; wherein, each processed multimodal information includes N preprocessed single-modal information; for any processed multimodal information, feature extraction is performed in the N preprocessed single-modal information of the processed multimodal information respectively to obtain the corresponding N single-modal feature data, the weight coefficients of the N single-modal feature data are determined respectively based on the attention mechanism, and the N single-modal feature data are weightedly fused according to the weight coefficients of the N single-modal feature data to obtain the feature fusion data corresponding to the processed multimodal information; based on the feature fusion data corresponding to each processed multimodal information, the model to be trained is trained to obtain a trained scar grade assessment model. The scar grade assessment model trained by the present invention can perform scar grade assessment on human scars based on multimodal information of human scars, thereby realizing intelligent assessment of human scar grade and diversifying intelligent assessment methods of scar grade. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 A flowchart of a method for training a scar grade assessment model provided by an embodiment of the present invention; Figure 2 A flowchart of another method for training a scar grade assessment model provided by an embodiment of the present invention; Figure 3A flowchart of another method for training a scar grade assessment model provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a training device for a scar grade assessment model provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0019] Combine the following Figure 1-Figure 3 The training method of the scar grade assessment model of the present invention is described.
[0020] like Figure 1 As shown, this embodiment proposes a first scar grade assessment model training method, which may include the following steps: S101. Acquire multimodal information of multiple human scars; wherein the multimodal information of each human scar includes N single-modal information; and N is an integer not less than 2.
[0021] Optionally, the N single-modal information include at least two of scar image, scar scale score and blood flow data. Specifically, scar image, scar scale score and blood flow data can all be single-modal information of human scar.
[0022] The scar image is an image obtained by photographing a scar on a human body.
[0023] Specifically, the scar scale score can be a score obtained by a doctor through subjective evaluation of human scars from multiple aspects according to the scar scale.
[0024] Optionally, the scar scale may be the Vancouver Scar Scale. In this case, the scar scale score may be obtained by a doctor making a subjective score of a human scar based on the Vancouver Scar Scale in terms of pigmentation, scar thickness, vascularity, and flexibility.
[0025] Optionally, the scar scale may be a modified Vancouver Scar Scale, which adds evaluation of the type of pigmentation and the degree of pain and itching perceived by the patient compared to the Vancouver Scar Scale.
[0026] It is understandable that the scar scale score may include scores for multiple aspects, such as scores for pigmentation, scar thickness, vascularity, and flexibility, and may also include a comprehensive score for each aspect.
[0027] The blood flow data may be related data calculated based on the blood flow of scar tissue in the human body. Optionally, the blood flow data includes at least one of the root mean square value, average value, amplitude, peak value, maximum peak difference value and signal period of the blood flow. The maximum peak difference value is the peak-peak value.
[0028] Specifically, this embodiment can detect the tissue blood flow of human scars based on near-infrared spectroscopy technology such as diffusion correlation spectroscopy (DCS) to obtain the tissue blood flow, and calculate the corresponding blood flow data based on the detected tissue blood flow.
[0029] Specifically, this embodiment can simultaneously collect multimodal information of human scars covering all age groups and lesion distributions involving all parts of the body, so as to increase the sample quantity and sample representativeness, and improve the model training effect and generalization ability.
[0030] S102, preprocessing N unimodal information in each multimodal information to obtain corresponding multiple processed multimodal information; wherein each processed multimodal information includes N preprocessed unimodal information.
[0031] Specifically, for any multimodal information of a human scar, this embodiment can preprocess N unimodal information in the multimodal information to obtain N preprocessed unimodal information, and obtain processed multimodal information including the N preprocessed unimodal information.
[0032] Optionally, when the N single-modal information includes scar images, scar scale scores, and blood flow data, step S102 may include: For any multimodal information, if it is determined that the scar image, scar scale score or blood flow data in the multimodal information is invalid information, the multimodal information is determined to be invalid multimodal information; Deleting each invalid multimodal information from the multimodal information of multiple human scars to obtain multiple valid multimodal information; Each valid multimodal information is standardized and normalized respectively to obtain the processed multimodal information corresponding to each valid multimodal information.
[0033] Specifically, this embodiment can screen scar images of various age groups and various lesion distributions involving various parts of the body from the multimodal information of all human scars obtained, identify scar images that do not meet the requirements such as non-scar tissue, and determine them as invalid information.
[0034] Specifically, this embodiment can screen the blood flow data obtained based on the diffusion correlation spectroscopy technology from the multimodal information of all human scars obtained, determine the invalid signals generated due to external environmental interference and other reasons, and determine them as invalid information.
[0035] Specifically, this embodiment can screen the scar scales for various age groups and lesion distributions involving various parts of the body from the multimodal information of all human scars obtained, that is, all scar scale scores, determine the scar scale scores that do not meet the requirements due to various factors such as incomplete scoring, missing part of the information, etc., and determine them as invalid information.
[0036] It can be understood that, for any multimodal information of a human scar, as long as the scar image, scar scale score or blood flow data in the multimodal information is invalid information, this embodiment can determine the multimodal information as invalid multimodal information.
[0037] Specifically, in this embodiment, all invalid multimodal information can be deleted from the multimodal information of all human scars, and all the remaining multimodal information after the deletion is valid multimodal information.
[0038] Optionally, each processed multimodal information includes a pre-processed scar image, a pre-processed scale score, and a pre-processed blood flow data. At this time, the above standardizes and normalizes each valid multimodal information to obtain the processed multimodal information corresponding to each valid multimodal information, including: Normalizing the scar image in each valid multimodal information to obtain a preprocessed scar image corresponding to each valid multimodal information; Normalizing the scar scale score in each valid multimodal information to a first preset interval to obtain a pre-processed scale score corresponding to each valid multimodal information; and normalizing the blood flow data in each valid multimodal information to a second preset interval to obtain the pre-processed blood flow data corresponding to each valid multimodal information; The preprocessed scar image, preprocessed scale score and preprocessed blood flow data corresponding to the same valid multimodal information are taken as a processed multimodal information to obtain the processed multimodal information corresponding to each valid multimodal information.
[0039] Specifically, this embodiment can standardize the scar image in each valid multimodal information, and after flipping, correcting, compressing, cutting and other operations on scar images of different sizes and resolutions, uniformly process them into standardized scar images. For example, after cutting the non-scar tissue part in the scar image, a 512*512 (or other pixel standards) standardized scar image is obtained, that is, a preprocessed scar image.
[0040] Specifically, when the scar scale score includes scores of multiple aspects and the total score, this embodiment can normalize the score of each aspect and the total score respectively, and use the normalized data as a normalized scale score, that is, the pre-processed scale score. In this embodiment, when normalizing a certain type of score, the maximum and minimum values of the type of score can be determined first, and the type of score can be normalized according to the maximum and minimum values of the type of score.
[0041] Specifically, this embodiment can normalize the scores of different aspects (such as color, vascular distribution, etc.) and the total score, and normalize each type of score and the total score to a specific variation interval, such as 0 to 1, or 0 to 100, etc.
[0042] Specifically, this embodiment can normalize each item of the blood flow data, and use the normalized items of data as normalized blood flow data, that is, pre-processed blood flow data. In this embodiment, when normalizing a certain item of the blood flow data, the maximum value and the minimum value of the data can be determined first, and the data can be normalized according to the maximum value and the minimum value of the data.
[0043] Optionally, the first preset interval and the second preset interval are the same interval. It should be noted that both the normalized scale score and the normalized blood flow data will be used for the training of the scar grade assessment model. Whether the data magnitude is unified will affect the training effect of the model. Therefore, this embodiment can normalize the scar scale score and the blood flow data to the same interval to ensure the model training effect.
[0044] S103 . For any processed multimodal information, feature extraction is performed respectively in N pre-processed unimodal information of the processed multimodal information to obtain corresponding N unimodal feature data.
[0045] It can be understood that each processed multimodal information has corresponding N unimodal feature data.
[0046] Optionally, when the processed multimodal information includes a preprocessed scar image, a preprocessed scale score, and a preprocessed blood flow data, this embodiment can perform feature extraction on the preprocessed scar image, the preprocessed scale score, and the preprocessed blood flow data of the processed multimodal information, respectively, to obtain corresponding first unimodal feature data, second unimodal feature data, and third unimodal feature data. It can be understood that the first unimodal feature data is feature data extracted from the preprocessed scar image, the second unimodal feature data is feature data extracted from the preprocessed scale score, and the third unimodal feature data is feature data extracted from the preprocessed blood flow data.
[0047] S104. Determine the weight coefficients of N unimodal feature data respectively based on the attention mechanism.
[0048] It should be noted that, in this embodiment, for the N unimodal feature data extracted from any processed multimodal information, the contribution of the N unimodal feature data to the model target task, namely scar grade assessment, can be studied based on the attention mechanism. The size of the contribution depends on the accuracy of the feature data used to achieve the model target task. The greater the contribution, the greater the weight coefficient of the feature data, and the smaller the contribution, the smaller the weight coefficient of the feature data.
[0049] Specifically, in this embodiment, for any processed multimodal information, weight coefficients corresponding to N unimodal feature data can be determined based on the attention mechanism.
[0050] Optionally, when the processed multimodal information includes preprocessed scar images, preprocessed scale scores and preprocessed blood flow data, this embodiment can determine the weight coefficients of the corresponding first unimodal feature data, second unimodal feature data and third unimodal feature data based on the attention mechanism for any processed multimodal information.
[0051] Optionally, step S104 may include: Inputting N unimodal feature data into the created attention mechanism model respectively, so that the attention mechanism model evaluates the contribution values of the N unimodal feature data to scar grade assessment respectively; For any unimodal feature data among the N unimodal feature data, calculate the square value of the contribution value of the unimodal feature data to the scar grade assessment, use the calculated square value as the initial weight coefficient of the unimodal feature data, judge whether the contribution value of the unimodal feature data to the scar grade assessment is greater than the set threshold, and obtain the corresponding judgment result, and determine the weight coefficient of the unimodal feature data according to the judgment result and the initial weight coefficient.
[0052] The threshold value may be set by a technician according to actual conditions, and is not limited in this embodiment.
[0053] The initial weight coefficient of the unimodal feature data is the square of the contribution value corresponding to the unimodal feature data.
[0054] Specifically, when the N unimodal feature data corresponding to the processed multimodal information include the above-mentioned first unimodal feature data, second unimodal feature data and third unimodal feature data, this embodiment can input the first unimodal feature data, the second unimodal feature data and the third unimodal feature data into the attention mechanism model respectively, and the attention mechanism model can respectively evaluate the contribution values of the first unimodal feature data, the second unimodal feature data and the third unimodal feature data to the scar grade assessment.
[0055] Specifically, this embodiment can determine the initial weight coefficients of the first unimodal feature data, the second unimodal feature data and the third unimodal feature data according to their contribution values to scar grade assessment.
[0056] Afterwards, this embodiment can respectively compare the contribution values of the first unimodal feature data, the second unimodal feature data and the third unimodal feature data to the scar grade assessment with the set threshold, and finally determine the weight coefficients of the first unimodal feature data, the second unimodal feature data and the third unimodal feature data based on the comparison results and the initial weight coefficient.
[0057] Optionally, determining a weight coefficient of the single-modal feature data according to the judgment result and the initial weight coefficient includes: When the judgment result is that the contribution value of the single-modal feature data to the scar grade assessment is greater than the set threshold, the single-modal feature data is determined to be the dominant feature data, and the initial weight coefficient of the single-modal feature data is multiplied by the set enhancement coefficient to obtain the corresponding product as the weight coefficient of the single-modal feature data; wherein the enhancement coefficient is greater than 1; When the judgment result is that the contribution value of the unimodal feature data to the scar grade assessment is not greater than the set threshold, the unimodal feature data is determined to be non-dominant feature data, and the initial weight coefficient of the unimodal feature data is directly used as the weight coefficient of the unimodal feature data.
[0058] Among them, the reinforcement coefficient can be determined according to the attention mechanism model and network adaptation, and this embodiment does not limit it.
[0059] S105 , performing weighted fusion on the N unimodal feature data according to their weight coefficients to obtain feature fusion data corresponding to the processed multimodal information.
[0060] Specifically, this embodiment can perform weighted fusion on the N unimodal feature data extracted from any processed multimodal information based on the weight coefficients of the N unimodal feature data to obtain feature fusion data corresponding to the processed multimodal information. Among them, this embodiment can perform corresponding calculations on the N unimodal feature data based on the vector weighted average algorithm and the weight coefficients of the N unimodal feature data, and the calculation result obtained is the feature fusion data.
[0061] S106 , training the to-be-trained model based on the feature fusion data corresponding to each processed multimodal information to obtain a trained scar grade assessment model.
[0062] The model to be trained may be a pre-trained scar grade assessment model or a grade classification model that has not been pre-trained.
[0063] Specifically, in this embodiment, the feature fusion data corresponding to each processed multimodal information can be used as training data, and the training data can be used to train the model to be trained until a scar grade assessment model that meets the performance requirements is obtained through training and used as the trained scar grade assessment model.
[0064] Specifically, when the feature fusion data corresponding to the multimodal information after any processing is used to train the model to be trained, the accurate scar level corresponding to the feature fusion data can be determined in advance. Among them, the accurate scar level can be obtained by experts evaluating the corresponding human scars. Afterwards, the present embodiment can input the feature fusion data into the model to be trained to evaluate the scar level, and obtain the scar level evaluated by the model to be trained. Afterwards, the present embodiment can calculate the corresponding loss function according to the difference between the evaluated scar level and the accurate scar level. If the loss function does not meet the requirements, the model parameters of the model to be trained can be updated according to the damage function, and the updated model to be trained can be used as the new model to be trained. Subsequently, other feature fusion data can be used to train the new model to be trained until a scar level evaluation model that meets the performance requirements is obtained.
[0065] Optionally, in this embodiment, all the obtained feature fusion data may be divided into a training set, a validation set and a prediction set, and a scar grade assessment model that meets the performance requirements may be trained by using the training set, the validation set and the prediction set.
[0066] In practical applications, after obtaining the trained scar grade assessment model, this embodiment can use the trained scar grade assessment model to perform scar grade assessment.
[0067] Specifically, when the above-mentioned N unimodal information include scar images, scar scale scores and blood flow data, this embodiment can first obtain multimodal information of a human scar to be graded, that is, the scar image, scar scale score and blood flow data of the human scar, preprocess the scar image, scar scale score and blood flow data of the human scar to obtain a preprocessed scar image, a preprocessed scar scale score and preprocessed blood flow data, and perform feature extraction on the preprocessed scar image, the preprocessed scar scale score and the preprocessed blood flow data, respectively, to obtain corresponding first unimodal feature data, second unimodal feature data and third unimodal feature data. Based on the attention mechanism, weight coefficients of the first unimodal feature data, the second unimodal feature data and the third unimodal feature data are determined, and based on the weight coefficients, the first unimodal feature data, the second unimodal feature data and the third unimodal feature data are weightedly fused to obtain corresponding feature fusion data, and the feature fusion data is input into the trained scar grade assessment model to perform scar grade assessment to obtain the corresponding scar grade.
[0068] The training method of the scar grade assessment model proposed in the present embodiment can obtain multimodal information of multiple human scars; wherein, the multimodal information of each human scar includes N unimodal information; N is an integer not less than 2; the N unimodal information in each multimodal information is preprocessed to obtain corresponding multiple processed multimodal information; wherein, each processed multimodal information includes N preprocessed unimodal information; for any processed multimodal information, feature extraction is performed in the N preprocessed unimodal information of the processed multimodal information respectively to obtain the corresponding N unimodal feature data, the weight coefficients of the N unimodal feature data are determined respectively based on the attention mechanism, and the N unimodal feature data are weightedly fused according to the weight coefficients of the N unimodal feature data to obtain the feature fusion data corresponding to the processed multimodal information; the model to be trained is trained based on the feature fusion data corresponding to each processed multimodal information to obtain a trained scar grade assessment model. The scar grade assessment model trained in this embodiment can perform scar grade assessment on human scars based on multimodal information of human scars, thereby realizing intelligent assessment of human scar grade and diversifying intelligent assessment methods of scar grade.
[0069] based on Figure 1This embodiment proposes a second training method for the scar grade assessment model. In this method, after obtaining N unimodal feature data corresponding to each processed multimodal information, for any N unimodal feature data corresponding to the processed multimodal information, this embodiment can construct global feature data based on the N unimodal feature data, determine the weight coefficients of the N unimodal feature data and the global feature data respectively, and perform weighted fusion on the N unimodal feature data and the global feature data based on the weight coefficient to obtain feature fusion data.
[0070] It should be noted that the contribution value of the single-modal feature data to the scar grade assessment is used to measure the improvement of the accuracy of the scar grade assessment by the single-modal feature data.
[0071] After obtaining N unimodal feature data corresponding to the processed multimodal information, this embodiment can determine the relationship between different unimodal feature data based on the attention mechanism. This relationship refers to the impact on the model evaluation accuracy, that is, improvement or reduction, after splicing another unimodal feature data on the basis of a certain unimodal feature data and using the spliced global feature data to achieve the model target task.
[0072] Specifically, this embodiment can freely splice N single-modal feature data, and the result of splicing is the global feature data. This embodiment can determine the contribution value of the global feature data to the scar level assessment based on the attention mechanism. If the contribution value of the global feature data to the scar level assessment is greater than the contribution value corresponding to a certain single-modal feature data participating in the splicing, the global feature data can be determined as valid global feature data. If the contribution value of the global feature data to the scar level assessment is not greater than the contribution value corresponding to all the single-modal feature data participating in the splicing, the global feature data is determined as invalid global feature data.
[0073] For example, when the N unimodal feature data include the first unimodal feature data, the second unimodal feature data and the third unimodal feature data, the contribution value corresponding to the first unimodal feature data is Acc1, the contribution value corresponding to the second unimodal feature data is Acc2, and the contribution value corresponding to the third unimodal feature data is Acc3. At this time, the initial weight coefficients of the first unimodal feature data, the second unimodal feature data and the third unimodal feature data are Acc1, Acc2 and Acc3 respectively. 2 、Acc2 2 and Acc3 2 .
[0074] The first unimodal feature data and the second unimodal feature data are spliced to obtain the corresponding global feature data B12, the first unimodal feature data and the third unimodal feature data are spliced to obtain the corresponding global feature data B13, the second unimodal feature data and the third unimodal feature data are spliced to obtain the corresponding global feature data B13, and the first unimodal feature data, the second unimodal feature data and the third unimodal feature data are spliced to obtain the corresponding global feature data B123.
[0075] Based on the attention mechanism, the contribution value Acc12 corresponding to the global feature data B12 is determined, the contribution value Acc13 corresponding to the global feature data B13 is determined, and based on the attention mechanism, the contribution value Acc123 corresponding to the global feature data B123 is determined.
[0076] If Acc12>Acc1 or Acc12>Acc2, the global feature data B12 can be determined as valid global feature data. If Acc12 is not greater than Acc1 and not greater than Acc2, the global feature data B12 can be determined as invalid global feature data.
[0077] If Acc13>Acc1 or Acc13>Acc3, the global feature data B13 can be determined as valid global feature data. If Acc13 is not greater than Acc1 and not greater than Acc3, the global feature data B13 can be determined as invalid global feature data.
[0078] Similarly, it can be determined whether the global feature data B23 and B123 are valid global feature data.
[0079] Similarly, this embodiment may also determine the weight coefficient of the effective global feature data by referring to the process of determining the initial weight coefficient of the unimodal feature data and the process of determining whether the unimodal feature data is the dominant feature data.
[0080] Afterwards, this embodiment can perform weighted fusion on the N unimodal feature data and all corresponding valid global feature data based on the weight coefficients of the N unimodal feature data and the corresponding valid global feature data to obtain feature fusion data. At this time, this embodiment can still use each obtained feature fusion data to train the training model to obtain a trained scar grade assessment model.
[0081] The training method of the scar grade assessment model proposed in this embodiment can construct global feature data based on unimodal feature data, generate feature fusion data based on unimodal feature data and global feature data, and use the feature fusion data for model training, thereby further improving the model training effect and ensuring the accuracy of model assessment.
[0082] like Figure 2As shown, in another training method of scar grade assessment model proposed in this embodiment, the method includes: S201. Simultaneously collect multimodal scar information covering all age groups and lesion distributions involving all parts of the body, including the collection of scar images and Vancouver Scar Scale scores, as well as the collection of multiple data such as scar tissue blood flow based on diffusion correlation spectroscopy technology.
[0083] Specifically, this embodiment can collect a two-dimensional image of scar tissue, and the doctor can score the patient's scar using the Vancouver Scar Scale. At the same time, the scar tissue blood flow and other data are obtained based on the diffusion correlation spectroscopy technology, and the scar tissue under test is irradiated with a near-infrared light signal, and the near-infrared scattered light spot signal after diffuse reflection is received. The changes in the tissue blood flow of the scar tissue under test are detected in real time, and multiple parameters such as the root mean square value used to characterize the scar grade characteristics are calculated based on the tissue blood flow data. Multiple parameters include the root mean square value, average value, amplitude, peak value, peak-to-peak value and period of the tissue blood flow calculated based on the tissue blood flow data.
[0084] S202: Preprocess the collected data to establish a multimodal scar dataset.
[0085] Specifically, multimodal information includes scar images, scar tissue blood flow and other data, Vancouver Scar Scale, etc., involving three different types of information, including image information, multi-parameter signals, and doctor's subjective scoring information. According to the preprocessing of multimodal information, scar images are cleaned and standardized, multi-parameter signals such as scar tissue blood flow are cleaned and normalized, and Vancouver Scar Scale scores are cleaned and normalized. The preprocessed scar images, multi-parameter signals such as scar tissue blood flow, and Vancouver Scar Scale scores are sorted and divided into training sets, validation sets, and prediction sets to establish a multimodal scar dataset.
[0086] S203. Propose a deep learning scar grade assessment model based on multimodal information fusion to achieve end-to-end automatic assessment.
[0087] S204. Use the multimodal dataset to complete the training of the proposed model and realize automatic assessment of scar grade.
[0088] This embodiment can use the model evaluation algorithm to perform in-depth analysis on three different types of information, namely, image information of the scar tissue being tested, multi-parameter signal information, and doctor's subjective scoring information, thereby achieving accurate evaluation of the scar grade.
[0089] like Figure 3As shown, this embodiment can obtain scar image information, scar DCS tissue blood flow multi-parameter signal information and Vancouver Scar Scale score information of any human scar, and extract features of the three information respectively based on the feature extraction module in the multimodal feature extraction module, and input the extracted three single-modal feature data into the multimodal feature fusion module for weighted fusion to obtain feature fusion data and input it into the classifier, train the scar grade assessment performance of the classifier to obtain a trained scar grade assessment model, and use the trained scar grade assessment model to implement scar grade assessment.
[0090] The scar grade assessment model training method proposed in this embodiment can obtain a trained scar grade assessment model, and the trained scar grade assessment model can be used to perform intelligent scar grade assessment on human scars, thereby diversifying the intelligent scar grade assessment methods.
[0091] like Figure 4 As shown, this embodiment provides a training device for a scar grade assessment model, which may include: The information acquisition unit 401 is used to acquire multimodal information of multiple human scars; wherein the multimodal information of each human scar includes N single-modal information; N is an integer not less than 2; A preprocessing unit 402 is used to preprocess the N unimodal information in each multimodal information to obtain corresponding multiple processed multimodal information; wherein each processed multimodal information includes N preprocessed unimodal information; A feature extraction unit 403 is used to extract features from N pre-processed unimodal information of any processed multimodal information to obtain corresponding N unimodal feature data; A weight determination unit 404, used to determine weight coefficients of N unimodal feature data respectively based on an attention mechanism; A feature fusion unit 405 is used to determine the weight coefficients of N single-modal feature data respectively based on the attention mechanism; The model training unit 406 is used to train the to-be-trained model based on the feature fusion data corresponding to each processed multimodal information to obtain a trained scar grade assessment model.
[0092] It should be noted that the processing of the information acquisition unit 401, the preprocessing unit 402, the feature extraction unit 403, the weight determination unit 404, the feature fusion unit 405 and the model training unit 406 and the beneficial effects thereof can be respectively referred to in Figure 1 Steps S101 to S106 in the above are not described in detail.
[0093] Optionally, the N single-modality information include at least two of scar images, scar scale scores, and blood flow data.
[0094] Optionally, when the N single-modal information includes scar images, scar scale scores, and blood flow data, the preprocessing unit 402 is further configured to: For any multimodal information, if it is determined that the scar image, scar scale score or blood flow data in the multimodal information is invalid information, the multimodal information is determined to be invalid multimodal information; Deleting each invalid multimodal information from the multimodal information of multiple human scars to obtain multiple valid multimodal information; Each valid multimodal information is standardized and normalized respectively to obtain the processed multimodal information corresponding to each valid multimodal information.
[0095] Optionally, each processed multimodal information includes a pre-processed scar image, a pre-processed scale score, and pre-processed blood flow data; The preprocessing unit 402 is further configured to: Normalizing the scar image in each valid multimodal information to obtain a preprocessed scar image corresponding to each valid multimodal information; Normalizing the scar scale score in each valid multimodal information to a first preset interval to obtain a pre-processed scale score corresponding to each valid multimodal information; and normalizing the blood flow data in each valid multimodal information to a second preset interval to obtain the pre-processed blood flow data corresponding to each valid multimodal information; The preprocessed scar image, preprocessed scale score and preprocessed blood flow data corresponding to the same valid multimodal information are taken as a processed multimodal information to obtain the processed multimodal information corresponding to each valid multimodal information.
[0096] Optionally, the first preset interval and the second preset interval are the same interval; the blood flow data includes at least one of the root mean square value, average value, amplitude, peak value, maximum peak difference value and signal period of the blood flow.
[0097] Optionally, the weight determination unit 404 is further configured to: Inputting N unimodal feature data into the created attention mechanism model respectively, so that the attention mechanism model evaluates the contribution values of the N unimodal feature data to scar grade assessment respectively; For any unimodal feature data among the N unimodal feature data, calculate the square value of the contribution value of the unimodal feature data to the scar grade assessment, use the calculated square value as the initial weight coefficient of the unimodal feature data, judge whether the contribution value of the unimodal feature data to the scar grade assessment is greater than the set threshold, and obtain the corresponding judgment result, and determine the weight coefficient of the unimodal feature data according to the judgment result and the initial weight coefficient.
[0098] Optionally, the weight determination unit 404 is further configured to: When the judgment result is that the contribution value of the single-modal feature data to the scar grade assessment is greater than the set threshold, the single-modal feature data is determined to be the dominant feature data, and the initial weight coefficient of the single-modal feature data is multiplied by the set enhancement coefficient to obtain the corresponding product as the weight coefficient of the single-modal feature data; wherein the enhancement coefficient is greater than 1; When the judgment result is that the contribution value of the unimodal feature data to the scar grade assessment is not greater than the set threshold, the unimodal feature data is determined to be non-dominant feature data, and the initial weight coefficient of the unimodal feature data is directly used as the weight coefficient of the unimodal feature data.
[0099] The training device of the scar grade assessment model proposed in this embodiment can obtain multimodal information of multiple human scars; wherein, the multimodal information of each human scar includes N unimodal information; N is an integer not less than 2; the N unimodal information in each multimodal information is preprocessed to obtain corresponding multiple processed multimodal information; wherein, each processed multimodal information includes N preprocessed unimodal information; for any processed multimodal information, feature extraction is performed in the N preprocessed unimodal information of the processed multimodal information respectively to obtain the corresponding N unimodal feature data, the weight coefficients of the N unimodal feature data are determined respectively based on the attention mechanism, and the N unimodal feature data are weightedly fused according to the weight coefficients of the N unimodal feature data to obtain the feature fusion data corresponding to the processed multimodal information; based on the feature fusion data corresponding to each processed multimodal information, the model to be trained is trained to obtain a trained scar grade assessment model. The scar grade assessment model trained in this embodiment can perform scar grade assessment on human scars based on multimodal information of human scars, thereby realizing intelligent assessment of human scar grade and diversifying intelligent assessment methods of scar grade.
[0100] The training device of the scar grade assessment model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0101] The embodiment of the present invention also provides a computer device having the above Figure 4 The training device of the scar grade assessment model is shown.
[0102] See also Figure 5 , a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.
[0103] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0104] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0105] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0106] The memory 20 may include a volatile memory, such as a random access memory. The memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive. The memory 20 may also include a combination of the above-mentioned types of memory.
[0107] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0108] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A training method for a scar grade assessment model, characterized in that: include: Acquire multimodal information of a plurality of human scars; wherein each multimodal information of the human scar includes N single-modal information; N is an integer not less than 2; Preprocessing each of the N unimodal information in the multimodal information to obtain corresponding multiple processed multimodal information; wherein each of the processed multimodal information includes N preprocessed unimodal information; For any of the processed multimodal information, feature extraction is performed in the N pre-processed unimodal information of the processed multimodal information to obtain corresponding N unimodal feature data, weight coefficients of the N unimodal feature data are determined based on the attention mechanism, and weighted fusion is performed on the N unimodal feature data according to the weight coefficients of the N unimodal feature data to obtain feature fusion data corresponding to the processed multimodal information; The model to be trained is trained based on the feature fusion data corresponding to each of the processed multimodal information to obtain a trained scar grade assessment model.
2. The method according to claim 1, characterized in that: The N single-modality information include at least two of scar images, scar scale scores, and blood flow data.
3. The method according to claim 2, characterized in that When the N single-modal information include scar images, scar scale scores and blood flow data, the preprocessing of the N single-modal information in each of the multimodal information to obtain corresponding multiple processed multimodal information includes: For any of the multimodal information, if it is determined that the scar image, scar scale score or blood flow data in the multimodal information is invalid information, then the multimodal information is determined to be invalid multimodal information; Deleting each of the invalid multimodal information from the multimodal information of the plurality of human scars to obtain a plurality of valid multimodal information; Each of the valid multimodal information is standardized and normalized respectively to obtain processed multimodal information corresponding to each of the valid multimodal information.
4. The method according to claim 3, characterized in that: Each of the processed multimodal information includes a pre-processed scar image, a pre-processed scale score and pre-processed blood flow data; The respectively standardizing and normalizing each of the valid multimodal information to obtain processed multimodal information corresponding to each of the valid multimodal information includes: Normalizing each scar image in the valid multimodal information to obtain a preprocessed scar image corresponding to each valid multimodal information; Normalizing the scar scale score in each of the valid multimodal information to a first preset interval to obtain a pre-processed scale score corresponding to each of the valid multimodal information; and normalizing the blood flow data in each of the valid multimodal information to a second preset interval to obtain the pre-processed blood flow data corresponding to each of the valid multimodal information; The preprocessed scar image, preprocessed scale score and preprocessed blood flow data corresponding to the same effective multimodal information are taken as a processed multimodal information to obtain the processed multimodal information corresponding to each effective multimodal information.
5. The method according to claim 4, characterized in that The first preset interval and the second preset interval are the same interval; the blood flow data includes at least one of the root mean square value, average value, amplitude, peak value, maximum peak difference value and signal period of the blood flow.
6. The method according to any one of claims 1 to 5, characterized in that The step of respectively determining weight coefficients of the N unimodal feature data based on the attention mechanism includes: Inputting the N unimodal feature data into the created attention mechanism model respectively, so that the attention mechanism model respectively evaluates the contribution value of the N unimodal feature data to scar grade assessment; For any one of the N unimodal feature data, calculate the square value of the contribution value of the unimodal feature data to the scar level assessment, use the calculated square value as the initial weight coefficient of the unimodal feature data, judge whether the contribution value of the unimodal feature data to the scar level assessment is greater than a set threshold, and obtain a corresponding judgment result, and determine the weight coefficient of the unimodal feature data according to the judgment result and the initial weight coefficient.
7. The method according to claim 6, characterized in that Determining the weight coefficient of the unimodal feature data according to the judgment result and the initial weight coefficient includes: When the judgment result is that the contribution value of the single-modal feature data to the scar grade assessment is greater than the set threshold, the single-modal feature data is determined to be the dominant feature data, and the initial weight coefficient of the single-modal feature data is multiplied by the set enhancement coefficient to obtain the corresponding product as the weight coefficient of the single-modal feature data; wherein the enhancement coefficient is greater than 1; When the judgment result is that the contribution value of the unimodal feature data to scar grade assessment is not greater than the set threshold, the unimodal feature data is determined to be non-dominant feature data, and the initial weight coefficient of the unimodal feature data is directly used as the weight coefficient of the unimodal feature data.
8. A training device for a scar grade assessment model, characterized in that: include: An information acquisition unit, used for acquiring multimodal information of a plurality of human scars; wherein each multimodal information of the human scar includes N single-modal information; N is an integer not less than 2; A preprocessing unit, configured to preprocess the N unimodal information in each of the multimodal information to obtain a corresponding plurality of processed multimodal information; wherein each of the processed multimodal information includes N preprocessed unimodal information; A feature extraction unit, configured to extract features from the N pre-processed unimodal information in any of the processed multimodal information, respectively, to obtain corresponding N unimodal feature data; A weight determination unit, used to determine the weight coefficients of the N unimodal feature data respectively based on an attention mechanism; A feature fusion unit, used to determine the weight coefficients of the N unimodal feature data respectively based on an attention mechanism; The model training unit is used to train the to-be-trained model based on the feature fusion data corresponding to each of the processed multimodal information to obtain a trained scar grade assessment model.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the training method of the scar grade assessment model according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the training method for the scar grade assessment model according to any one of claims 1 to 7.