Method for screening and auxiliary diagnosis of lesions based on deep learning of multi-modal skin images

Abstract

This application discloses a lesion screening and auxiliary diagnosis method based on deep learning of multimodal skin images, belonging to the field of lesion screening technology. It collects multimodal skin image data under different conditions, extracts color change information, texture difference information, edge offset information, and quantitative information on the regional diffusion range corresponding to the lesion area from the multimodal skin image data, and constructs dynamic image feature samples. Based on the dynamic image feature samples, it analyzes the feature migration pattern of the same lesion under different conditions, annotates the direction of continuous change, the magnitude of abnormal change, and risk correlation indicators, and generates corresponding state correlation parameters. This application, through the construction of dynamic features of multimodal skin images, lesion state correlation analysis, and image correspondence evaluation, continuously describes the lesion change process, optimizes the criteria for disease classification, and improves the accuracy and stability of complex skin lesion screening and auxiliary diagnosis.

Description

Technical Field

[0001] This application relates to the field of lesion screening technology, specifically to a lesion screening and auxiliary diagnosis method based on deep learning of multimodal skin images. Background Technology

[0002] Lesion screening and auxiliary diagnosis refer to the process of discovering, identifying, classifying, and assessing the risks of potential abnormal pathological changes in human tissues or organs using information from multiple sources such as medical imaging, clinical information, and test data, and providing auxiliary support for doctors to make further diagnostic and treatment decisions. "Lesion screening" focuses on rapidly identifying suspected abnormal areas from large populations or large-scale imaging data, such as pigmentation changes, nodules, ulcers, inflammatory areas, or neoplastic lesions on the skin surface, to achieve early detection and risk warning. "Auxiliary diagnosis," after discovering suspected lesions, combines imaging characteristics, lesion morphology, structural information, and relevant patient data to analyze the lesion type, severity, and benign / malignant risk, providing diagnostic references for clinicians. This process does not replace doctors in making the final diagnosis, but rather improves the efficiency, accuracy, and consistency of lesion identification through computer analysis technology.

[0003] Deep learning combined with multimodal skin imaging for lesion screening and assisted diagnosis refers to an intelligent method that integrates multiple types of skin image data and utilizes deep learning models to achieve automatic detection, feature extraction, classification, and diagnostic assistance for skin diseases. Specifically, multimodal skin images include, but are not limited to, ordinary skin photographs, dermoscopic images, reflection confocal microscopy images, optical coherence tomography (OCT) images, and other medical imaging data. Different image modalities can reflect the characteristics of skin lesions from different perspectives, such as surface morphology, pigment distribution, cell structure, and tissue layers. Deep learning models, through algorithms such as convolutional neural networks, visual Transformers, and multimodal fusion networks, automatically learn the color, texture, boundary, spatial structure, and pathological features in different images, and improve lesion localization, classification, and risk prediction capabilities through cross-modal information fusion. For example, in the screening process for skin diseases such as melanoma, basal cell carcinoma, dermatitis, and psoriasis, the system can automatically mark suspected lesion areas, analyze their image characteristics, and output the probability or risk level of the disease category, providing dermatologists with more objective, rapid, and accurate auxiliary diagnostic evidence.

[0004] In existing technologies, auxiliary diagnostic models for skin lesions typically perform feature extraction and disease classification based on a single acquisition of skin image data. They primarily learn the color, texture, boundary, and structural information contained in the current image. However, actual skin lesions exhibit dynamic evolutionary characteristics and may be affected by factors such as inflammatory changes, pigment migration, tissue proliferation, necrosis, crusting, and changes in surface structure at different stages of development. This results in significant differences in the characteristics of image data of the same lesion acquired at different times.

[0005] Because existing models typically lack a temporal correlation mechanism for changes in the life cycle of lesions, they cannot effectively establish the correspondence between images at different stages, making it difficult for the models to determine whether different images originate from the same lesion area.

[0006] Furthermore, in the process of multimodal skin image fusion, different image modes reflect the surface morphology, tissue structure and cell change information of lesions. When lesions undergo irregular changes, there may be inconsistencies in feature expression between different modalities, making the model susceptible to local abnormal changes and resulting in lesion category judgment bias.

[0007] Therefore, existing technologies have problems such as being unable to effectively adapt to the dynamic changes in skin lesions, insufficient multimodal image correlation capabilities, and low stability of auxiliary diagnostic results.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] The purpose of this application is to provide a lesion screening and auxiliary diagnosis method based on deep learning of multimodal skin images, so as to solve the problems in the background art mentioned above.

[0010] To achieve the above objectives, this application provides the following technical solution: a lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning, comprising the following steps: Multimodal skin image data under different conditions were collected, and color change information, texture difference information, edge offset information, and quantitative information of regional diffusion range corresponding to the lesion area were extracted from the multimodal skin image data to construct dynamic image feature samples. Based on dynamic image feature samples, the feature migration pattern of the same lesion under different states is analyzed, and the direction of continuous change, the magnitude of abnormal change and risk association indicators are labeled to generate corresponding state association parameters. Adjust the feature focus range in the lesion image recognition process according to the state association parameters, and filter key visual information based on the direction of change in the state association parameters. By utilizing key visual information, we can establish correspondence analysis relationships between different types of multimodal skin images, calculate the consistency of lesion change characteristics corresponding to different image sources, and output continuous lesion status assessment information. The judgment criteria in the auxiliary diagnosis process are updated based on the lesion status assessment information, and the disease classification is corrected according to the dynamic change pattern corresponding to the lesion status assessment information to complete lesion screening and auxiliary diagnosis.

[0011] Preferably, dynamic image feature samples are constructed using color change information, texture difference information, edge offset information, and quantification information of regional diffusion range. The specific steps are as follows: Multimodal skin image data corresponding to the same lesion area under different states are acquired, and the correspondence between image data under different states is determined based on image acquisition location, lesion location area and image acquisition time information. Based on the correspondence, the location of lesion areas under different states is matched, and the lesion state change features are extracted based on color change information, texture difference information, edge offset information, and quantitative information on the extent of regional diffusion. The color change information, texture difference information, edge offset information, and regional diffusion range quantification information corresponding to different states are arranged in the order of state change. The dynamic image feature description units are combined to construct a dynamic image feature sample containing the trajectory of lesion changes.

[0012] Preferably, the lesion state change features are extracted based on color change information, texture difference information, edge offset information, and regional diffusion range quantification information. Specifically, this includes: performing position matching based on the spatial position change relationship of the lesion area in the image, and retaining the image information of the overall lesion area and the related areas around the lesion.

[0013] Preferably, the direction of continuous change, the magnitude of abnormal change, and risk-related indicators are labeled to generate corresponding state-related parameters. The specific steps are as follows: Based on dynamic image feature samples, feature changes between different states of the same lesion are extracted, and feature transfer rules are established according to color changes, texture differences, edge offset changes, and regional diffusion changes. The direction of continuous change is determined based on the feature migration law, and the magnitude of abnormal change is determined based on the feature change between adjacent states. The continuous change direction, abnormal change magnitude, color abnormal change index, texture abnormal change index, edge change index, regional diffusion change index, and multimodal change consistency index are correlated and recorded to generate state correlation parameters.

[0014] Preferably, the abnormal change amplitude is determined based on the characteristic change amount between different states, and the abnormal change amplitudes of color change amplitude, texture change amplitude, edge offset amplitude, and region diffusion amplitude are described based on the degree of change of the corresponding characteristic change amount during the continuous state change process.

[0015] Preferably, key visual information is filtered based on the direction of change in the state association parameters. The specific steps are as follows: Obtain the color change correlation parameters, texture change correlation parameters, edge change correlation parameters, region diffusion correlation parameters, and risk change correlation parameters from the state correlation parameters, and determine the feature focus area in the lesion image recognition process based on the corresponding change direction, change magnitude, and risk correlation information; Based on the defined feature focus area, obtain the corresponding color change information, texture difference information, edge offset information and region diffusion range quantification information, filter key visual information according to the continuous change direction in the state association parameters, and establish the continuous change association relationship between visual information in different states. Correlation analysis was performed using the changing trends of key visual information obtained from screening and state-related parameters. Based on the consistency index of multimodal changes, the attention level of corresponding visual information in different image sources was adjusted to reduce the impact of local abnormal manifestations on the lesion category identification results.

[0016] Preferably, correlation analysis is performed based on the changing trends of the key visual information and state-related parameters obtained through screening, and the corresponding visual information in different image sources is screened based on the multimodal change consistency index. Visual information that maintains the same direction of change in multiple image sources is identified as key visual information.

[0017] Preferably, continuous lesion status assessment information is output, and the specific steps are as follows: Based on key visual information, the correspondence between image sources, lesion regions, and status changes is determined according to the location and status change information of lesion areas in different image sources and the trend of key visual information changes. Based on the correspondence of image sources, lesion regions, and state changes, color change information, texture difference information, edge offset information, and quantitative information of regional diffusion range are correlated and described. The consistency of lesion change features is determined based on the consistency of change direction, the similarity of change amplitude, and the continuity of change process. Based on the consistency of lesion change characteristics and the continuous change direction, abnormal change magnitude and risk association indicators in the status association parameters, the data are associated and recorded to generate continuous lesion status assessment information, which includes current lesion status description information, status change direction information, change continuity information and risk change trend information.

[0018] Preferably, based on the correspondence between image sources, the correspondence between lesion regions, and the correspondence between state changes, key visual information from different image sources is arranged according to image source, state time, and direction of change to establish a sequence describing the changes of the same lesion in different image sources.

[0019] Preferably, the disease classification is corrected according to the dynamic change pattern corresponding to the lesion status assessment information to complete lesion screening and auxiliary diagnosis. The specific steps are as follows: Acquire continuous lesion status assessment information, and incorporate current lesion status description information, status change direction information, change continuity information, and risk change trend information into the disease classification judgment process to update the judgment basis in the auxiliary diagnosis process; Based on the updated judgment criteria, the disease classification direction is adjusted according to the information on the direction of state change and the information on the trend of risk change. The change trends corresponding to the information on color change, texture difference, edge offset and the quantitative information on the extent of regional diffusion are described in association. Based on the revised disease classification, the system outputs lesion category information, risk change trend information, state change direction information, and auxiliary diagnostic reference information. It also updates lesion state assessment information based on new multimodal skin imaging data to complete lesion screening and auxiliary diagnosis.

[0020] The technical effects and advantages provided by this application in the above technical solution are as follows: This application collects multimodal skin image data under different conditions and constructs dynamic image feature samples containing information on color changes, texture differences, edge offsets, and quantitative information on the extent of regional diffusion. This allows for a continuous description of the changes in the same lesion at different stages. By combining state-related parameters to analyze the direction of lesion changes and the magnitude of abnormal changes, it improves the ability to identify the dynamic evolution of complex skin lesions and reduces the impact of single image feature differences on lesion screening results.

[0021] This application establishes a correspondence analysis relationship between different types of multimodal skin images, generates continuous lesion status assessment information based on the consistency of lesion change characteristics corresponding to different image sources, and corrects the disease classification direction according to the dynamic change law, so that the auxiliary diagnosis process can comprehensively utilize the change information of different image layers, improve the multimodal image correlation ability, and enhance the stability of lesion screening and auxiliary diagnosis results. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0023] Figure 1 This is a flowchart illustrating the overall method of this application. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0025] like Figure 1 As shown, this application provides a lesion screening and assisted diagnosis method based on multimodal skin image deep learning, including the following steps: Step 1: Collect multimodal skin image data under different conditions, extract color change information, texture difference information, edge offset information, and regional diffusion range quantification information corresponding to the lesion area in the multimodal skin image data, and construct dynamic image feature samples through color change information, texture difference information, edge offset information, and regional diffusion range quantification information. Dynamic image feature samples are constructed by using color change information, texture difference information, edge offset information, and quantification information of region diffusion range. The specific steps are as follows: Multimodal skin imaging data under different conditions was acquired, and a correspondence was established based on the imaging manifestations of the same lesion area under different conditions. During multiple skin examinations of the patient, multimodal skin imaging data corresponding to the same lesion area was acquired. This multimodal skin imaging data included one or more types of data, such as ordinary skin imaging data, dermoscopy imaging data, reflection confocal microscopy imaging data, and optical coherence tomography (OCT) imaging data. For the multimodal skin imaging data acquired at different examination stages, corresponding patient information, image acquisition location, lesion location, and image acquisition time information were recorded. Based on this information, the correspondence between the imaging data under different conditions was determined, ensuring that the lesion area imaging data acquired at different acquisition stages corresponded to the same skin lesion.

[0026] In the specific mapping process, the skin region containing the lesion is determined based on the image acquisition location. Then, based on the spatial changes of the lesion region in the images, positional matching is performed on the lesion region in different states to maintain spatial correspondence. Considering that the same lesion may exhibit enlargement, edge movement, color changes, and local structural alterations at different stages of development, image information of the entire lesion region and its surrounding associated areas is preserved. This ensures that the acquired multimodal skin image data in different states simultaneously includes information on changes in the lesion itself and changes in the surrounding area.

[0027] Furthermore, the acquired multimodal skin image data is divided into states according to the degree of lesion change. The image data corresponding to different states are sorted according to the lesion change process to form a continuous image data set including the initial state, the changed state, and the abnormal change state. This provides basic data for the subsequent extraction of color change information, texture difference information, edge offset information, and quantitative information of regional diffusion range corresponding to the lesion area.

[0028] Based on multimodal skin image data under different states, information on color changes, texture differences, edge offsets, and quantitative information on the extent of diffusion in the lesion area are extracted, and dynamic image feature samples are constructed based on the relationship between changes in different states.

[0029] To analyze color change information, the color distribution in the lesion area under different states is analyzed. By obtaining the proportion of different color areas within the lesion area, the degree of color intensity change, and the change in color distribution location, the color change information of the lesion area with state changes is determined. For example, the color distribution of the lesion area under adjacent states is compared to obtain the proportion of color area increase, the direction of color change, and the range of color change, which are used to describe the pigmentation change process of the lesion.

[0030] To address texture differences, the internal texture of lesion areas under different conditions is analyzed. By acquiring changes in texture density, texture continuity, and local texture abnormalities, the degree of difference in texture features between lesion areas under different conditions is determined.

[0031] To analyze the edge offset information, the boundary position of the lesion area under different states is analyzed. By obtaining the moving distance of the lesion edge position between adjacent states, the range of edge morphology changes, and the direction of local boundary changes, the offset of the lesion edge with the state changes is quantitatively described.

[0032] To quantify the extent of regional spread, the coverage of lesion areas under different conditions is calculated. By obtaining the proportion of change in lesion area, the direction of regional growth, and the amount of change in the spread range, the process of change in the spatial range of lesions is quantitatively described.

[0033] Furthermore, color change information, texture difference information, edge offset information, and regional diffusion range quantification information are arranged in the corresponding state order, and the correlation between different information is established. The lesion area feature information corresponding to each state is used as a dynamic image feature description unit. The dynamic image feature description units of multiple states are combined in the order of time change to construct a dynamic image feature sample containing the lesion change trajectory.

[0034] Among them, dynamic image feature samples not only record the visual appearance of the lesion area in a single state, but also record the trend of color, texture, edge and range change between different states, so that the subsequent analysis process can obtain continuous image feature basis based on the lesion change process.

[0035] Based on the constructed dynamic image feature samples, the lesion change information in different modal skin image data is correlated and described, establishing a correspondence between changes in multimodal images. Specifically, the process involves analyzing the lesion region change information under different states recorded in the dynamic image feature samples to ensure that information reflecting the same lesion state from different image sources remains correlated. For example, ordinary skin image data is used to describe lesion surface color changes and quantitative information on regional diffusion range; dermoscopy image data is used to describe lesion texture differences and edge offset information; reflection confocal microscopy image data is used to describe changes at the local cellular level of the lesion; and optical coherence tomography image data is used to describe changes at the tissue level of the lesion.

[0036] To address the issue of different image representation methods in skin imaging data of different modalities, dynamic image feature description units in different image sources are associated based on the lesion location, acquisition time, and state change information corresponding to various types of image data. This establishes a corresponding relationship between color change information, texture difference information, edge offset information, and regional diffusion range quantification information in different modal images.

[0037] Furthermore, based on the sequence of state changes in the dynamic image feature samples, the direction and magnitude of lesion change information in different modal image data are recorded, enabling different image sources to jointly describe the change process of the same lesion under different states. For example, when the recorded lesion area in ordinary skin image data expands, the changes in edge offset and texture difference information in dermoscopic image data are obtained through correspondence, while the change information of the corresponding area in reflection confocal microscopy image data and optical coherence tomography image data are also obtained. Thus, the dynamic image feature samples can contain descriptive information of the change process of the same lesion from different image layers.

[0038] Through the above methods, dynamic image feature samples not only contain information on color changes, texture differences, edge offsets, and quantitative information on the extent of regional diffusion in the lesion area, but also information on the correlation and changes between skin image data of different modalities under different states, providing continuous and complete image feature basis for subsequent lesion screening and auxiliary diagnosis based on the dynamic change patterns of lesions.

[0039] Step 2: Based on dynamic image feature samples, analyze the feature migration pattern of the same lesion under different states, label the direction of continuous change, the magnitude of abnormal change and risk association indicators, and generate corresponding state association parameters. The following steps are taken to label the direction of continuous change, the magnitude of abnormal change, and risk-related indicators, and generate corresponding state-related parameters: Based on dynamic image feature samples, feature changes between different states of the same lesion are extracted, and feature transfer rules are established according to the feature changes corresponding to different states. Specifically, multiple lesion state feature descriptions arranged in the order of state changes in dynamic image feature samples are obtained. The lesion state feature description information includes color change information, texture difference information, edge offset information, and quantification information of the area diffusion range in the corresponding state.

[0040] Correspondence analysis is performed on the lesion state feature description information between adjacent states. The color change information, texture difference information, edge offset information, and quantified information of the region diffusion range corresponding to the previous state are compared with the information corresponding to the current state to obtain the feature changes between different states. The color change is determined based on the change in the proportion of color regions between different states; the texture difference change is determined based on the change in the distribution of texture regions; the edge offset change is determined based on the change in the distance of the lesion boundary position; and the region diffusion change is determined based on the change in the proportion of the lesion region coverage. Furthermore, the various feature changes obtained between different states are arranged according to the order of lesion state changes, forming a feature transfer sequence containing multiple state nodes and the relationship between the state nodes.

[0041] The feature transfer sequence describes the feature transfer of the same lesion as it changes from the current state to the next, enabling the image features of the lesion, which were originally independent at different times, to form a continuous relationship of change. For example, for multiple sets of multimodal skin image data acquired continuously for the same lesion, when the amount of color change, the amount of edge offset, and the amount of regional diffusion continuously increase, the corresponding change information is recorded in the feature transfer sequence to represent the feature transfer pattern of the lesion area changing from the current state to the expanded state.

[0042] The above method transforms static feature information in dynamic image feature samples into feature migration rules with state change relationships, enabling subsequent processes to determine the direction of continuous change, the magnitude of abnormal change, and risk-related indicators based on the trend of lesion changes.

[0043] The direction of continuous change is determined based on the feature migration pattern, and the magnitude of abnormal changes is determined based on the degree of feature change between different states. The determination process is as follows: based on the feature change amounts corresponding to multiple consecutive state nodes in the feature migration sequence, trend analysis is performed on the color change amount, texture difference change amount, edge offset change amount, and regional diffusion change amount to determine the direction of continuous change corresponding to various image features. The direction of continuous change is used to describe the stable change trend of lesions in different state changes, including the direction of color enhancement, the direction of increased texture abnormalities, the direction of edge expansion, and the direction of change in regional extent.

[0044] In the specific implementation process, when the color change amount corresponding to multiple consecutive states maintains an increasing trend, the color enhancement trend is taken as the continuous change direction corresponding to the color change information; when the edge offset change amount corresponding to multiple consecutive states maintains the same movement direction, the edge movement direction is taken as the continuous change direction corresponding to the edge offset information; when the area diffusion change amount corresponding to multiple consecutive states maintains an increasing trend, the area expansion direction is taken as the continuous change direction corresponding to the area diffusion range quantification information.

[0045] Furthermore, the magnitude of abnormal changes is determined based on the amount of feature changes between adjacent states in the feature transfer sequence and the continuous change trend formed between multiple states. Specifically, the amount of color change, texture difference change, edge offset change, and region diffusion change between different states are recorded according to their corresponding feature types, and the magnitude of abnormal changes is determined based on the degree of change of the corresponding feature amounts during the continuous state change process.

[0046] When the change in a feature corresponding to a certain state changes significantly relative to the trend of changes in previous states, an abnormal change is described for the color change amplitude, texture change amplitude, edge offset amplitude, or region diffusion amplitude corresponding to that state. For example, if the proportion of change in lesion area remains stable during continuous state changes, but the proportion of change corresponding to the quantitative information of region diffusion range in a certain state increases rapidly, the change in region diffusion range is recorded as an abnormal change amplitude. Similarly, if the lesion edge position maintains a similar trend of change, but the edge offset distance increases in a certain state, the change in edge offset is recorded as an abnormal change amplitude.

[0047] By determining the direction of continuous change and the magnitude of abnormal change, dynamic image feature samples can reflect the directional and degree information during the lesion change process, providing a clearer data basis for the subsequent generation of state-related parameters.

[0048] Risk correlation indicators are established based on feature migration patterns, continuous change direction, and abnormal change magnitude, and corresponding state correlation parameters are generated based on these indicators. Specifically, the generated state correlation parameters are as follows: Based on lesion change information recorded in dynamic image feature samples under different states, the changing trends of color change information, texture difference information, edge offset information, and regional diffusion range quantification information are correlated with lesion risk changes to establish risk correlation indicators. These risk correlation indicators include color abnormality change indicators, texture abnormality change indicators, edge change indicators, regional diffusion change indicators, and multimodal change consistency indicators.

[0049] Among them, the color abnormality change index is determined based on the changes in the proportion of color areas, the changes in color depth, and the changes in color distribution range of lesion areas under different states, and is used to describe the correlation between the degree of lesion color change and the trend of risk change; the texture abnormality change index is determined based on the degree of change in texture density, the changes in texture continuity, and the changes in the range of abnormal texture areas of lesion areas, and is used to describe the correlation between the degree of increase in lesion texture difference and the trend of risk change; the edge change index is determined based on the distance of change in lesion boundary position, the degree of change in boundary morphology, and the changes in abnormal boundary areas, and is used to describe the correlation between the range of lesion edge movement, irregular changes, and the trend of risk change; the regional diffusion change index is determined based on the proportion of change in lesion area, the regional diffusion speed, and the diffusion direction, and is used to describe the correlation between the process of change in lesion coverage and the trend of risk change; the multimodal change consistency index is determined based on the degree of matching between the direction of color change, direction of texture change, direction of edge change, and direction of regional diffusion change of corresponding lesion areas in different types of multimodal skin image data, and is used to describe the degree of consistency of the direction of change of corresponding lesions in different image sources.

[0050] Furthermore, based on the direction of change, the magnitude of abnormal changes, and risk correlation indicators corresponding to the feature migration patterns, different state information in the dynamic image feature samples is parameterized to generate state correlation parameters. The specific generation process is as follows: The continuous change direction corresponding to different states is used as the state change trend parameter, the abnormal change magnitude is used as the state change degree parameter, and the risk correlation index is used as the risk correlation parameter. These parameters are then linked and recorded according to the order of lesion state changes to form state correlation parameters representing the dynamic change process of the same lesion. These state correlation parameters can simultaneously describe the color change trend, texture change trend, edge change trend, regional diffusion trend, and risk change trend of the lesion under different states.

[0051] For different types of multimodal skin image data, based on the image acquisition time, lesion location, and corresponding status information, the color change information in ordinary skin image data, the texture difference information and edge offset information in dermoscopy image data, the local change information in reflection confocal microscopy image data, and the tissue change information in optical coherence tomography image data are respectively associated with the corresponding status association parameters, so that the lesion change information corresponding to different image sources maintains a consistent description.

[0052] By generating state-related parameters, the lesion screening and auxiliary diagnosis process can be analyzed based on the changing patterns of the same lesion under different states, avoiding disease classification based solely on single-state image features, and improving the continuity and stability of auxiliary diagnosis results in cases of complex lesions.

[0053] Step 3: Adjust the feature focus area in the lesion image recognition process according to the state association parameters, and filter key visual information based on the direction of change in the state association parameters to reduce the impact of local abnormal manifestations on the lesion category recognition results. The key visual information is filtered based on the direction of change in the state association parameters. The specific steps are as follows: The feature focus area in lesion image recognition is determined based on the changes in state association parameters, and the focus priority of different regions is adjusted according to the degree of change corresponding to different types of state association parameters. Specifically, color change association parameters, texture change association parameters, edge change association parameters, region diffusion association parameters, and risk change association parameters included in the state association parameters are obtained. Based on the change direction, change magnitude, and risk association information corresponding to each state association parameter, different visual regions in the lesion image are analyzed accordingly.

[0054] The areas of focus for feature analysis include the lesion center, lesion edge, color change areas, texture abnormality areas, and area diffusion. For color change areas, based on the direction and degree of color change corresponding to the color change correlation parameters, the locations where color distribution changes within the lesion area are identified, with areas showing increased color change range, continuous color variation, and migration of color distribution location designated as key areas of focus. For texture abnormality areas, based on the direction and degree of texture difference change corresponding to the texture change correlation parameters, areas showing continuous texture variation, texture density variation, and increased local texture abnormalities within the lesion area are identified, with areas corresponding to texture change trends designated as key areas of focus. For lesion edge areas, based on the direction of edge movement, edge offset range, and edge morphology changes corresponding to the edge change correlation parameters, the areas where the lesion edge position changes are identified, with areas showing continuous edge movement and boundary changes designated as key areas of focus. For area diffusion areas, based on the direction and degree of change in the area of ​​diffusion corresponding to the area of ​​diffusion correlation parameters, the locations of lesion area expansion are identified, with newly added areas and areas corresponding to the diffusion direction designated as key areas of focus.

[0055] Furthermore, based on the degree of change corresponding to the parameters associated with different types of states, the priority of different feature attention ranges is adjusted. When the degree of change corresponding to a certain region is high and the direction of change is consistent with the direction of change of the overall state of the lesion, the attention of that region in the lesion image recognition process is increased; when the degree of change corresponding to a certain region is low or the direction of change is inconsistent with the direction of change of the overall state of the lesion, the attention of that region in the lesion image recognition process is reduced.

[0056] By using the above method, the feature focus area is no longer fixed to a single image region, but is dynamically adjusted according to the lesion change process reflected by the state association parameters, so that the lesion image recognition process can obtain relevant visual information around the actual change area of ​​the lesion.

[0057] Based on the adjusted feature focus area, key visual information is filtered according to the direction of change in the state association parameters, and a continuous change relationship between visual information in different states is established. Specifically, color change information, texture difference information, edge offset information, and quantified information of region diffusion range corresponding to the feature focus area are obtained, and different visual information is filtered according to the continuous change direction in the state association parameters.

[0058] Regarding color change information, based on the continuous change direction recorded by the color change association parameters, the proportion of color areas, color intensity, and color distribution range of the lesion area under different states are analyzed accordingly. When the color change information maintains the same change direction in multiple consecutive states, the corresponding color change information is identified as key visual information; when the color change information exists only in a single state and does not form a continuous change relationship with other states, the attention level of the color change information is reduced.

[0059] For texture difference information, based on the direction of change recorded by texture change correlation parameters, the changes in texture density, texture continuity, and abnormal texture regions under different states are analyzed, and visual information that continuously conforms to the direction of texture change is identified as key visual information. For edge offset information, based on the direction of change recorded by edge change correlation parameters, the distance of change in lesion edge position, the range of edge movement, and the changes in boundary morphology are analyzed, and visual information consistent with the continuous direction of edge change is identified as key visual information.

[0060] Based on the quantitative information of regional spread, and according to the changing direction of the regional spread-related parameters, the change ratio of lesion area, the spread direction, and the changes of newly added areas are analyzed, and visual information consistent with the regional spread trend is identified as key visual information.

[0061] Furthermore, based on the state association parameters corresponding to the multimodal change consistency index, corresponding visual information in different types of multimodal skin image data is correlated and filtered. When the change direction of the corresponding lesion area in ordinary skin image data, dermoscopy image data, reflection confocal microscopy image data, and optical coherence tomography image data is consistent, the visual information in the corresponding image sources is jointly identified as key visual information; when the change direction of the corresponding lesion area in different image sources is different, the degree of attention to different visual information is adjusted according to the risk change association parameters and the continuous change direction.

[0062] By using the above methods, key visual information is not determined solely based on a single image state, but rather by filtering based on the direction of lesion change under different states and the relationship between changes in different image sources, thereby improving the correlation between key visual information and the actual process of lesion change.

[0063] The judgment criteria in the lesion image recognition process are adjusted based on the key visual information obtained through screening, and the impact of local abnormal manifestations on the lesion category recognition results is reduced based on state correlation parameters. Specifically, the changing trends of the key visual information and the corresponding state correlation parameters are analyzed, and the degree of influence of different visual manifestations is adjusted according to whether the key visual information conforms to the overall change direction of the lesion.

[0064] For areas of local inflammation, local pigmentation changes, surface crusting, and local abnormalities generated during image acquisition, the direction of continuous change in state-related parameters is used to determine whether it is consistent with the overall change process of the lesion. When a local abnormality exists only in a single state and does not show the same direction of change in continuous state changes, the local abnormality is identified as non-primary change information, reducing its impact on lesion category identification results. When a local abnormality is consistent with the direction of color change, texture change, edge change, or area diffusion in multiple states, the local abnormality is identified as visual information related to lesion changes, improving its reference value in the lesion category identification process.

[0065] Furthermore, based on the multimodal change consistency index in the state association parameters, a comprehensive judgment is made on the local visual manifestations in different image sources. When a local change appears only in a single image source, while other image sources do not show a corresponding change trend, the impact of this local change on the lesion category identification result is reduced; when multiple image sources show the same change direction, the attention to the corresponding visual information is increased. For example, when a local color change appears in a normal skin image, but no corresponding change is found in dermoscopy, reflection confocal microscopy, or optical coherence tomography images, the impact of this color change information is reduced based on the multimodal change consistency index; when multiple image sources show a lesion edge diffusion trend, the attention to this edge change information is increased based on the state association parameters.

[0066] By adjusting the feature focus range according to the state association parameters and filtering key visual information based on the direction of change, the lesion image recognition process can pay more attention to information that conforms to the dynamic change pattern of the lesion, reduce the adverse effects of local abnormal manifestations, single state changes and differences in single image sources on the lesion category recognition results, and improve the stability and accuracy of complex skin lesion screening and auxiliary diagnosis.

[0067] Step 4: Utilize key visual information to establish correspondence analysis relationships between different types of multimodal skin images, calculate the consistency of lesion change characteristics corresponding to different image sources, and output continuous lesion status assessment information. The specific steps for outputting continuous lesion status assessment information are as follows: First, a correspondence analysis relationship is established between different types of multimodal skin images based on key visual information. The association between corresponding lesion regions is determined based on the location and state changes of lesion areas from different image sources, as well as the changing trends of key visual information. The specific process for determining the association between corresponding lesion regions is as follows: The key visual information obtained from the aforementioned steps is acquired. This key visual information includes color change information, texture difference information, edge offset information, and quantitative information on the extent of regional diffusion, all of which reflect the trend of lesion status changes. For different types of multimodal skin image data, lesion areas from different image sources are described according to image acquisition time, lesion location, and corresponding status number, so that lesion areas from different image sources correspond to the same lesion location under the same status.

[0068] In the specific implementation process, key visual information of the corresponding lesion areas from ordinary skin images, dermoscopy images, reflection confocal microscopy images, and optical coherence tomography (OCT) images is acquired from each image source and arranged in relation to the order of lesion state changes. For color change information and quantification of regional diffusion range in ordinary skin images, the corresponding regions are determined based on the location of the lesion areas and correlated with texture difference information and edge offset information in dermoscopy images, ensuring a correspondence between surface visual changes and local morphological changes. For edge offset information and texture difference information in dermoscopy images, based on the spatial location of the lesion area and the time of state change, they are correlated with local change information in reflection confocal microscopy images, ensuring that lesion change information reflected at different image layers corresponds to the same stage of change. For tissue change information in OCT images, based on the location and state change information of the lesion area, it is correlated with color change, texture change, and regional diffusion change information from other image sources.

[0069] Furthermore, key visual information from different image sources is arranged according to image source, state time, and direction of change to establish a state correspondence between image sources. This ensures that the change information of the same lesion in different image sources is placed in a corresponding change description sequence. The correspondence analysis includes image source correspondence, lesion region correspondence, and state change correspondence. Among them, the image source correspondence is used to describe the information source relationship between different types of multimodal skin images, the lesion region correspondence is used to describe the positional relationship of corresponding lesion regions in different images, and the state change correspondence is used to describe the change relationship of the same lesion in different states in different image sources.

[0070] By establishing corresponding analytical relationships among key visual information from different types of multimodal skin images, different image sources can continuously describe the changes in the same lesion, providing a basis for subsequent calculation of the consistency of lesion change characteristics from different image sources.

[0071] Based on the established correspondence analysis, the consistency of lesion change characteristics corresponding to different image sources is analyzed. The consistency of lesion change characteristics among different image sources is determined according to the consistency of change direction, the similarity of change magnitude, and the continuity of the change process. Specifically, based on the correspondence between different image sources, the key visual information changes of corresponding lesion regions in each image source are obtained, and color change information, texture difference information, edge offset information, and quantitative information of regional diffusion range are respectively correlated and described.

[0072] For color change characteristics, the direction and degree of color change in the corresponding lesion areas from different image sources are analyzed. When the corresponding lesion areas from multiple image sources all show color enhancement, color range expansion, or color distribution change trends, the color change characteristics are determined to have a consistent relationship. For texture difference characteristics, the direction of texture change, the range of texture abnormality, and the continuity of texture appearance in the corresponding lesion areas from different image sources are analyzed. When the corresponding lesion changes from multiple image sources all show the same texture change trend, the texture difference characteristics are determined to have a consistent relationship. For edge offset characteristics, the distance of boundary position change, edge movement direction, and boundary change range of the corresponding lesion areas from different image sources are analyzed. When the lesion edges from different image sources all show the same movement direction or change trend, the edge change characteristics are determined to have a consistent relationship. For quantitative information on regional diffusion range, the proportion, diffusion direction, and change rate of the corresponding lesion area range from different image sources are analyzed. When the lesion area range changes from different image sources maintain the same trend, the regional diffusion change characteristics are determined to have a consistent relationship.

[0073] Furthermore, based on the consistency of the direction of change, the similarity of the magnitude of change, and the continuity of change of key visual information from different image sources, the consistency of lesion change features from different image sources is comprehensively described. When the direction of lesion change in multiple image sources is consistent and the corresponding magnitude of change is in the same trend, the consistency of the lesion change features is increased; when the change information in a certain image source is only shown in a single state and does not form a continuous change relationship with other image sources, the consistency of the lesion change features corresponding to that image source is decreased.

[0074] Furthermore, the differences in changes between different image sources are processed based on the multimodal change consistency index. When key visual information from multiple image sources shows the same direction of change, the corresponding key visual information is identified as information that can reflect the continuous change process of the lesion. When there are differences in the direction of change between different image sources, the difference information is associated and described based on the continuous change direction, abnormal change amplitude, and risk association index in the state association parameters, thereby reducing the impact of differences from a single image source on the overall lesion status assessment.

[0075] By utilizing the correlation between the direction, degree, and continuity of change among different image sources, the consistency of lesion change characteristics corresponding to different image sources can be determined, so that multimodal skin images not only maintain information correspondence, but can also jointly describe the dynamic change process of the same lesion.

[0076] Based on the consistency of lesion change characteristics from different imaging sources and state-related parameters, continuous lesion state assessment information is generated to describe the changes of the same lesion under different states. Specifically, the consistency of lesion change characteristics from different imaging sources, the trend of key visual information changes, and the direction of continuous change, the magnitude of abnormal change, and risk-related indicators in the state-related parameters are linked and recorded, and continuous lesion state assessment information is generated according to the order of lesion state changes.

[0077] Lesion status assessment information includes current lesion status description, direction of status change, degree of continuity of change, and risk trend information. Specifically, the current lesion status description indicates the color, texture, edge appearance, and extent of the lesion area in the current state; the direction of status change indicates the direction of color, texture, edge appearance, and diffusion as the lesion progresses from its current state to the next; the degree of continuity of change indicates the consistency of lesion change characteristics across different imaging sources; and the risk trend information indicates the correspondence between the lesion change process and risk-related indicators.

[0078] In practice, lesion change information under different states is continuously recorded based on the consistency of lesion change characteristics from different image sources. For example, when ordinary skin images, dermoscopy images, reflective confocal microscopy images, and optical coherence tomography images all show an increasing trend of lesion area expansion, edge changes, and texture abnormalities, the corresponding change information is recorded as continuous lesion state change information, and the corresponding risk change trend is described according to risk association indicators. When there are some differences in changes between different image sources, the change information from different image sources is comprehensively described based on the multimodal change consistency index and the change direction information in the state association parameters, so that the lesion state assessment information can retain key change content that conforms to the overall change trend. Furthermore, lesion state assessment information formed under multiple consecutive states is associated and recorded according to the order of lesion changes, so that the output lesion state assessment information can reflect the continuous change process of the lesion from the initial state, the changing state, to the abnormal change state.

[0079] By utilizing key visual information, we can establish correspondence analysis relationships between different types of multimodal skin images, calculate the consistency of lesion change characteristics corresponding to different image sources, and output continuous lesion status assessment information. This enables lesion change information from different image sources to form a unified and continuous state description, providing a basis for judgment based on the dynamic change pattern of lesions in the subsequent auxiliary diagnostic process, and improving the stability and accuracy of screening and auxiliary diagnosis of complex skin lesions.

[0080] Step 5: Update the judgment criteria in the auxiliary diagnosis process based on the lesion status assessment information, and correct the disease classification direction according to the dynamic change pattern corresponding to the lesion status assessment information to complete the lesion screening and auxiliary diagnosis. The disease classification is corrected based on the dynamic changes in the lesion status assessment information to complete lesion screening and auxiliary diagnosis. The specific steps are as follows: The judgment criteria in the auxiliary diagnostic process are updated based on lesion status assessment information. Information on the current lesion status, direction of status change, degree of continuity of change, and risk trend is incorporated into the disease classification judgment process. The specific implementation process is as follows: This process acquires continuous lesion status assessment information, including the color, texture, edge, and extent of lesions under different states, as well as the consistency of lesion change characteristics across different image sources. Based on the current lesion status description in the assessment information, the visual appearance of the lesion region under the current acquisition state is recorded, and color change information, texture difference information, edge offset information, and quantitative information on the extent of regional diffusion are used as the basis for disease classification under the current state. Based on the direction of state change information, the direction of color change, texture change, edge change, and regional diffusion during the transition from the current state to the next state is correlated and described, enabling the auxiliary diagnostic process to obtain information on lesion change trends. Based on the continuity of change information, the consistency of corresponding lesion change characteristics across different types of multimodal skin image data is recorded, enabling the auxiliary diagnostic process to distinguish between local changes observed from a single image source and lesion change trends reflected by multiple image sources. Based on the risk change trend information, the information corresponding to the color abnormality change index, texture abnormality change index, edge change index, regional diffusion change index and multimodal change consistency index during the lesion change process is introduced into the judgment criteria, so that the disease classification judgment process can be adjusted according to the lesion risk change process.

[0081] In practice, when the lesion status assessment information indicates that the lesion area is continuously darkening in color, continuously expanding in edge, and continuously increasing in area, and multiple types of multimodal skin image data show the same trend of change, the degree of reference for the judgment of the disease classification related to the trend of change is increased; when the lesion status assessment information indicates that a certain local area shows abnormal changes only in a single image source or in a single state, and does not show the corresponding trend of change in other states, the influence of the local abnormality on the judgment of disease classification is reduced.

[0082] Using lesion status assessment information as a dynamic basis for judgment in the auxiliary diagnosis process, the disease classification process no longer relies solely on the single image manifestation in the current state, but combines the continuous change information of lesions under different states, thereby improving the completeness of the judgment basis in the auxiliary diagnosis process.

[0083] Based on the updated judgment criteria, the disease classification direction is revised according to the dynamic change pattern corresponding to the lesion status assessment information, so that the disease classification results can reflect the state transition relationship during the lesion change process. Specifically, the correspondence between the current disease classification direction and the dynamic change pattern of lesions is adjusted based on the state change direction information and risk change trend information in the lesion status assessment information.

[0084] For the classification correction corresponding to the direction of color change, the color change characteristics corresponding to different disease types are described in association based on the continuous changes in the color performance of the lesion area. When the lesion status assessment information shows that the color area continues to expand and the color intensity continues to change, and this trend is related to the risk performance of the corresponding disease type, the disease classification is enhanced. For the classification correction corresponding to the direction of texture change, the texture performance corresponding to the disease type is re-associated based on the continuity of texture difference information under different states. When the abnormal texture changes in the lesion area continue to increase, and the corresponding texture change trend is shown in different image sources, the disease classification related to this texture change is improved. For the classification correction corresponding to the direction of edge change, the boundary change factors in the disease classification judgment are adjusted based on the lesion edge offset information and the quantitative information of the area diffusion range. When the lesion edge continues to move outward and the area diffusion change trend remains continuous, the disease classification is corrected based on the risk change trend in the lesion status assessment information.

[0085] Furthermore, based on the changing relationships between lesion status assessment information under different states, the disease classification direction is continuously adjusted to adapt to the performance differences of lesions as their states change. For example, if the same lesion initially presents as low-risk, but subsequently exhibits increased color changes, irregular edge changes, and increased regional spread, the original disease classification direction is adjusted based on the continuous lesion status assessment information, enabling the auxiliary diagnostic process to reflect the risk change trend brought about by the lesion status change. For situations where there are performance differences between different types of multimodal skin images, the disease classification reference information from different image sources is comprehensively adjusted based on the continuity of changes in lesion status assessment information and the consistency information of multimodal changes. When multiple image sources support the same lesion change trend, the corresponding disease classification direction is enhanced based on consistent change information; when there are differences in changes between different image sources, classification reference information that conforms to the overall change pattern is retained based on the direction of status change and the risk change trend.

[0086] By using the above methods, the disease classification direction is corrected based on the dynamic change pattern corresponding to the lesion status assessment information, so that the disease classification process can be transformed from static image judgment to continuous judgment based on the lesion change process.

[0087] Based on the revised disease classification, lesion screening and auxiliary diagnosis are completed, and diagnostic reference information corresponding to the dynamic status of the lesions is output. The output process is as follows: Based on the updated judgment criteria and revised disease classification guidelines, the lesion screening results are comprehensively described, outputting diagnostic results including lesion category information, risk change trend information, state change direction information, and auxiliary diagnostic reference information. In specific implementation, the current lesion state description information in the lesion state assessment information is mapped to the revised disease classification guidelines, ensuring the output reflects the possible disease type corresponding to the lesion in the current state. The state change direction information is correlated with changes in the disease classification guidelines, allowing the output to reflect the development trend of the lesion from its initial state to its current state. The continuity of change information is combined with the risk change trend information, enabling the output to reflect the common descriptions of the same lesion's change process from different imaging sources.

[0088] Furthermore, based on the dynamic changes in the lesion status assessment information, the judgment criteria in the auxiliary diagnostic process are continuously updated, so that subsequently acquired lesion image data can continue to participate in the analysis of lesion status changes. When new multimodal skin image data corresponds to the same lesion area, the corresponding key visual information is obtained based on the new image data, and the description of the lesion change process is updated in combination with the existing lesion status assessment information, so that the disease classification can be adjusted according to the changes in lesion status.

[0089] By using the above methods, the judgment criteria in the auxiliary diagnosis process are updated based on the lesion status assessment information, and the disease classification direction is corrected according to the dynamic change pattern corresponding to the lesion status assessment information. This completes the lesion screening and auxiliary diagnosis, enabling the auxiliary diagnosis process to comprehensively utilize the dynamic change information in multimodal skin imaging data, thereby improving the continuity, stability, and accuracy of disease identification in complex skin lesion cases.

[0090] The foregoing has only described certain exemplary embodiments of this application by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of this application. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of this application.

Claims

1. A lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning, characterized in that, Includes the following steps: Multimodal skin image data under different conditions were collected, and color change information, texture difference information, edge offset information, and quantitative information of regional diffusion range corresponding to the lesion area were extracted from the multimodal skin image data to construct dynamic image feature samples. Based on dynamic image feature samples, the feature migration pattern of the same lesion under different states is analyzed, and the direction of continuous change, the magnitude of abnormal change and risk association indicators are labeled to generate corresponding state association parameters. Adjust the feature focus range in the lesion image recognition process according to the state association parameters, and filter key visual information based on the direction of change in the state association parameters. By utilizing key visual information, we can establish correspondence analysis relationships between different types of multimodal skin images, calculate the consistency of lesion change characteristics corresponding to different image sources, and output continuous lesion status assessment information. The judgment criteria in the auxiliary diagnosis process are updated based on the lesion status assessment information, and the disease classification is corrected according to the dynamic change pattern corresponding to the lesion status assessment information to complete lesion screening and auxiliary diagnosis.

2. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 1, characterized in that, Dynamic image feature samples are constructed by using color change information, texture difference information, edge offset information, and quantification information of region diffusion range. The specific steps are as follows: Multimodal skin image data corresponding to the same lesion area under different states are acquired, and the correspondence between image data under different states is determined based on image acquisition location, lesion location area and image acquisition time information. Based on the correspondence, the location of lesion areas under different states is matched, and the lesion state change features are extracted based on color change information, texture difference information, edge offset information, and quantitative information on the extent of regional diffusion. The color change information, texture difference information, edge offset information, and regional diffusion range quantification information corresponding to different states are arranged in the order of state change. The dynamic image feature description units are combined to construct a dynamic image feature sample containing the trajectory of lesion changes.

3. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 1, characterized in that, Based on color change information, texture difference information, edge offset information, and quantitative information on the extent of regional diffusion, the features of lesion status change are extracted. Specifically, this includes: position matching based on the spatial position change relationship of the lesion area in the image, and retaining the image information of the overall lesion area and the related areas around the lesion.

4. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 2, characterized in that, The following steps are taken to label the direction of continuous change, the magnitude of abnormal change, and risk-related indicators, and generate corresponding state-related parameters: Based on dynamic image feature samples, feature changes between different states of the same lesion are extracted, and feature transfer rules are established according to color changes, texture differences, edge offset changes, and regional diffusion changes. The direction of continuous change is determined based on the feature migration law, and the magnitude of abnormal change is determined based on the feature change between adjacent states. The continuous change direction, abnormal change magnitude, color abnormal change index, texture abnormal change index, edge change index, regional diffusion change index, and multimodal change consistency index are correlated and recorded to generate state correlation parameters.

5. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 4, characterized in that, The abnormal change amplitude is determined based on the characteristic change amount between different states. Based on the degree of change of the corresponding characteristic change amount during the continuous state change process, the abnormal change amplitude of color change amplitude, texture change amplitude, edge offset amplitude, and region diffusion amplitude are described.

6. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 4, characterized in that, The key visual information is filtered based on the direction of change in the state association parameters. The specific steps are as follows: Obtain the color change correlation parameters, texture change correlation parameters, edge change correlation parameters, region diffusion correlation parameters, and risk change correlation parameters from the state correlation parameters, and determine the feature focus area in the lesion image recognition process based on the corresponding change direction, change magnitude, and risk correlation information; Based on the defined feature focus area, obtain the corresponding color change information, texture difference information, edge offset information, and region diffusion range quantification information, and filter key visual information according to the continuous change direction in the state association parameters.

7. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 6, characterized in that, The correlation analysis is performed based on the changing trends of the key visual information and the state-related parameters obtained by screening. The corresponding visual information in different image sources is screened based on the multimodal change consistency index, and the visual information that maintains the same direction of change in multiple image sources is identified as key visual information.

8. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 6, characterized in that, The specific steps for outputting continuous lesion status assessment information are as follows: Based on key visual information, the correspondence between image sources, lesion regions, and status changes is determined according to the location and status change information of lesion areas in different image sources and the trend of key visual information changes. Based on the correspondence of image sources, lesion regions, and state changes, color change information, texture difference information, edge offset information, and quantitative information of regional diffusion range are correlated and described. The consistency of lesion change features is determined based on the consistency of change direction, the similarity of change amplitude, and the continuity of change process. Based on the consistency of lesion change characteristics and the continuous change direction, abnormal change magnitude and risk association indicators in the status association parameters, the data are associated and recorded to generate continuous lesion status assessment information, which includes current lesion status description information, status change direction information, change continuity information and risk change trend information.

9. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 8, characterized in that, Based on the correspondence between image sources, lesion regions, and state changes, key visual information from different image sources is arranged according to image source, state time, and direction of change to establish a sequence describing the changes of the same lesion in different image sources.

10. The lesion screening and auxiliary diagnosis method based on multimodal skin image deep learning according to claim 8, characterized in that, The disease classification is corrected based on the dynamic changes in the lesion status assessment information to complete lesion screening and auxiliary diagnosis. The specific steps are as follows: Acquire continuous lesion status assessment information, and incorporate current lesion status description information, status change direction information, change continuity information, and risk change trend information into the disease classification judgment process to update the judgment basis in the auxiliary diagnosis process; Based on the updated judgment criteria, the disease classification direction is adjusted according to the information on the direction of state change and the information on the trend of risk change. The change trends corresponding to the information on color change, texture difference, edge offset and the quantitative information on the extent of regional diffusion are described in association. Based on the revised disease classification, the system outputs lesion category information, risk change trend information, state change direction information, and auxiliary diagnostic reference information. It also updates lesion state assessment information based on new multimodal skin imaging data to complete lesion screening and auxiliary diagnosis.

Images