Intelligent scalp health assessment system based on multi-modal imaging and data fusion

Abstract

The invention discloses an intelligent scalp health assessment system based on multi-modal imaging and data fusion, and relates to the technical field of hair quality detection and analysis, comprising multi-source heterogeneous scalp data acquisition and fusion decision, abnormal image and sensor data correction through high-order PDE interpolation and multi-scale filtering, and multi-modal imaging and data fusion. CNN / LSTM and adaptive attention are utilized to extract multi-modal features such as hair follicle morphology, inflammation hot spots and grease fluctuation, multi-head attention and a knowledge graph are fused to generate a weighted decision result, and robust processing on a missing mode is enhanced by means of interpretable visualization and a dynamic completion mechanism; after data acquisition is completed, abnormal readings can be automatically detected, and local interpolation is performed by using a depth prediction model, so that the accuracy and stability of multi-modal decision making are improved; meanwhile, personalized model fine tuning is introduced, longitudinal incremental training and risk prediction are carried out on multiple detection records, and a fine management scheme is provided for long-term monitoring and nursing intervention of scalp health.

Description

Technical Field

[0001] The present invention relates to the technical field of hair quality detection and analysis, and specifically to an intelligent scalp health assessment system based on multimodal imaging and data fusion. Background Art

[0002] In today's scalp health management landscape, with the growing demand for personal care and medical aesthetics, technologies related to scalp care and symptom diagnosis are gaining increasing attention. In particular, many professional institutions are beginning to utilize multimodal data acquisition equipment, performing visible light microscopy, infrared thermal imaging, and multiple sensor measurements, such as humidity and oil content, on the scalp to more comprehensively assess hair follicle status, inflammation distribution, and sebum secretion levels. This typically occurs in beauty salons, professional scalp care centers, or in self-service home monitoring settings. In these settings, operators use visible light microscopes to observe hair follicle morphology and infrared imaging to measure scalp surface temperature distribution to detect early signs of inflammation. Simultaneously, humidity meters and oil content sensors collect information about the environment and the water-oil balance on the scalp surface. Combined with the user's lifestyle and grooming history, these sensors can help determine whether dry scalp conditions, excessive oil secretion, or clogged hair follicles are present. Due to the complex scalp environment and its susceptibility to external interference, factors such as temperature fluctuations, varying equipment performance, and individual physical differences can lead to missing or abnormally high data values, thus compromising the reliability of the final assessment conclusions. In this regard, the market calls for a technical solution that can not only cope with multimodal data noise but also combine medical prior knowledge to make accurate decisions to meet the needs of personalized care and long-term preventive management.

[0003] A search revealed a Chinese invention patent application with publication number CN119048726A, which discloses a hair feature analysis method and system. The method includes the following steps: first, capturing high-magnification and low-magnification images of a scalp region, followed by preprocessing these images. Next, the preprocessed high-magnification and low-magnification images are fed into an artificial intelligence model to simultaneously detect hair follicles in the scalp region and calculate hair width. The artificial intelligence model then calculates at least one hair feature based on the high-magnification and low-magnification images analyzed. The artificial intelligence model can employ an R-CNN model or a variant of the R-CNN model. The present invention advantageously provides more accurate and consistent results for detecting hair follicles and calculating hair width.

[0004] However, combined with the above existing technologies and actual application scenarios: In the aforementioned multimodal scalp monitoring and assessment, analyzing only visible light / infrared images and sensor outputs, such as those for humidity and oil, often fails to accurately assess the multifaceted health of the scalp, leading to assessment bias and overlooking potential symptoms. For example, if infrared thermal imaging data shows a transient abnormal increase, but the image does not exhibit significant signs of inflammation, relying solely on the infrared signal could easily misinterpret normal blood circulation as inflammation of the hair follicles. Alternatively, if a low hygrometer reading is simply a temporary fluctuation due to dry conditions, without image-based comparison, it would be difficult to determine whether a dry scalp is actually present. This not only results in inaccurate assessment results but can also lead to increased time and cost in developing care strategies, and even delay the early detection of potential conditions. In particular, in professional care settings, basing care plans on a single data source can lead to treatment bias or the use of inappropriate products, further contributing to serious consequences such as scalp irritation or seborrheic alopecia. It can be seen that although the acquisition of multi-source heterogeneous data provides more comprehensive scalp information, it also brings high complexity to data fusion and decision-making analysis. It requires an intelligent technology that can not only accurately correct noise or missing data, but also use prior knowledge to enhance the rationality of decision-making.

[0005] To this end, the present invention provides an intelligent scalp health assessment system based on multimodal imaging and data fusion. Summary of the Invention

[0006] (1) Technical problems solved In response to the shortcomings of the existing technology, the present invention provides an intelligent scalp health assessment system based on multimodal imaging and data fusion. It corrects abnormal images and sensor data through high-order PDE interpolation and multi-scale filtering, uses CNN / LSTM and adaptive attention to extract multimodal features such as hair follicle morphology, inflammatory hotspots and oil fluctuations, generates weighted decision results through multi-head attention and knowledge graph fusion, and enhances robust processing of missing modalities with the help of interpretable visualization and dynamic completion mechanism. After data collection is completed, abnormal readings can be automatically detected and local interpolation can be performed using a deep prediction model, thereby improving the accuracy and stability of multimodal decision-making. At the same time, personalized model fine-tuning is introduced to perform longitudinal incremental training and risk prediction on multiple detection records, providing a refined management plan for long-term monitoring and care intervention of scalp health; thereby solving the technical problems recorded in the background technology.

[0007] (2) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: a scalp health intelligent assessment system based on multimodal imaging and data fusion, including: after detecting the completion of multimodal scalp acquisition, extracting microscopic and infrared images and sensor sequences, invoking high-order PDE interpolation, multiscale filtering and contrast enhancement, and outputting high-quality data with anomaly correction and metadata; After receiving the corrected image and sensor time series data, the deep network structure is read and combined with adaptive attention to generate multimodal features; After receiving multimodal features, the fusion network reads the multi-head attention and knowledge graph, uses the collaborative gain metric to measure feature conflicts and calls graph matching, and outputs the fusion representation and decision results; Load Grad-CAM and saliency maps, call the missing prediction model to perform missing completion and dynamic weight update, and generate image thermal visualization and time series deviation description; Identify users' multiple test records and classify them into long-term archive vectors, read user fine-tuning parameters and embed them into knowledge graph constraint functions, use incremental training and lifestyle mapping, and output longitudinal predictions and intervention plans.

[0008] Furthermore, visible light microscopic image data, infrared thermal imaging data and sensor data are collected; a metadata tag set is established for each set of collected data, the tags are bound to the data body for storage, an integrated data management module is constructed, and a unique identifier is assigned to each piece of collected data.

[0009] Furthermore, moments of sudden changes or extreme readings in sensor data are marked, and anomaly masks are created for significant noise pixels in visible light microscopy data and infrared thermal imaging data. After preprocessing, the obtained corrected visible light microscopic image, corrected infrared thermal image and corrected sensor data are written back to the data management module.

[0010] Furthermore, the corrected visible light microscopic image and the corrected infrared thermal image are sent to an image processing pipeline consisting of multi-layer convolution operators and nonlinear activations; a multi-scale kernel combination is introduced in each convolution layer and fused in the output feature map, and the feature map obtained by the last convolution layer is globally pooled to obtain the image modal feature vector.

[0011] Furthermore, multi-channel sensor data are extracted from the corrected sensor sequence and input into the time series processing pipeline to capture the dynamic pattern of each channel in the time dimension; An improved gated memory update function is adopted in combination with an exponential decay mechanism to enhance the parallel modeling of long-term trends and short-term fluctuations. After defining the update rules and iterating, the hidden state at the last moment is taken as the temporal modal feature vector.

[0012] Furthermore, the image modal feature vector and the time series modal feature vector are spliced ​​or cascaded to obtain a preliminary combined vector. In the multi-head mode, different heads are introduced to calculate the attention outputs separately and the results are spliced ​​or weightedly merged to output a preliminary fusion vector.

[0013] Furthermore, based on the feature information of each modality, potentially relevant nodes and corresponding edges are retrieved from the pre-constructed scalp health knowledge graph. The potential symptom features learned from the network are matched with the medical knowledge in the graph at the keyword or vector level. The symptom node set with the highest correlation with the preliminary fusion vector is found in the scalp health knowledge graph. The matched nodes and their edges are converted into graph enhancement vectors, and then concatenated or weighted fused with the preliminary fusion vectors to obtain a fusion representation.

[0014] Furthermore, the fused representation is input into the final decision module composed of a multi-layer perceptron or an attention-based classification head. The decision module verifies the rationality of the inference result based on the graph prior information built into the graph enhancement vector. A synergistic gain metric is introduced for conflict judgment and decision correction. If the synergistic gain metric is not greater than the corresponding threshold, the conflict handling logic is triggered, and the conflict handling or secondary verification logic is started; if the opposite is true, the decision result output by the decision module is directly regarded as the scalp health assessment conclusion.

[0015] Furthermore, based on the decision results and the fusion representation, positioning and highlighting are performed on the image channels. After the key areas of the network's attention are projected onto the original image, the obtained attention distribution map is overlaid and rendered with the corrected visible light microscopic image or the corrected infrared thermal image. In the output of the time series sensor, through key point labeling and interval differentiation, the peak or valley values ​​that may lead to a diagnostic bias towards a certain conclusion will be significantly marked. Combined with the threshold information in the scalp health knowledge map, the high-risk interval for oiliness will be prompted at the UI level.

[0016] Furthermore, the attention parameters are dynamically updated and written into the system configuration library. If a path is missing or abnormal during subsequent use, the historical feature sequence of the time series modality feature vector and the channel information in the image modality feature vector are called to supplement the missing data through the missing prediction model; If the prediction reliability does not reach the threshold, the user will be prompted to re-collect or manually verify; The adjustment parameters, completion behavior and user feedback information are written into the corresponding database. If the user confirms that the completion result is accurate, the positive feedback of the missing prediction model parameters is strengthened.

[0017] Furthermore, a heat map is superimposed on the image area to mark key areas such as scalp inflammation and oil accumulation; key peaks and intervals are highlighted in the time series curve; and the current attention parameters and the prediction accuracy of the completion model are displayed in pop-up windows or charts; Continuous iterative evolution through global retraining or periodic iteration of missing prediction models and attention parameters; If a model or modality performs poorly for a long time in actual scenarios, prompts will be automatically pushed to R&D personnel or maintenance teams, and external prompts will be issued.

[0018] Furthermore, the system collects the corrected visible light microscopic images, corrected infrared thermal images, and corrected sensor data obtained by users at different time periods, and archives them together with the decision results, interpretable outputs, and user feedback records to establish a long-term archive vector. The long-term archive vector also includes the user's living habits and dietary preferences. Based on the data management module and scalp health knowledge graph, hierarchical indexing is performed on the image modality feature vector, time series modality feature vector and fusion representation of each detection in the long-term archive vector.

[0019] Furthermore, when a user records multiple tests within a certain period, or when symptom indicators fluctuate significantly within a short period of time, the personalized training process is triggered, and after fine-tuning is completed, a personalized model is obtained; When detecting the input corresponding to the user, the personalized model is prioritized for inference. If there is a large-scale missing modality or outliers, the global basic model is automatically returned.

[0020] Furthermore, based on long-term profile vectors and personalized models, scalp health trends are periodically evaluated: If the abnormal hair quality indicators show signs of increasing in the future evaluation cycle, the corresponding risk factors will be matched according to the cause-symptom chain in the scalp health knowledge map, and targeted intervention or care suggestions will be made to the user; when the evaluation cycle is short and the risk level of the predicted result is high, an immediate reminder will be triggered.

[0021] (3) Beneficial effects The present invention provides an intelligent scalp health assessment system based on multimodal imaging and data fusion, which has the following beneficial effects: Establishing a unified acquisition protocol and data management framework, and using high-order PDE interpolation and multi-scale wavelets to adaptively correct abnormal images and sensor data can ensure data integrity and consistency, reduce noise interference, and improve the availability of subsequent models.

[0022] Construct image modality feature vectors for image and time series features respectively and the modal eigenvector By taking into account both fine-grained and global features through the learnable attention mechanism, the model can use multi-head attention and scalp health knowledge graph to deal with scalp inflammation and oil abnormalities. Deep fusion to generate fused representations , and uses collaborative gain measurement to detect multimodal conflicts or omissions, which not only ensures the accuracy of decision-making but also can quickly identify potential anomalies. The knowledge graph prior can constrain the personalized model through functions to maintain a balance between medical common sense and individual differences.

[0023] At the interpretability level, tools such as Grad-CAM are used to analyze the corrected visible light microscopy images. and corrected infrared thermal image Highlighting the areas of interest and displaying sensor reading deviations allows users and professionals to understand the basis for system decisions; At the same time, through the missing prediction model Dealing with missing modalities and implementing dynamic weighting Adaptive optimization and building a feedback loop ensure that corrections are made at any time based on actual usage and user feedback.

[0024] In terms of crowd personalization and long-term tracking, long-term archive vectors are used With personalized models Incremental fine-tuning, combined with external information such as diet and lifestyle habits for longitudinal management, once inflammation or oil fluctuations in the long-term trend are identified, the scalp health knowledge map can be used to Provide preventive intervention and nursing recommendations based on the cause-symptom link within the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the structure of the scalp health intelligent assessment system based on multimodal imaging and data fusion of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] See also Figure 1 The present invention provides a scalp health intelligent assessment system based on multimodal imaging and data fusion, comprising: Step 1: After completing the scalp multimodal data collection, the data management framework extracts the micrograph With infrared image and sensor sequence , and call high-order PDE interpolation, multi-scale filtering and contrast enhancement to output a corrected visible light microscopy image with anomaly correction and associated meta-information labels , Corrected infrared thermal image and corrected sensor data ; The step 1 includes the following: Step 101: Unified and synchronized multimodal data collection and metadata annotation: Acquisition of visible light microscopy image data , infrared thermal imaging data And sensor data such as humidity, grease, etc. , where visible light microscopy image data and infrared thermal imaging data Use uniform resolution and frame rate, and compare with sensor data in time dimension Maintain synchronous trigger acquisition; use a unified acquisition clock Benchmark correction is performed on all modalities to ensure that visible light microscopy image data are acquired at the same time stamp , infrared thermal imaging data And sensor data such as humidity, grease, etc. Can represent the same scalp condition; Establish a set of metadata tags for each set of collected data, bind the tags to the data body for storage, build an integrated data management module, and assign a unique identifier to each piece of collected data to facilitate subsequent backtracking or indexing in the feature extraction and deep fusion stages; When using, by using a unified acquisition protocol and clock , ensuring that multimodal data can correspond one-to-one in the time dimension; this is crucial for extracting cross-modal correlation features in the subsequent steps. With the help of the data management module DMG and a complete set of meta-information tags, subsequent steps (such as multimodal feature extraction in step 2) can quickly locate and match visible light microscopy image data at the same time , infrared thermal imaging data And sensor data such as humidity, grease, etc. , laying the data management foundation for deep integration.

[0028] Step 102: Adaptive cleaning and high-order interpolation completion: Extract visible light microscopy image data collected at the same time from the data management module using the metadata tag ID , infrared thermal imaging data and sensor data ; Sensor data The moments when there are mutations or extreme readings are marked, and the visible light microscopy image data are , infrared thermal imaging data Build an anomaly mask for significant noise pixels (such as too dark or too bright blocks) that appear in the , exception mask Used to determine the range required for subsequent interpolation or correction, ensuring that operations are performed only on suspicious or missing areas without accidentally damaging the overall information; For visible light microscopy images The local missing or extreme pixels in are corrected using the image completion model based on partial differential equations (PDE):

[0029] in: Indicates the evolution time The intermediate corrected image under Define the domain for the image; is the failure or noise area corresponding to the abnormal mask; is the correction rate coefficient (the range is , the specific value can be adjusted according to the degree of image noise); and Represent the gradient and divergence operators respectively, which are used to smooth and gradually repair abnormal blocks in space; Update evolution time through iteration The intermediate corrected image ,exist Corrected visible light microscopic images were obtained , and complete the effective completion of local missing or abnormal pixels; For sensor data and infrared thermal imaging data For time series or local anomalies in the data, multi-scale wavelet transform is used to suppress high-frequency noise and perform missing interpolation, where: Sensor data Perform multi-scale wavelet decomposition in the time dimension to extract high-frequency detail coefficients and low-frequency trend coefficients; perform attenuation processing on the high-frequency detail coefficients corresponding to the abnormal segment to eliminate the resonance noise; use the interpolation basis function to locally reconstruct the low-frequency coefficients of the missing segment, and perform inverse transformation on them and the attenuated high-frequency coefficients to obtain the completed sensor data. Similarly, infrared images It is also possible to locate the abnormal hot spot area by performing wavelet decomposition on the rows and columns respectively, and perform multi-scale coefficient correction and reconstruction on the area to obtain the corrected ; Finally, the corrected visible light microscopy image , Corrected infrared thermal image and corrected sensor data Write it back to the data management module and keep its ID unchanged, only updating the status tag for use in subsequent steps; When used, through the exception mask According to the definition of , high-order PDE interpolation and wavelet transform correction are only performed on areas with abnormalities or missing information, which reduces the adverse effects of over-repair on the original valid information; the combination of partial differential equations (PDE) and multi-scale wavelets can perform high-precision completion on the acquired images and time series signals respectively, and has strong adaptive ability. On the one hand, PDE smoothly repairs the defective area in the spatial dimension, and on the other hand, wavelets can suppress noise and reconstruct key waveforms on the time scale or frequency scale.

[0030] Step 2: After receiving the corrected data, the feature extraction module uses deep structures such as CNN and LSTM and injects adaptive attention , generate image modal feature vectors with both spatial texture and temporal dynamics and the time series modal eigenvector ; The second step includes the following: Step 201: Deep feature extraction of image modality: The corrected visible light microscopy image output from step 1 and corrected infrared thermal image Uniformly fed into an image processing pipeline consisting of multi-layer convolution operators and nonlinear activations ; Different from the traditional model with fixed convolution kernel size, a multi-scale kernel combination is introduced in each convolution layer. , and fused in the output feature map to adapt to the spatial distribution of scalp features of various sizes (such as hair follicles, inflammatory spots, etc.); Define image feature map For the Layer output:

[0031] in: Represents an input image from visible light or infrared; Represents the convolution operation; Represents channel-by-channel Hadamard product; is the learnable attention gating coefficient (value range ), used to dynamically allocate weights in the multi-scale convolution kernel output; It is a nonlinear activation function (high-order activation forms such as Swish or GELU can be used);

[0032] Through learnable attention gating coefficients ,The network can adaptively amplify or suppress convolution channels of different scales,,so as to better capture multiple morphologies such as minor inflammation and large,area scalp damage; Perform global pooling on the feature map obtained by the last layer of convolution to obtain the image feature vector ,in is the image feature dimension, and the final image modality feature vector is recorded as ,in and The polymerization results corresponding to the visible light / microscopic channel and infrared channel respectively; When used, by introducing multi-scale kernel combination and learnable attention gating, the network can extract more refined features for scalp symptoms of different shapes and sizes, improving the perception of hair follicle blockage and inflammation distribution; Compared with the traditional single-kernel convolution method, it can significantly enhance the ability to detect local lesions and global abnormalities at the same time, and can carry more expressive image representation in the subsequent fusion network; multi-scale kernel combination and learnable gating coefficient The joint design enables the network to dynamically adjust its dependence on convolutional channels of different scales, breaking through the bottleneck of the previous fixed scale that was difficult to take into account both the microscopic hair follicle structure and the macroscopic scalp texture.

[0033] Step 202: Deep feature extraction of time series modality: The corrected sensor sequence output from step 1 Extract humidity data and oil data Different sensor channels are uniformly represented as multi-channel sensor data ,in is the number of sensor channels, is the sequence length.

[0034] Multi-channel sensor data Input to the sequential processing pipeline , used to capture the dynamic patterns of each channel in the time dimension; An improved gated memory update function is used in combination with an exponential decay mechanism to enhance the parallel modeling of long-term trends and short-term fluctuations; and are the hidden state and memory unit state of the previous moment respectively, and the current input is , define the update rules as follows:

[0035] ,

[0036]

[0037]

[0038] in: is an optional high-order activation function (such as SiLU or GELU); For a natural exponential function, an adaptive memory decay or memory enhancement mechanism for the state at the previous moment is introduced; , , , etc. are learnable parameter matrices (the value range can be converged through training iterations after random initialization); To update the gate vector, is the index memory gate vector, through The element-by-element multiplication of allows the network to flexibly retain or decay historical information; go through After the iteration of , take the last moment As the time series modal eigenvector ,in is the time series feature dimension; if there are multiple sensor subsequences (such as humidity, grease, temperature), the final state vectors of each subsequence can be spliced ​​into a time series comprehensive vector ; Specifically, the image modality feature vector and the time series modal eigenvector They are the integrated outputs of images and sensor sequences, including in-depth characterization of core elements such as scalp inflammation, hair follicle blockage, and oil fluctuations.

[0039] When used, compared to the traditional LSTM or GRU structure, through the exponential memory gate It realizes weighted adjustment of historical states, which can not only prevent the disappearance of long-term dependence, but also quickly complete state switching when sudden data fluctuations occur; the multi-channel decomposition strategy is combined with adaptive gated memory, so that the network can capture the key fluctuations of a single channel when facing multiple types of sensors such as humidity and oil, while taking into account the coupling relationship between channels, which can provide rich timing clues for scalp health assessment. By introducing exponential memory gates in the loop structure, the network can adaptively amplify or attenuate the hidden state and memory units of the previous moment, avoiding the stereotyped processing of long-term and short-term information by traditional timing networks, and realizing more flexible timing learning capabilities.

[0040] Step 3: Obtain image modal feature vector and the time series modal eigenvector Finally, the fusion network uses multi-head attention and graph embedding , and using the synergy gain metric Determine multimodal collaborative consistency and output a fusion representation of integrated image and temporal features and decision results ; The step three includes the following: Step 301: Multi-head attention alignment and preliminary fusion: The image modal feature vector output in step 2 and the time series modal eigenvector Perform splicing or cascade operations to obtain a preliminary combined vector ,in:

[0041] Map to query first ,key Sum Vector space for multi-head attention fusion:

[0042] , and are respectively learnable projection matrices, and their possible ranges can be set according to the network structure (e.g. ); , , All in Abstract features that represent multimodality in space; Under single-head attention, the attention output can be expressed as:

[0043] Softmax is a commonly used normalized activation function that maps any real vector to a vector whose sum of all elements is 1 and each element is in The probability distribution between is a learnable amplification factor (which can be used to amplify or smooth the attention score), with a value range of ; In multi-head mode, introduce different heads , calculate the attention output respectively and concatenate or weight the results;

[0044] in, represents vector concatenation, is the learnable head weight, and the final output is That is the initial fusion vector; When used, by initially splicing the image and temporal features and applying multi-head attention, adaptive alignment between multimodal features is achieved. When certain modalities contribute more in the current context, the attention score will automatically increase, and vice versa. Compared with simple feature splicing or weighted averaging, multi-head attention can capture more fine-grained correlations between channels, and at the same time, combined with a learnable amplification coefficient With head weight , the optimal fusion mode can be dynamically determined during the training process; by introducing multi-head attention after cross-modal splicing, it can not only handle dimensional asymmetry It can also automatically adjust the attention to image features and temporal features in the same network structure.

[0045] Step 302: Knowledge graph-assisted decision mapping: Build a scalp health knowledge graph in advance , where nodes include various scalp symptoms and potential causes (such as abnormal oil content, blocked hair follicles, scalp allergies, etc.), and edges represent the association, strength, or temporal sequence between symptoms and causes; Based on the feature information of each modality obtained in the previous step 2 (such as high temperature hot spots or low humidity, such as visible light / microscopic image modality, infrared thermal imaging modality, environmental / scalp sensor modality and user feedback / questionnaire modality, etc.), the scalp health knowledge map can be used to generate the scalp health knowledge map. Retrieve the potential related nodes and corresponding edge sets from ,The retrieval process is mainly based on graph indexing and concept mapping ,algorithms, which match the potential symptom features learned in the ,network with the medical knowledge in the graph at the keyword or vector level; For the initial fusion vector , in the scalp health knowledge map Find the most relevant symptom node set in , and through the following mapping function accomplish:

[0046] in: Representation node Vectorized representation in the knowledge graph (trained by the graph embedding model, the value range can be dynamically adjusted according to the scale of the graph): Calculate the mapping correlation function (not simple inner product or cosine similarity), which can be implemented based on high-order projection or curve fitting to measure the initial fusion vector With node vector The matching degree between them:

[0047] in: is the multi-head attention fusion vector, Embedding for knowledge graph nodes; , , is a learnable weight matrix, acting on 、 and their element-wise products ; is the bias vector; Hyperbolic tangent activation is used to introduce nonlinearity; is the projection vector, mapping the hidden layer output to a scalar; Ensure final relevance is positive and differentiable.

[0048] The nodes that will be matched and its edges Converted into graph enhancement vector , and with the initial fusion vector Perform splicing or weighted fusion to obtain fusion representation :

[0049] in, It can be a vector concatenation or high-order combination operation to fully inject graph prior knowledge into the model representation.

[0050] When used, by introducing knowledge graph mapping after multi-head attention fusion, medical prior knowledge is superimposed on data-driven features, which helps to make reasonable inferences in the case of symptom conflict or modality loss; fusion representation After fusing the initial fusion vector Feature information and graph enhancement vector After the knowledge is associated, the interpretability and stability of the next decision can be significantly improved; the high-order mapping correlation To match graph nodes, it can avoid the limitations of simple inner product or cosine similarity, and can continuously iteratively optimize the adaptability of graph embedding and feature fusion during model training.

[0051] Step 303: Comprehensive reasoning and conflict resolution: Fusion Representation Input to the final decision module , decision module It is composed of a multi-layer perceptron or an attention-based classification head, used to perform qualitative or quantitative assessment of scalp health (such as determining whether there is excessive oil, dry scalp, localized folliculitis, etc.); At the same time, the decision module Vectors can be enhanced based on graphs The built-in graph prior information is used to verify the rationality of the inference results. For example, when the infrared thermal image is obviously abnormal and the humidity is too low, the possibility can be evaluated through the inflammation model or dryness symptom node in the graph, and then the conflicting results can be balanced or iteratively corrected.

[0052] For conflict judgment and decision correction, introduce collaborative gain measurement , which can be understood as the consistency index of fusion information (the larger the value, the more supportive each modality and the knowledge graph are, and the smaller the value, the more likely there is a conflict). Assuming that the current decision result is , can be defined as follows:

[0053] in: Multi-scale or multi-stage mapping function is used to project vector input to different resolutions or different feature subspaces. It can be wavelet transform, morphological multi-scale analysis or other operators that can reflect local-global differences. In terms of It will generate multiple different feature representations as 0 changes; is the exponential decay factor, ; is a continuous parameter in the multi-scale mapping process, ; is the reference symptom vector that can be obtained from the atlas, corresponding to the known pathological pattern; is the selected high-order norm form ( , to avoid being similar to the conventional L1 or L2 norm); like , indicating that the current fusion result deviates significantly from the atlas reference, and the conflict handling logic needs to be triggered, such as re-adjusting the initial fusion vector Or prompt for missing modal completion; can initiate conflict handling or secondary verification logic, such as prompting the need for re-collection of data or manual review; like , then the fusion representation Good consistency with the knowledge graph, decision module Output decision results This can be directly regarded as the conclusion of the scalp health assessment. If the multimodal conflict remains unresolved, the result will be marked as pending review with accompanying information (e.g., if the infrared thermal image and the humidity sensor conflict, it is recommended to re-collect or further manually verify), and the user or professional will be prompted through a visual interface. When used, combined with fusion representation and decision-making module , taking into account medical prior knowledge during automated evaluation, and timely reminding or correcting when multimodal conflicts occur, collaborative gain measurement It provides a quantitative way to detect conflicts and consistencies, allowing the fusion network to not only output results but also identify when additional data collection or manual confirmation is needed, significantly improving the security and reliability of the system. On the one hand, the collaboration between the multi-head attention mechanism and the knowledge graph improves the accuracy and transparency of reasoning when faced with complex scalp symptoms; on the other hand, the collaborative gain measurement and conflict handling ensure that a high level of robustness can be maintained even in the presence of missing modalities or contradictory information. By combining deep fusion with medical prior knowledge, the limitations of single modality or simple voting fusion are significantly expanded, laying the foundation for the core decision-making capabilities for building an intelligent and trustworthy scalp health assessment system.

[0054] Step 4: When the fusion network outputs the decision result and fusion representation The interpretable module calls Grad-CAM and time series visualization, and uses the missing prediction model Dynamically complete missing modes, generate visual image thermal and index deviation reports, and make timely corrections to anomalies; The step 4 includes the following contents: Step 401: Multimodal visualization and interpretability presentation: Based on the decision result obtained in step 3 and fusion representation , in the image channel (such as the corrected visible light microscopy image and corrected infrared thermal image ) for positioning and highlighting. To this end, we can use an interpretable method based on gradient activation (such as Grad-CAM) or a technique based on a saliency map to project the key areas of the network's attention onto the original image:

[0055] in: For the image position and fusion representation the associated activation mapping function; are trainable convolution or attention weights; is a function that maps from gradient to visual heat map (which may include normalization or threshold operations), It can be regarded as a distribution map of attention; The attention distribution map Corrected visible light microscopy images Or corrected infrared thermal image Overlay rendering allows users to intuitively see the specific areas where potential abnormalities exist (such as blocked hair follicles or spread of inflammation); In the output of time-series sensors such as humidity and oil, key point annotation and interval differentiation can significantly mark peaks or valleys that may lead to a diagnostic bias towards a certain conclusion. For example, when the oil sensor's reading is abnormally high during a specific time period, it will be highlighted with a striking color or symbol in the visualization curve; combined with the threshold information in the scalp health knowledge map (for example, exceeding a certain value may indicate excessive oil secretion), the high-risk oil range will be indicated at the UI level, allowing users or professionals to quickly understand why the system has determined an oil-related condition; When in use, by explicitly showing which areas or segments the network has given high-weight attention to at the image and sensor levels, users can intuitively verify the rationality of the system's judgment; this process is combined with the prior information of the knowledge graph (such as certain numerical intervals corresponding to specific symptoms) to provide a dual-level explanation of medicine and data; it enhances the user's trust in the model output and contributes to the positive cycle of subsequent feedback links; the traditional Grad-CAM idea is further extended to multimodal situations, that is, not only for image convolution features, but also for fusion representations. Visualize time series features to cover a wider range of explanations.

[0056] Step 402: Dynamic weight self-adjustment and data completion learning: Dynamically update the head weights used in multi-head attention fusion based on user and expert feedback and the amplification factor , so that the next analysis can better match the actual usage environment; for example, when users repeatedly report that the data of a certain modality is noisy, the initial weight of the corresponding modality in the multi-head attention can be reduced to enhance the influence of other modalities. Specific methods can include incremental learning and adaptive adjustment, feedback correction mechanism, expert guidance fine-tuning, and incremental learning and adaptive adjustment; The updated attention parameters Write into the system configuration library for the next round of multimodal fusion to form an adaptive evolution mechanism based on usage feedback. If a path (such as infrared thermal imaging or grease sensor) is missing or abnormal in subsequent use, the previous time series modal feature vector is called Historical feature sequence and image modality feature vector The relevant channel information in Perform estimation and fill in missing data :

[0057] in: It is a function that can be trained based on a graph neural network or a sequence generation model to infer the missing modality from the existing modality features; If the prediction reliability does not meet the threshold, the user is prompted to re-collect or manually verify. For example, the prediction reliability can be evaluated by the maximum category probability or regression error output by the model. The adjustment parameters, completion behaviors and user feedback information in this stage are written into the corresponding database so that they can be further optimized in subsequent version iterations. If the user confirms that the completion result is accurate, the missing prediction model will be strengthened. Positive feedback of model parameters, on the contrary, punishes and encourages more exploration updates.

[0058] When in use, by combining the explainable output with the actual experience feedback of users / experts, it continuously self-optimizes during use to reduce the accuracy drop caused by fixed weights or missing modalities; the internal weights of the multi-head attention are directly exposed and can be adjusted manually or automatically, forming a human-machine collaborative optimization mechanism, rather than the traditional black-box self-learning. The cross-modal inference idea is introduced. When a certain signal is missing, the image and other sensor information can be dynamically used to complete it with high credibility.

[0059] Step 403: Visual interaction and closed-loop management: The output results of steps 401 and 402 are integrated with the dynamic update strategy to provide an interactive visualization interface. This interface overlays a heat map on the image area to indicate key areas such as scalp inflammation and oil accumulation; highlights key peaks and intervals in the time series curve; and displays in-depth information such as the system's current attention parameters and the prediction accuracy of the completion model in pop-up windows or charts, allowing experts to review and correct the entire process. After users use the system multiple times, the system will accumulate enough feedback data and completion records, and use the missing prediction model to and attention parameters , Perform global retraining or regular iterations for continuous iterative evolution; If a model (such as a multimodal fusion model, a time series processing model, etc.) or modality (such as sensor data, image data, etc.) performs poorly for a long time in actual scenarios, a prompt will be automatically pushed to the R&D personnel or maintenance team, and hardware replacement or algorithm upgrade can be performed to maintain the long-term high reliability of the scalp health assessment platform.

[0060] During use, through visual interaction and closed-loop management, the system not only has interpretability and adaptive completion capabilities during a single use, but can also be continuously upgraded according to actual usage during long-term evolution, providing users and professionals with a transparent and controllable diagnostic method. Through the multimodal visual explanation of step 401, the dynamic weight self-adjustment and data completion of step 402, and the interactive interface and closed-loop management of step 403, while ensuring the transparency and credibility of the diagnostic process, a complete feedback path of self-learning-human-computer collaboration-continuous iteration is constructed; combined with the multimodal fusion and knowledge graph-assisted decision-making mentioned above, a strict and flexible chain is formed from data acquisition to decision output to interpretability and adaptive optimization. Ultimately, users and professionals can not only obtain accurate and understandable scalp health assessments, but can also participate in the continuous improvement of the model in real time, providing higher credibility and practical value for intelligent scalp management.

[0061] Step 5: When the user's multiple test results are accumulated into a long-term file vector , in fine-tuning parameters The knowledge graph constraint function is introduced into the system, and combined with lifestyle habits and care cycle information, a longitudinal individualized risk prediction plan is output to intervene in lipid imbalance or inflammation trends in advance and continuously optimize the individual care path; The step five includes the following contents: Step 501: Establishment of individual historical archives and cross-period data management: Collect the corrected visible light microscopy images obtained by the user at different time periods (corresponding to the multiple acquisition batches in step 1) and corrected infrared thermal image With corrected sensor data , and compare it with the decision result produced in step 4 , interpretable output (such as saliency map, oil anomaly interval) and user feedback records are archived uniformly and a system named Long-term archive vector, long-term archive vector It also includes fields such as user living habits and dietary preferences; Based on data management module and scalp health knowledge graph , for long-term archive vectors The image modality feature vector for each detection in , time series modal eigenvector and fusion representation To do hierarchical indexing: The first layer is the timeline (reflecting the vertical evolution of users); The second layer is the symptom category or focus (linking nodes such as hair follicle blockage and oil overflow in the knowledge graph); The third layer is personalized tags (such as user diet and care habits); Through this hierarchical index, the long-term performance of the target user in a certain time period or under a specific symptom can be quickly retrieved, laying the data management foundation for personalized decision-making.

[0062] When in use, it fully utilizes the longitudinal information of multiple tests to build an individual-level archive, providing reliable data support for subsequent personalized model fine-tuning and preventive intervention; the combination of hierarchical indexing and knowledge graphs can quickly associate symptoms-time periods-user habits, and can quickly compare historical records when the system has conflicting judgments or abnormal warnings; the user's multimodal characteristics and diagnostic results are integrated with external information such as their lifestyle, and through the linkage of hierarchical indexing and graphs, the system can more flexibly retrieve and analyze individual differentiation factors, rather than being limited to a single collection of data.

[0063] Step 502: Personalized model fine-tuning and incremental training: When a user accumulates more than M test records within a certain period, or when symptom indicators fluctuate significantly within a short period of time (such as an abnormal increase in fat readings), the personalized training process is triggered as follows: Vector from the long-term archive Extract the most recent L high-quality samples and combine them with the global basic model Fine-tune the network weights used in steps 2 and 3; The fine-tuning parameters of the current user are recorded as , in the original loss function A knowledge graph consistency term is superimposed on top to prevent overfitting to personal data:

[0064] in: Is the learning rate (can be adjusted automatically, the value range can be Select within);

[0065] is the knowledge constraint coefficient, , which allows the personalized model to maintain consistency with the scalp health knowledge graph reasonable connection; is a deviation penalty function based on the graph prior (if the personalized model inference result is significantly different from the common pathology link, the penalty is increased); After fine-tuning is completed, A personalized model version marked as the current user (e.g. Indicates the Updated times), stored in the model management module of the system; In subsequent analysis, when it is detected that the input is from this user, the personalized model is prioritized. Execute inference and automatically return to the global basic model if there are large-scale missing modes or outliers ; When in use, without changing the overall model structure, through incremental training and knowledge graph consistency constraints, the system can adapt to individual differences without deviating from medical common sense, support segmented populations or special cases (such as users with more sensitive scalps), and make the diagnosis results closer to the individual's actual condition; by combining personalized training with knowledge graphs, The function introduces constraints to enable the model to take into account user differences and medical priors, avoiding extreme fitting or erroneous learning during personalized parameter adjustment.

[0066] Step 503: Preventive intervention and full-cycle monitoring: In the long-term archive vector and personalized models Periodically assess scalp health trends based on: If indicators such as inflammation and lipid abnormalities are predicted in the future evaluation cycle If there are signs of rising, then according to the scalp health knowledge map Match the cause-symptom chain in the algorithm to the corresponding risk factors and provide users with targeted intervention or care suggestions, such as adjusting the frequency of washing and care or using specific products; When the evaluation cycle When the time interval is shorter and the risk level of the predicted result is higher, an immediate reminder is triggered, suggesting that the user undergo retesting or seek professional diagnosis as soon as possible. The risk level can be obtained through comprehensive evaluation by the trained neural network based on factors such as prediction credibility, collected data quality, and environmental changes. Combined with the dynamic feedback mechanism in step 4, a retrospective analysis of the user's abnormal periods is conducted to identify the key causes or lifestyle habits that caused the abnormalities. Based on the persistence or frequency of the abnormalities, multi-level alarm rules (such as low-risk prompt, moderate alarm, and high-risk warning) are set. The alarm information is synchronized to the back-end database to further improve the personalized profile. When used, by building prediction and intervention modules at the individual level, the system can not only make diagnoses after a single test, but also proactively manage the user's scalp health in the medium and long term. The risk grading mechanism allows users and professionals to timely understand the evolution of their own or group scalp conditions, take intervention measures in advance, and prevent small problems from evolving into big risks. The cause-symptom relationship in the system is analyzed, personalized model predictions are connected with medical prior knowledge, and intervention plans are proactively pushed, transforming the system from passive diagnosis to active protection.

[0067] Through the construction of individual historical profiles in step 501, personalized model fine-tuning in step 502, and preventive intervention and full-cycle monitoring in step 503, the solution not only enables single-time scalp health analysis but also provides continuous evolution and precision care across the vertical dimension. This involves collecting and indexing user information across multiple time periods at the data level, performing incremental training and introducing knowledge constraints at the model level, and incorporating graph priors for early intervention at the decision-making level. This allows the system to evolve from collection-analysis-interpretation-feedback to a comprehensive health management framework focused on individualization, long-term care, and prevention. Ultimately, users receive scalp health assessments and intervention plans more tailored to their individual constitutions and habits, enabling the intelligent system to deliver greater practical value and personalized care in the field of scalp care.

[0068] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only for some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0071] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0072] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An intelligent scalp health assessment system based on multimodal imaging and data fusion, characterized by: include, After the multimodal acquisition of the scalp is completed, the microscopic and infrared images and sensor sequences are extracted, and high-order PDE interpolation, multi-scale filtering and contrast enhancement are called to output high-quality data with anomaly correction and metadata attached; After receiving the corrected image and sensor time series data, the deep network structure is read and combined with adaptive attention to generate multimodal features; The fusion network reads the multi-head attention and knowledge graph, uses the collaborative gain metric to measure feature conflicts and calls graph matching, and outputs the fusion representation and decision results; Load Grad-CAM and saliency maps, call the missing prediction model to perform missing completion and dynamic weight update, and generate image thermal visualization and time series deviation description; Identify users' multiple test records and classify them into long-term archive vectors, read user fine-tuning parameters and embed them into knowledge graph constraint functions, use incremental training and lifestyle mapping, and output longitudinal predictions and intervention plans.

2. The scalp health intelligent assessment system according to claim 1, characterized in that: Collect visible light microscopic image data, infrared thermal image data, and sensor data; Establish a set of meta-information tags for each set of collected data, bind the tags to the data body for storage, build an integrated data management module, and assign a unique identifier to each piece of collected data.

3. The scalp health intelligent assessment system according to claim 2, characterized in that: Mark moments of sudden or extreme readings in sensor data, and create anomaly masks for significant noise pixels in visible light microscopy and infrared thermal imaging data. After preprocessing, the acquired corrected visible light microscopic image, corrected infrared thermal image and corrected sensor data are written back to the data management module.

4. The scalp health intelligent assessment system according to claim 3, characterized in that: The corrected visible light microscopic image and the corrected infrared thermal image are fed into an image processing pipeline consisting of multi-layer convolution operators and nonlinear activations; A multi-scale kernel combination is introduced in each convolution layer and fused in the output feature map. The feature map obtained by the last convolution layer is globally pooled to obtain the image modality feature vector.

5. The scalp health intelligent assessment system according to claim 3, characterized in that: Extract multi-channel sensor data from the corrected sensor sequence and input it into the time series processing pipeline to capture the dynamic pattern of each channel in the time dimension; An improved gated memory update function is adopted in combination with an exponential decay mechanism to enhance the parallel modeling of long-term trends and short-term fluctuations. After defining the update rules and iterating, the hidden state at the last moment is taken as the temporal modal feature vector.

6. The scalp health intelligent assessment system according to claim 5, characterized in that: Performing splicing or cascading operations on the image modality feature vector and the time series modality feature vector to obtain a preliminary combined vector; In multi-head mode, after introducing different heads, the attention output is calculated separately and the results are spliced ​​or weightedly merged to output a preliminary fusion vector.

7. The scalp health intelligent assessment system according to claim 6, characterized in that: Based on the feature information of each modality, potential related nodes and corresponding edges are retrieved from the pre-built scalp health knowledge graph. The potential symptom features learned from the network are matched with the medical knowledge in the graph at the keyword or vector level. The symptom node set with the highest correlation with the preliminary fusion vector is found in the scalp health knowledge graph. The matched nodes and their edges are converted into graph enhancement vectors, and then concatenated or weighted fused with the preliminary fusion vectors to obtain a fusion representation.

8. The scalp health intelligent assessment system according to claim 7, characterized in that: The fused representation is input into the final decision module composed of a multi-layer perceptron or an attention-based classification head. The decision module verifies the rationality of the inference result based on the graph prior information built into the graph enhancement vector. A collaborative gain metric is introduced for conflict judgment and decision correction. If the collaborative gain metric is not greater than the corresponding threshold, the conflict handling logic is triggered, and the conflict handling or secondary verification logic is started. If the opposite is true, the decision result output by the decision module is directly regarded as the scalp health assessment conclusion.

9. The scalp health intelligent assessment system according to claim 8, characterized in that: Based on the decision results and fusion representation, positioning and highlighting are performed on the image channel. After projecting the key areas of network attention onto the original image, the obtained attention distribution map is overlaid and rendered with the corrected visible light microscopic image or corrected infrared thermal image. In the output of the time series sensor, through key point labeling and interval differentiation, the peak or valley values ​​that may lead to a diagnostic bias towards a certain conclusion will be significantly marked. Combined with the threshold information in the scalp health knowledge map, the high-risk interval for oiliness will be prompted at the UI level.

10. The scalp health intelligent assessment system according to claim 9, characterized in that: After dynamically updating the attention parameters, they are written into the system configuration library. If a path is missing or abnormal during subsequent use, the historical feature sequence of the time series modality feature vector and the channel information in the image modality feature vector are called to supplement the missing data through the missing prediction model. If the prediction reliability does not meet expectations, the user will be prompted to re-collect or manually verify; The adjustment parameters, completion behavior and user feedback information are written into the corresponding database. If the completion result is confirmed to be accurate, the positive feedback of the missing prediction model parameters is strengthened.

11. The scalp health intelligent assessment system according to claim 10, characterized in that: Overlay heat maps on image areas to highlight key locations; highlight key peaks and intervals in time series curves; display current attention parameters and complete model prediction accuracy in pop-up windows or charts; Continuous iterative evolution through global retraining or periodic iteration of missing prediction models and attention parameters; If a model or modality performs poorly for a long time in actual scenarios, prompts will be automatically pushed to R&D personnel or maintenance teams, and external prompts will be issued.

12. The scalp health intelligent assessment system according to claim 11, characterized in that: Collect the corrected visible light microscopic images, corrected infrared thermal images, and corrected sensor data obtained by users at different time periods, and archive them together with decision results, interpretable outputs, and user feedback records to establish a long-term archive vector. The long-term archive vector also includes user living habits and dietary preferences. Based on the data management module and scalp health knowledge graph, hierarchical indexing is performed on the image modality feature vector, time series modality feature vector and fusion representation of each detection in the long-term archive vector.

13. The scalp health intelligent assessment system according to claim 12, characterized in that: When a user accumulates multiple test records within a certain period, or when symptom indicators fluctuate significantly within a short period of time, the personalized training process is triggered. After fine-tuning is completed, a personalized model is obtained. When the corresponding user input is detected, the personalized model is scheduled to perform reasoning first. If there is a large-scale missing mode or outliers, it will automatically return to the global basic model.

14. The scalp health intelligent assessment system according to claim 13, characterized in that: Periodically evaluate scalp health trends based on long-term profile vectors and personalized models: When the assessment period is short and the risk level of the predicted result is high, an immediate reminder is triggered; If abnormal hair quality indicators show signs of increasing in future evaluation cycles, the corresponding risk factors will be matched according to the cause-symptom chain in the scalp health knowledge map, and targeted intervention or care suggestions will be provided to the user.

Images