A method and system for multi-task analysis of facial images, and a storage medium

Abstract

The present application provides a kind of rash multi-task analysis method, system and storage medium, belong to medical image processing technical field.The method is input into segmentation network and obtains back region mask, according to mask fitting back midline and dividing left half graph and right half graph;After direction alignment to left and right half graph, input parameter shared feature extraction network obtains left and right multi-scale features, after channel projection and spatial size alignment, through bidirectional mirror image cross attention calculation two directions mirror image response, and decoupling is mirror image common representation, absolute difference representation and direction difference representation with left and right direction identifier;Again, two groups of gate aggregation are formed respectively to form description space fusion features and classification global fusion features, input Transform description generation module and multi-label classification module, and through classification-description bidirectional semantic reflux, obtain mutually constrained rash description text and constitution tendency probability, improve the accuracy, consistency and explainability of rash analysis.

Description

Technical Field

[0001] This invention discloses a multi-task analysis method, system, and storage medium for sha (a type of facial paralysis), relating to the field of computer vision image recognition technology. Background Technology

[0002] Sha phenomena refer to the changes in color, shape, texture, and regional distribution of human skin after gua sha intervention. Professionals typically make comprehensive observations based on the color of the sha, the amount of sha, the shape of petechiae or transverse veins, the distribution on the back, and the degree of left-right asymmetry. Manual observation is easily affected by experience, shooting conditions, skin base color, and descriptive habits, making it difficult to form uniform and repeatable quantitative results.

[0003] Existing deep learning methods can extract the back region through segmentation networks, extract image features through convolutional neural networks, and further complete the classification or description generation of the bruises. Considering that the human back has an approximately symmetrical structure, some methods divide the back image into left and right halves, use a shared feature extraction network to extract left and right features, and then fuse them through concatenation, interpolation, or attention weighting.

[0004] For example, patent application number 202511301184.X discloses a method for intelligent analysis of Sha symptoms in Traditional Chinese Medicine based on the ShaMTLNet model. This method organically combines the generation of Sha symptoms descriptions with the identification of TCM constitutions and introduces a dynamic weighting strategy based on task uncertainty to achieve parameter sharing and feature complementarity between tasks. This method improves the overall performance and robustness of the model and can capture Sha symptoms information more accurately.

[0005] However, existing techniques that directly segment the left and right halves of an image based on its geometric center are easily affected by differences in shooting angle, body posture, and back contour, resulting in inaccurate correspondence between the left and right anatomical regions. Simple attention-weighted methods typically use a single weight to select between left and right features, which may mask truly meaningful asymmetry information through weighted averaging. While absolute differences can represent the degree of asymmetry, they cannot indicate the direction of the difference, making it impossible to distinguish whether the feature is primarily located on the left or right side.

[0006] Furthermore, the description generation branch and the constitution classification branch usually only share visual features, and the outputs of the two tasks lack direct constraints, which may lead to inconsistencies between the description content and the constitution tendency.

[0007] Therefore, it is necessary to improve the left-right alignment method, attention fusion structure, and multi-task collaborative training method without changing the overall architecture. Summary of the Invention

[0008] To address the problems in existing technologies, such as inaccurate correspondence of anatomical positions in the left and right halves of the image, easy dilution of asymmetrical features by attention weighting, difficulty in preserving the direction of difference, lack of consistency constraints between the description of the symptoms and the classification of constitution, and unclear decision-making basis of the model, this invention provides a multi-task analysis method, system, and storage medium for symptoms. While maintaining the overall architecture of U-Net segmentation, sharing ResNet101 between the left and right halves of the image, attention fusion, Transformer description generation, and multi-label classification, this invention improves the accuracy, consistency, and interpretability of symptoms analysis.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0010] The first aspect of this invention provides a multi-task analysis method for petechiae (sha) on the human back. The method involves inputting an acquired original image of petechiae on the back into a segmentation network to obtain a back region mask; determining the back midline based on the back region mask to form a left and right half-image; performing orientation alignment on the left and right half-images; inputting the orientation-aligned left and right half-images into a parameter-shared feature extraction network to obtain left-side multi-scale features and right-side multi-scale features; performing channel projection and spatial size alignment on the left-side and right-side multi-scale features to obtain aligned left-side and right-side multi-scale features; and calculating a first mirror response using the aligned left-side multi-scale features as the query and the aligned right-side multi-scale features as the key and value. The second mirror response is calculated using the aligned right-side multi-scale features as the query and the aligned left-side multi-scale features as the key and value. Based on the first and second mirror responses, mirror commonality representations, absolute difference representations, and directional difference representations with left and right direction indicators are generated. The mirror commonality representations, absolute difference representations, and directional difference representations with left and right direction indicators are subjected to a first set of gated aggregation and a second set of gated aggregation to form description space fusion features and classification global fusion features. The description space fusion features are input into the Transformer description generation module, and the classification global fusion features are input into the multi-label classification module. A classification-description bidirectional semantic backflow is performed to obtain the description text of the sha phenomenon and the probability of physical constitution tendency.

[0011] The classification-description bidirectional semantic backflow is performed by inputting the global fusion features of the classification into the multi-label classification module, feeding back the constitution semantic information output by the multi-label classification module to the Transformer description generation module to constrain the generation of descriptive text, and feeding back the Sha phenomenon attribute information output by the Transformer description generation module to the multi-label classification module to correct the classification result, thereby obtaining the Sha phenomenon descriptive text and constitution tendency probability.

[0012] Preferably, this method utilizes U-Net to segment the back region and form left and right halves of the image. The left and right halves are then input into a ResNet101 algorithm with shared parameters, extracting multi-scale features from multiple residual stages. A multi-scale mirror interaction routing network is set at the original attention fusion location. This network includes a scale alignment layer, a bidirectional mirror cross-attention layer, a commonality-difference decoupling layer, and a dual-task conditional routing layer, arranged sequentially. The bidirectional mirror cross-attention layer queries the right-side features using left-side features and the left-side features using right-side features to obtain mirror responses. The commonality-difference decoupling layer separates mirror commonality representations, absolute difference representations, and directional difference representations from the mirror responses. The dual-task conditional routing layer generates two sets of routing gates using the description task query vector and the classification task query vector, forming spatial fusion features for Transformer description generation and global fusion features for multi-label classification, respectively.

[0013] The multi-scale mirror interaction routing network includes a scale alignment layer, a bidirectional mirror cross attention layer, a commonality-difference decoupling layer, and a dual-task conditional routing layer connected sequentially along the feature processing direction. The output of the previous layer serves as the input of the next layer. The specific connections between each layer and the upstream and downstream networks are as follows: The U-Net outputs a back region mask after independent training and fixed parameters. The back region mask is then input into a ResNet101 with shared parameters after midline localization, orientation alignment, and generation of left and right half-images. The left and right same-scale features output from each residual stage of ResNet101 are fed into a scale alignment layer in pairs. The scale alignment layer unifies the features of each scale to the same number of channels and spatial size. Each scale alignment result is fed into a bidirectional mirror cross-attention layer to obtain mirror responses in two directions. The mirror responses in two directions are fed into a commonality-difference decoupling layer to form three types of representation components: mirror commonality, absolute difference, and directional difference at each scale. All scale-representation components are fed into a dual-task conditional routing layer, which aggregates them into a spatial fusion feature that maintains the spatial layout and a spatially pooled global fusion feature for classification. The former is connected to the Transformer description generation module, and the latter is connected to the multi-label classification module. The classification probability forms the physical condition semantics and is fed back to the Transformer description generation module. The bruise attributes extracted by the description branch are classified into logits after gating correction. The above semantic backflow is not connected to the segmentation network, and the segmentation loss is not included in the multi-task joint loss.

[0014] Specifically, the scale alignment layer of the multi-scale mirror interactive routing network is used to convert the left-side multi-scale features and the right-side multi-scale features into the same channel dimension and preset spatial size through channel projection and upsampling or downsampling, respectively, and to add scale position encoding to the left-side multi-scale features and the right-side multi-scale features.

[0015] The bidirectional mirror cross-attention layer of the multi-scale mirror interactive routing network is used to calculate the first mirror response using the aligned left multi-scale features as the query and the aligned right multi-scale features as the key and value, and to calculate the second mirror response using the aligned right multi-scale features as the query and the aligned left multi-scale features as the key and value.

[0016] First mirror response Second mirror response The calculation formula is:

[0017] ;

[0018] ;

[0019] Where s represents the scale; L S and R S These are the left and right feature matrices, respectively, after channel projection and spatial size alignment, and flattened into N spatial locations, where N is the number of spatial locations in the aligned feature map; Q(·), K(·), and V(·) represent the linear projections of the query, key, and value, respectively, with the two attention directions sharing the corresponding projection parameters; d is the dimension of the key vector, T represents the matrix transpose, and softmax normalizes along all key positions corresponding to each query position.

[0020] A commonality-difference decoupling layer for a multi-scale mirror interactive routing network is used to generate mirror commonality representations, absolute difference representations, and directional difference representations with left and right direction indicators based on the first mirror response and the second mirror response; the mirror commonality representation... Absolute difference representation and The formula for calculating the directional difference is as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] in, , and These are the learnable projection parameters corresponding to the commonality representation, absolute difference representation, and directional difference representation, respectively; the symbol ";" indicates concatenation in the channel dimension, ⊙ indicates element-wise multiplication, and |·| indicates taking the absolute value of each element; To correspond to spatial location and be encoded in the left and right directions that can be broadcast along the channel dimension, the left and right sides use encoding with opposite signs to preserve the source direction of the difference.

[0025] The dual-task conditional routing layer in a multi-scale mirror interactive routing network stores the query vector describing the task. and classification task query vector The representation components are the components of the mirror commonality representation, the absolute difference representation, and the directional difference representation with left and right direction indicators at each scale.

[0026] Route scoring first converts spatial representation components into channel description vectors, defined as follows:

[0027]

[0028] Where GAP(·) represents global average pooling over the spatial dimension; the routing weights are normalized over all scale-representation component combinations (s,k) under a fixed task t, and the ergodices in the denominator are s′ and k′, which are calculated using the following formula:

[0029]

[0030] Where t∈{cap,cls} represent the description task and the classification task, respectively; Let be the learnable query vector for task t. and These are used to describe tasks and classify tasks, respectively. Let k represent the k-th type of representation component at scale s, where k is taken from the mirror commonality representation, absolute difference representation, and directional difference representation. Let be the learnable route projection parameters for task t; s′ and k′ are the traversal indices in the normalized summation, and the denominator is summed over all candidate scale-representation components to make the sum of all route weights for the corresponding task equal to 1.

[0031] It needs to be explained that the spatial representation component is the spatial feature extracted by the model; the feature can be transformed from the spatial dimension and broken down into the three types of representation components mentioned above (mirror commonality representation, absolute difference representation, and directional difference representation with left and right direction indicators).

[0032] The dual-task conditional routing layer queries the task description vector. The generated routing weights are weighted and aggregated across the representation components at each scale while maintaining spatial resolution, forming spatial fusion features that describe the route. :

[0033]

[0034] The dual-task conditional routing layer queries the categorized task vector. The generated routing weights are weighted and aggregated across the representation components at each scale, then pooled to form a global fusion feature for classification. :

[0035]

[0036] The first set of gated aggregations corresponds to the description task, and the second set of gated aggregations corresponds to the classification task. The routing projection parameters and task query vectors of the two sets are not shared. `Pool(·)` indicates that global average pooling or global attention pooling is performed on the spatial dimension, used for classification tasks to aggregate the spatial dimension into a global vector; no pooling is performed on the description task to maintain spatial resolution for description generation. Spatial location is preserved for description generation. A global vector is generated for multi-label classification.

[0037] In one specific implementation, multi-task joint training is performed on the Transformer description generation module, the multi-label classification module, and the multi-scale mirror interaction routing network. The total loss of the multi-task joint training is formed by a weighted sum of description generation loss, multi-label classification loss, mirror response consistency loss, commonality-difference orthogonality loss, dual-task routing separation loss, and bidirectional semantic consistency loss.

[0038] The second aspect of the present invention provides a multi-task analysis system for sha (a type of skin disease), including a back segmentation and left / right half image generation module, a feature extraction and alignment module, a bidirectional mirror cross-attention module, a commonality-difference decoupling module, a dual-task conditional routing module, and a result output module; each module is used to execute the corresponding steps of the aforementioned multi-task analysis method for sha.

[0039] Further, the system includes an image generation module for inputting the acquired original back image of the human body into a segmentation network to obtain a back region mask; determining the back midline based on the back region mask and forming a left and right half-image. A feature extraction and alignment module is used to perform directional alignment processing on the left and right half-images, inputting the directional-aligned left and right half-images into a parameter-shared feature extraction network to obtain left-side multi-scale features and right-side multi-scale features; performing channel projection and spatial size alignment on the left and right-side multi-scale features to obtain aligned left-side and aligned right-side multi-scale features. A bidirectional mirror cross-attention module is used to calculate a first mirror response using the aligned left-side multi-scale features as the query and the aligned right-side multi-scale features as the key and value, and to calculate a second mirror response using the aligned right-side multi-scale features as the query and the aligned left-side multi-scale features as the key and value. A commonality-difference decoupling module is used to generate mirror commonality representations, absolute difference representations, and directional difference representations with left and right direction identifiers based on the first and second mirror responses. The dual-task conditional routing module performs a first set of gated aggregation and a second set of gated aggregation on the mirror commonality representation, the absolute difference representation, and the directional difference representation with left and right direction indicators to form a description space fusion feature and a classification global fusion feature. The result output module inputs the description space fusion feature into the Transformer description generation module and the classification global fusion feature into the multi-label classification module, and performs a bidirectional semantic backflow of classification and description to obtain the description text of the sha phenomenon and the probability of physical constitution tendency.

[0040] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the aforementioned multi-task analysis method for sha phenomena.

[0041] Beneficial effects:

[0042] This invention fits the back midline based on the back region mask output by the segmentation network, and divides the left and right halves of the image based on the back midline. The corresponding anatomical regions in the left and right halves no longer depend on the geometric center of the image. Therefore, it can avoid the misalignment of the left and right anatomical regions caused by differences in shooting angle, body posture and back contour. This allows subsequent left and right feature comparisons to be based on the back anatomical structure, improving the spatial consistency of left and right feature comparisons.

[0043] This invention uses a feature extraction network that shares input parameters after directional alignment of the left and right halves of the image. By channel projection and spatial size alignment, the left and right features at each scale have the same channel dimension and spatial size. As a result, the left and right features are in the same semantic space and correspond to each other position by position. This provides an alignable feature basis for mirror comparison of each region and reduces spurious differences caused by semantic offset of left and right features.

[0044] This invention uses features on one side as queries and features on the other side as keys and values ​​to calculate the mirror response bidirectionally. It decouples the mirror commonality representation, absolute difference representation, and directional difference representation with left and right direction indicators from the mirror response. Any region on one side can actively retrieve the corresponding region on the other side, and the direction and magnitude of the asymmetry are explicitly preserved. Therefore, the truly meaningful left and right asymmetry information will not be smoothed or masked by single-weight attention weighting, thereby improving the sensitivity and directional accuracy of detecting left and right asymmetry of the sha image.

[0045] This invention forms spatial fusion features and global classification fusion features through a first set of gated aggregation and a second set of gated aggregation, respectively. The description generation task and the constitution classification task obtain task-specific fusion features while sharing the visual backbone. Therefore, it avoids the negative transfer caused by the two tasks being forced to share the same fusion ratio, and thus ensures the accuracy of the description of the sha phenomenon and the accuracy of the constitution classification at the same time.

[0046] This invention performs bidirectional semantic backflow between the description generation branch and the constitution classification branch. The constitution tendency probability can constrain the generation of the Sha phenomenon description text, and the Sha phenomenon attribute probability can correct the constitution classification result. The outputs of the two tasks corroborate each other, so the Sha phenomenon description text and the constitution tendency probability maintain semantic consistency, thereby avoiding the problem of contradictory outputs of the two tasks and improving the credibility of the Sha phenomenon analysis results. Attached Figure Description

[0047] Figure 1 This is a diagram of the overall model architecture of the present invention.

[0048] Figure 2 The diagram shows the multi-scale left and right feature access and scale alignment structure for sharing ResNet101.

[0049] Figure 3 This is a diagram of bidirectional mirror cross-attention and commonality-difference decoupling structure.

[0050] Figure 4 A dual-task conditional routing structure diagram for describing tasks and classifying tasks.

[0051] Figure 5 This is a diagram illustrating the bidirectional semantic backflow structure for classification.

[0052] Figure 6 The image shows the original image of a typical case of sha (a type of skin ailment) and its asymmetric heatmap. Detailed Implementation

[0053] The invention will now be further described with reference to the accompanying drawings.

[0054] It should be understood that the flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed in order or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts. Some block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0055] The core idea of ​​this invention is to decompose the analysis of sha (a type of facial paralysis) into back region localization, left and right mirror feature modeling, task-specific feature routing, description generation, and multi-label classification, and to coordinate the outputs of the two tasks through bidirectional semantic backflow. Unless otherwise specified, the input size, number of network layers, feature dimensions, threshold, and loss weights can be adjusted according to the data scale and computational resources.

[0056] This invention utilizes a back region mask output by U-Net to fit the back midline, and performs geometric correction and left and right half-image division accordingly. This allows the corresponding left and right regions to be established based on the back anatomical structure rather than the geometric center of the image, thereby improving the spatial consistency of left and right feature comparison.

[0057] This invention employs bidirectional mirror cross attention, enabling features on the left to retrieve corresponding regions on the right, and features on the right to retrieve corresponding regions on the left. By decoupling commonalities and differences, it preserves mirror commonalities, asymmetric amplitudes, and asymmetric directions, thereby reducing the smoothing effect of ordinary splicing or unidirectional attention on local differences.

[0058] This invention utilizes the query vectors of the description task and the query vector of the classification task to form two sets of conditional routing weights with non-shared parameters, enabling the description generation task and the physical classification task to obtain task-specific fusion features while sharing the visual backbone, thereby reducing the negative transfer caused by multiple tasks sharing the same fusion ratio.

[0059] This invention establishes a bidirectional semantic backflow between the classification branch and the description branch, enabling physical constitution tendency to constrain description generation and the Sha phenomenon attribute to correct classification results, thereby improving the consistency between the Sha phenomenon description text and the multi-label physical constitution tendency probability.

[0060] Furthermore, existing technologies also suffer from the problem that the attention region output by the model may not simultaneously support descriptive text and classification results, and the interpretability of the model's decision-making basis remains insufficient.

[0061] This invention can combine mirror cross attention, task routing weights, Transformer cross attention, and classification gradient response to generate interpretable heatmaps, providing common evidence regions for professionals to verify model outputs.

[0062] The following is for reference Figures 1 to 6 Each step is explained in detail.

[0063] The input of this invention is an RGB image of a human back with petechiae (sha). The training samples include at least a binary mask of the back region corresponding to the input image, descriptive text of the symptoms (sha), and a multi-label vector of body constitution. The binary mask of the back region is used to supervise the segmentation network, the descriptive text of the symptoms is used to supervise the Transformer description generation module, and the multi-label vector of body constitution is used to supervise the multi-label classification module. During the inference phase, the outputs are a sequence of descriptive words for the symptoms, the probabilities of nine body constitution types, and an optional differential heatmap.

[0064] I. Image Preprocessing and Left / Right Half Calibration of Sha Images

[0065] Step S1: Back region segmentation and midline fitting

[0066] The original image of the back bruises is input into the U-Net segmentation network to obtain a binary mask of the back region. After obtaining the back region mask, the leftmost and rightmost boundary points are extracted at each preset height position of the mask, and the midpoint between the two is calculated. Then, the back midline is fitted based on the midpoints of multiple height positions, so that the back midline is based on the back anatomical structure rather than the geometric center of the image.

[0067] After fitting the back midline, the back region is rotated or non-rigidly corrected based on the back midline to make the back midline coincide with the longitudinal central axis of the image. Then, the corrected back region is divided into a left half and a right half, so that the corresponding anatomical regions in the left and right half correspond accurately in spatial position.

[0068] After dividing the image into left and right halves, the right half is horizontally flipped so that the anatomical positions on both sides have the same orientation, laying the foundation for subsequent bidirectional mirror cross-attention calculation. It should be noted that U-Net is still used for back region segmentation; the newly added midline fitting and correction steps are driven by the U-Net output mask and do not change the main structure of the segmentation network.

[0069] The U-Net segmentation network is trained separately before the multi-task network. Its training samples are manually labeled binary masks along the back contour, and it uses an independent training loss. We employ a weighted sum of Dice loss and binary cross-entropy loss; after completing the segmentation training, we fix the U-Net parameters and use the back region mask output by it to perform midline fitting, geometric correction, and generation of left and right half images. It is used only for independent training of U-Net and does not participate in the multi-task joint training described in step S9.

[0070] II. Shared Feature Extraction

[0071] Step S2: Shared ResNet101 multi-scale feature extraction, see [link to relevant documentation] Figure 2 .

[0072] After aligning the left and right halves of the image, the left half and the flipped right half are input into the same ResNet101 algorithm. Both input branches share all ResNet101 parameters to ensure the left and right features reside in the same semantic space. Specifically, the final classification layer and global average pooling layer of the ResNet101 algorithm are removed, retaining the convolutional feature maps as spatial features. Global average pooling is then performed on the left and right spatial features separately to obtain the left and right global features. The spatial features are used for Transformer description generation, while the global features are used for multi-label body classification.

[0073] Specifically, ResNet101 is initialized with ImageNet pre-trained parameters and jointly fine-tuned on the training data of this invention. The number of channels in the output feature map of its fifth residual stage is 2048, and the spatial size of the feature map of each residual stage is halved as the stage deepens. Random brightness perturbation, slight rotation and color jitter are used for data augmentation during training, and horizontal flipping is not used to maintain the consistency of semantic direction of the left and right halves of the image.

[0074] III. Multi-scale Mirror Interaction

[0075] Step S3: Multi-scale feature access and scale alignment, see [link to relevant documentation] Figure 2 .

[0076] After obtaining the left and right multi-scale features, left and right features are obtained from the second, third, fourth, and fifth residual stages of the shared ResNet101. The features from each stage are first projected to the same channel dimension through 1×1 convolution, then upsampled or downsampled to form a preset spatial size, and scale position encoding is added. Through this scale alignment process, both shallow color and texture information and deep semantic information can participate in left-right comparisons.

[0077] Let the left and right features of the s-th residual stage be respectively and The two channels are projected to a unified number of channels C using a parameter-shared 1×1 convolution, and then downsampled to a unified spatial size H×W using bilinear interpolation upsampling or stride convolution to obtain L. S R S ∈ Then to L S and RS Add the same scale encoding; reverse the right-side position encoding horizontally before adding R. S This ensures that the left and right mirror positions have consistent indices.

[0078] Step S4: Calculate bidirectional mirror cross-attention, see [link / reference] Figure 3 .

[0079] After scale alignment, a bidirectional mirror cross-attention mechanism with shared parameters is implemented for each scale. In the first direction, the left feature is used as the query and the right feature as the key and value; in the second direction, the right feature is used as the query and the left feature as the key and value. Furthermore, the positional encoding of the right feature is horizontally reversed before participating in the computation. Through bidirectional computation, A(L→R) and A(R→L) are obtained, enabling either side of the region to actively retrieve its mirror counterpart, rather than simply statically concatenating the features from both sides.

[0080] L S and R S The spatial dimension is flattened to have N = H × W positions. The first direction is Q(L) S ) for query, K(R) S ) and value V(R) S The second direction uses Q(R) as the key and value; S ) for query, K(L S ) and value V(L S Let be the key and value. For each query location, divide the dot product similarity with all opposing key locations by √d, then perform softmax to obtain the weight matrix. Finally, use this weight matrix to sum the corresponding value matrices. The two directions share the Q, K, and V projection parameters, ensuring that the two mirror responses reside in the same feature space.

[0081] Let L be the left and right features at the s-th scale. S and R S Q, K, and V are shared projections, and d is the dimension of the key vector. The mirrored responses of left query right and right query left are calculated using the following formulas:

[0082] ;

[0083] ;

[0084] The above response serves as the direct input to the commonality-difference decoupling layer, used to establish a bidirectional dependency between corresponding left and right regions.

[0085] When bidirectional mirror cross-attention is implemented in a multi-head manner, the unified channel dimension C is divided into h subspaces, and the key vector dimension of each attention head is... =C / h, each head calculates its mirror response separately, then concatenates them in the channel dimension, and finally restores the C-dimensional response through output projection. One or more attention layers are set, with residual connections and layer normalization after each layer; the number of attention heads, stacked layers, and channel dimension are determined based on the training data size.

[0086] IV. Commonality-Difference Decoupling and Task Condition Routing

[0087] Step S5: Commonality-Difference Decoupling (see...) Figure 3 )

[0088] After obtaining the mirror responses in both directions, a common representation C is generated from A(L→R) and A(R→L). S Absolute difference representation and directional difference representation with left and right direction coding Among them, commonality representation extracts common patterns by summing and element-wise multiplication of bidirectional responses, absolute difference representation measures asymmetric amplitude, and directional difference representation is achieved through signed difference and directional encoding. Retaining the left and right directions, calculate the three values ​​separately using the following formula:

[0089] ;

[0090] ;

[0091] ;

[0092] in, , and These are the learnable projection parameters corresponding to commonality representation, absolute difference representation, and directional difference representation, respectively. , and Implemented by 1×1 convolutional or position-wise linear layers, projecting the corresponding inputs to a uniform number of output channels; the symbol ";" indicates channel concatenation, ⊙ indicates element-wise multiplication, and |·| indicates element-wise absolute value. Directional encoding. Take broadcastable vectors with opposite signs from the left and right sources respectively, with all +1s from the left source and all -1s from the right source; Indicates absolute difference representation, This represents the directional difference characterization with left and right direction encoding.

[0093] To separate the commonality subspace from the difference subspace, a commonality-difference orthogonal loss is added during training. This makes the commonality representation and the two types of difference representation have low correlation in the feature subspace, thus structurally reducing the occlusion of asymmetric features by the commonality color.

[0094] To avoid the influence of characteristic amplitudes on orthogonality constraints, Frobenius normalization is first applied to each flattened representation matrix:

[0095]

[0096] Where ε is a positive number to prevent the denominator from being zero; after normalization, the commonality-difference orthogonal loss is:

[0097]

[0098] in, Let Frobenius norm be denoted. This loss minimizes the correlation between the commonality representation and the difference representation between the two classes; 0.01–0.1 represents the weights in the joint total loss. The range of possible values ​​is not the range of values ​​for the orthogonal loss itself.

[0099] Step S6: Dual-task conditional routing, see [link / reference] Figure 4 .

[0100] After obtaining the common and differential representations at each scale, the dual-task conditional routing layer utilizes the stored task description query vectors. and classification task query vector Two sets of route weights are generated. Among them, Generate scale weights and component weights for part, color, and shape descriptions, and output descriptive spatial fusion features that preserve spatial resolution; Another set of scale weights and component weights is generated, and after pooling, a global fusion feature for classification is formed. The two sets of routing parameters are not shared, so that the two tasks share the same visual backbone but are not forced to use the same fusion ratio.

[0101] Route scoring first converts spatial representation components into channel description vectors, defined as follows:

[0102]

[0103] Where GAP(·) represents global average pooling over the spatial dimension; the routing weights are normalized over all scale-representation component combinations (s,k) under a fixed task t, and the ergodices in the denominator are s′ and k′, which are calculated using the following formula:

[0104] ;

[0105] ;

[0106] ;

[0107] in, Input Transformer description to generate module, Input a multi-label classification module to achieve task-specific feature allocation.

[0108] The routing softmax is not normalized along the channel dimension or the spatial location dimension, but rather normalized only for all candidate scale-representation component combinations (s,k) under a fixed task t; each z s , k The internal channel structure and spatial layout remain unchanged. The temperature coefficient of 1 can be omitted from the routing weight formula.

[0109] Routing softmax normalizes all candidate scale-representation component combinations (s,k) under a fixed task t, without changing the spatial layout within each representation component. This describes the weighted results of the task. Maintaining the H×W spatial dimensions; the weighted results of the classification task are pooled into a global vector in the spatial dimension using Pool(·). In softmax, a temperature coefficient τ is set in the exponent term, with separate coefficients used for description and classification tasks. and .

[0110] V. Description and Generation of Sha Phenomena and Constitution Classification

[0111] Step S7: Transformer Sha Image Description Generation (see...) Figure 1 )

[0112] After obtaining the spatial fusion features, they are converted into a sequence of visual tokens, which serve as keys and values ​​in the Transformer's cross-attention mechanism. The Transformer queries the spatial fusion features based on the generated word sequence, sequentially outputting the location, color, shape, intensity, left-right distribution, and special features of the bruising. To improve the stability of the description structure, a consistent description format of "location + color + shape + intensity + special features" can be used in the training corpus, without altering the Transformer decoder structure.

[0113] In a preferred embodiment, the Transformer decoder is trained using teacher forcing and uses beam search decoding (with a typical beam width of 3) during inference; the maximum description length is typically 30 words, and the vocabulary is constructed based on the statistical description text of the training set; the number of layers, attention heads, hidden dimensions, and decoding strategies can all be adjusted according to the specific data scale, and the present invention is not limited thereto.

[0114] Step S8: Multi-label body type classification

[0115] After obtaining the global fusion features for classification, these features are mapped to a nine-dimensional output, and then the Sigmoid activation function is used to obtain the probabilities of nine different physical constitutions. Since the same object can have multiple physical constitutions simultaneously, a multi-label loss method is used for training.

[0116] In a preferred embodiment, the multi-label classification loss uses weighted binary cross-entropy, and the class weights can be set according to the reciprocal of the frequency of the labels in the training set to mitigate the impact of imbalance among samples of different physical conditions.

[0117] VI. Multi-task joint training and semantic collaboration

[0118] Step S9: Multi-task joint training

[0119] During model training, joint training simultaneously optimizes the description generation loss, multi-label classification loss, mirror response consistency loss, commonality-difference orthogonality loss, dual-task routing separation loss, and bidirectional semantic consistency loss. The total joint training loss is formed by weighted summation of each loss term according to the following formula:

[0120] During the multi-task training phase, the U-Net parameters remain fixed, and the following loss is jointly optimized: description generation loss, multi-label classification loss, mirror response consistency loss, commonality-difference orthogonality loss, dual-task route separation loss, and bidirectional semantic consistency loss.

[0121]

[0122] Here, λtext, λcls, λmir, λorth, λroute, and λcon are non-negative loss weights, controlling the contributions of description generation, multi-label classification, mirror response consistency, commonality-difference orthogonality, dual-task route separation, and bidirectional semantic consistency loss to multi-task training, respectively. The segmentation network is trained independently, and its segmentation loss... and weight The total losses from the aforementioned multi-task joint training are not included.

[0123] The above six losses are averaged within a batch and then added together according to their corresponding weights to form the total loss of multi-task joint training.

[0124] Calculated from the generated word probability and the labeled descriptive word sequence; Calculated from the nine-dimensional constitution probability and the multi-label vector of constitution; The absolute difference representation of the same sample before and after horizontal mirroring is kept consistent, while the sign of the directional difference representation is reversed. Calculate according to the aforementioned orthogonal formula; Used to reduce the correlation between the description task routing weight vector and the classification task routing weight vector; The probability of physical constitution output to constrain the classification branch is kept consistent with the corrected probability obtained from the descriptive attributes through the attribute-physical constitution association layer. The multi-task losses are averaged within a batch and then summed according to their corresponding weights. Segmentation loss. Used only for training U-Net independently.

[0125] Throughout the paper, the dual-task routing separation loss is consistently denoted as the routing separation loss term. The six terms in the total multi-task loss correspond to description generation, multi-label classification, mirror response consistency, commonality-difference orthogonality, dual-task routing separation, and bidirectional semantic consistency, respectively. The segmentation loss is only used for independently training U-Net. After U-Net is fixed, the above multi-task network is trained. Therefore, the segmentation loss and its weights are not included in the total multi-task loss.

[0126] Furthermore, and The algorithm can be dynamically adjusted based on the rate of loss descent, gradient magnitude, or task uncertainty of the two tasks. When the rate of loss descent of one task is significantly lower than that of the other task, the weight of the task corresponding to that task is increased; when the gradient directions of the two tasks conflict, the gradient can be projected or the weight of the conflicting task can be decreased.

[0127] Using the Adam optimizer, the learning rate is at ~ The range is determined by the validation set, and the batch size is set according to the video memory capacity; early stopping is performed using the validation set joint loss or the combined description and classification index. The training set, validation set, and test set are divided by subject to ensure that images of the same subject do not cross sets; when using a 7:2:1 division ratio, this ratio is only an optional embodiment.

[0128] Step S10: Classification—Describe the bidirectional semantic backflow, see [link / reference] Figure 5 .

[0129] During inference and training, a bidirectional semantic backflow is established between the classification branch and the description branch. On the one hand, after the classification branch outputs the probabilities of nine constitutions, it weights and sums each probability with the corresponding learnable constitution embedding to obtain a constitution semantic token. This token serves as an additional conditional input to the Transformer decoder, used to adjust the attention paid to different sha (sha) attributes during the generation process, thereby obtaining a corrected sha description text constrained by constitution tendency. This additional condition does not directly replace visual cross-attention.

[0130] On the other hand, the description branch extracts the probabilities of location, color, shape, degree, and left-right distribution attributes based on the word probabilities at each time step, and generates a classification correction vector through the learnable attribute-physical association layer. The classification correction vector is fused with the initial logits of the classification branch through confidence gating to obtain the corrected physical tendency probability. To avoid unconstrained amplification, both directions employ alternating stopping gradients or phased training.

[0131] The classification branch feeds back the physical semantic embedding to the description branch, and the description branch feeds back the attribute correction amount to the classification branch. The complete computational relationship of the bidirectional semantic backflow between classification and description is as follows:

[0132]

[0133] in, The input sequence for the Transformer, The physical semantic vector is projected onto the Transformer model dimension and concatenated with the description space feature label as a conditional label; y and the corrected classification logits are both C-dimensional. It is an A-dimensional attribute vector, therefore For a C×A matrix, It is a C×(C+A) matrix. Let g represent the feature bias vector, whose dimension and the gating vector g are both C-dimensional; Let c be the probability of physical predisposition. For the c-th class of learnable constitution embedding, For physical constitution semantic vectors; y represents the initial classification logits of the multi-label classification module; This is the corrected classification logits.

[0134] When the number of constitution categories C is 9 and the number of Sha (symptom) attributes A is 5, The shape is C×A, that is, 9×5; The shape is C×(C+A), which is 9×14. Both g and g are 9-dimensional; The output dimension is equal to the Transformer model dimension. The direction of the above matrix is ​​multiplied by the column vector on the left. and concatenating column vectors [y; The formula is consistent with that of ].

[0135] When the constitution embedding dimension is set to 64, the attribute vector must include at least five attribute categories: location, color, shape, degree, and left-right distribution. The output dimension of the attribute-constitution association projection is consistent with the number of constitution categories. During training, the segmentation, description, and classification backbones are trained first, then the constitution semantic conditions from classification to description are enabled, and finally the attribute correction from description to classification is enabled; or, when the second direction is enabled, the following can be applied: Execute the stopping gradient. The number of rounds in each stage is determined by the validation set; training only the backbone and the first direction in the first 5 rounds is an optional implementation.

[0136] VII. Model Training, Inference, and Alternative Implementation Methods

[0137] Step S11: Model Training and Inference Output

[0138] In one example, the input back image is uniformly scaled to 224×224, with both halves using the same ImageNet pre-trained ResNet101. The output spatial features are mapped to 512 dimensions via linear layers. The Transformer decoder consists of 6 layers and 8 attention heads, with a maximum description length of 30. The multi-label classifier outputs a 9-dimensional physical tendency. Training employs the Adam optimizer and an early stopping strategy. The above parameters are for illustrative purposes only and do not constitute a limitation on the scope of protection.

[0139] In alternative implementations, weights can be computed using convolutional layers, fully connected layers, or multi-head attention within the attention fusion mechanism; dynamic task weights can be calculated using homoscedasticity uncertainty, loss reduction rate, or gradient balancing methods; however, the overall architecture of U-Net segmentation, left and right half-graph sharing of ResNet101, attention fusion, Transformer description generation, and multi-label classification remains unchanged.

[0140] This invention can be implemented using existing image acquisition equipment, general-purpose computers, servers, or mobile terminals. It can repeatedly process images of petechiae on the back of the human body and output descriptions of petechiae, probabilities of physical constitution, and interpretable heat maps. It can be applied to the digital recording of petechiae in traditional Chinese medicine, health information analysis, teaching and research, and professional auxiliary assessment. It is manufacturable and reusable.

[0141] Example 1

[0142] This embodiment provides a multi-task analysis method for sha (petechiae) symptoms. After acquiring images of sha symptoms on the human back, a back region mask is first obtained using U-Net and the back midline is fitted to correct and divide the back image into left and right halves. Then, the left half and the horizontally flipped right half are input into a ResNet101 with shared parameters, and multi-scale features of the left and right sides are extracted from multiple residual stages. After scale alignment, bidirectional mirror cross-attention is used to obtain mirror responses in two directions, which are decoupled into common representations, absolute difference representations, and directional difference representations. The description task query vector and the classification task query vector are used to generate routing weights, forming description space fusion features and classification global fusion features, respectively. The Transformer generates sha descriptions based on the description space fusion features, and the multi-label classifier outputs the constitution tendency probability based on the classification global fusion features. The two branches perform bidirectional semantic backflow through constitution semantic embedding and sha attribute correction.

[0143] In one optional implementation, the input image is uniformly scaled to 224×224, the ResNet101 uses ImageNet pre-trained parameters, the Transformer decoder includes 6 layers and 8 attention heads, and the multi-label classifier outputs 9-dimensional physical tendency probabilities. The above values ​​are for illustrative purposes only and do not constitute a limitation on the model's input size, number of layers, or number of output categories.

[0144] See Figure 6 This illustrates an example of the differential heatmap, model description text, and constitution tendency output after processing a typical blood stasis sample using the method of this embodiment.

[0145] Figure 6 The original back images, differential heatmaps, model description text, and constitution tendency outputs for four samples are shown to illustrate the output format of the color, amount, shape, location, left-right distribution, and constitution tendency of the petechiae. Figure 6 The results listed illustrate the processing flow and example output of this embodiment and are not intended as a statistical conclusion on the overall recognition accuracy. The specific meaning of the constitution should be interpreted by professionals in light of the actual situation. The output of this invention is used to assist in analysis rather than to replace professional diagnosis.

[0146] Further comparison of the descriptive texts and physical constitutions of each sample reveals that the descriptive text of sample (c) clearly indicates that the petechiae are concentrated in the right breast projection area, heart and lung area, spleen area, and liver and spleen area. The descriptive texts of samples (a) and (b) indicate a distribution pattern of low in the middle and high on both sides of the lower abdomen. This shows that the present invention can not only detect the existence of left-right asymmetry, but also preserve the direction and amplitude characteristics of asymmetry. At the same time, the petechiae of each sample were accurately located in the back anatomical regions such as the upper and middle abdomen breast area, liver and spleen area, heart and lung area, spleen area, liver area, and kidney area. This indicates that after calibration along the back midline, the corresponding left and right areas are established on the basis of the back anatomical structure, providing an accurate spatial reference for the zoning and localization of petechiae. The constitution outputs of samples (a), (c), and (d) with dark red or bruised petechiae all included blood stasis constitution. The constitution outputs of samples (a), (b), and (c) with less petechiae concentrated in the upper jiao region all included qi stagnation constitution. The constitution outputs of samples (a) and (d) with darker petechiae included yang deficiency constitution. This indicates that there is an interpretable correspondence between the semantic attributes of petechiae in the descriptive text and the constitution output of the multi-label, verifying the constraining effect of the bidirectional semantic backflow between classification and description on the consistency of the outputs of the two tasks. In addition, the thin linear special feature of the kidney area and the transverse collaterals in the kidney area at the spine in samples (c) and (d) was identified and located separately, indicating that the commonality-difference decoupling avoids the obscuring of fine-grained difference features by symmetrical commonality components.

[0147] Figure 6The high-response areas in the heatmap of differences among samples correspond to the concentrated areas of petechiae indicated in the descriptive text, providing visual evidence for the model output. Professionals can use this to verify whether the areas of interest of the model are real areas of petechiae, thus improving the interpretability and credibility of the model's decisions.

[0148] This invention maintains the overall architecture of "U-Net segmentation, left and right half-image sharing ResNet101, attention fusion, Transformer description generation, and multi-label classification" unchanged, but reconstructs ordinary attention fusion into a data flow network with multi-scale, bidirectional interaction, feature decoupling, and task-conditional routing. It can simultaneously preserve the commonalities of the images, asymmetric amplitude, and asymmetric direction, and reduce the negative transfer caused by the sharing of the same fusion feature between description and classification tasks.

[0149] Example 2

[0150] Based on the same inventive concept as Embodiment 1, this embodiment provides a multi-task analysis system for Sha (a type of facial paralysis), including a back segmentation and left / right half image generation module, a shared feature extraction module, a multi-scale mirror interaction routing module, a Transformer description generation module, a multi-label classification module, and a bidirectional semantic backflow module. Each module can be deployed on the same computing device or distributed across multiple computing devices. Each module is used to execute the corresponding steps in the multi-task analysis method for Sha described in Embodiment 1.

[0151] Furthermore, it includes an image generation module, used to input the acquired original back bruise image of the human body into a segmentation network to obtain a back region mask; and to determine the back midline based on the back region mask to form a left half image and a right half image;

[0152] The feature extraction and alignment module is used to perform directional alignment processing on the left half image and the right half image, and input the directional aligned left half image and the right half image into the parameter-shared feature extraction network to obtain left multi-scale features and right multi-scale features; and perform channel projection and spatial size alignment on the left multi-scale features and the right multi-scale features to obtain aligned left multi-scale features and aligned right multi-scale features.

[0153] The bidirectional mirror cross-attention module is used to calculate the first mirror response using the aligned left multi-scale features as the query and the aligned right multi-scale features as the key and value, and to calculate the second mirror response using the aligned right multi-scale features as the query and the aligned left multi-scale features as the key and value.

[0154] The commonality-difference decoupling module is used to generate a commonality representation of the mirror image, an absolute difference representation, and a directional difference representation with left and right direction indicators based on the first mirror image response and the second mirror image response.

[0155] The dual-task conditional routing module is used to perform a first set of gated aggregation and a second set of gated aggregation on the mirror common representation, the absolute difference representation, and the directional difference representation with left and right direction indicators to form a description space fusion feature and a classification global fusion feature.

[0156] The result output module is used to input the description space fusion features into the Transformer description generation module, input the classification global fusion features into the multi-label classification module, and perform bidirectional semantic backflow of classification and description to obtain the description text of the sha phenomenon and the probability of physical constitution tendency.

[0157] Example 3

[0158] This embodiment provides an electronic device, including at least one processor, at least one memory, and a bus connecting various system components. The memory stores a computer program, and when the processor executes the computer program, it implements the steps in the multi-task analysis method for image processing described in Embodiment 1. The electronic device can be a general-purpose computer, a server, a mobile terminal, or other device with data processing capabilities.

[0159] The modules in Embodiment 2 can be stored in a memory, and the processor is used to operate according to the instructions in the memory to execute the steps of the method described in Embodiment 1.

[0160] The system may include, but is not limited to, at least one processor, at least one memory, and a bus connecting different system components (including the memory and the processor). The memory stores program code that can be executed by the processor, causing the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of the present invention. For example, the processor may perform actions such as... Figure 1 The steps are shown.

[0161] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0162] The memory may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0163] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0164] The system can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable user interaction with the system, and / or any device that enables the system to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be achieved through input / output (I / O) interfaces. Furthermore, the system can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the system via a bus.

[0165] It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the system, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0166] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0167] Example 4

[0168] In exemplary embodiments of the present invention, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0169] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0170] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0171] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of outputting, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0172] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0173] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0174] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-task analysis method for Sha symptoms, characterized in that, include: The original image of the human back with petechiae is input into the segmentation network to obtain a mask for the back region. The back centerline is determined based on the back region mask, and the left and right halves of the image are formed. The left and right halves of the image are aligned by direction. The aligned left and right halves are then input into a parameter-shared feature extraction network to obtain left-side multi-scale features and right-side multi-scale features. Channel projection and spatial size alignment are then performed on the left-side and right-side multi-scale features to obtain aligned left-side and right-side multi-scale features. The first mirror response is calculated using the aligned left-side multi-scale features as the query and the aligned right-side multi-scale features as the key and value. The second mirror response is calculated using the aligned right-side multi-scale features as the query and the aligned left-side multi-scale features as the key and value. Based on the first mirror response and the second mirror response, generate mirror commonality representation, absolute difference representation, and directional difference representation with left and right direction indicators; The mirror commonality representation, the absolute difference representation, and the directional difference representation with left and right direction indicators are subjected to a first group of gated aggregation and a second group of gated aggregation to form a spatial fusion feature and a global fusion feature for classification. The description space fusion features are input into the Transformer description generation module, and the classification global fusion features are input into the multi-label classification module. A bidirectional semantic backflow of classification and description is performed to obtain the description text of the sha phenomenon and the probability of physical constitution.

2. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, The step of determining the back midline based on the back region mask and forming the left and right halves of the image includes: The leftmost and rightmost boundary points of the back region mask at multiple preset height positions are obtained respectively; Calculate the midpoint between the leftmost boundary point and the rightmost boundary point at each height position to obtain a midpoint sequence composed of multiple midpoints; The back midline is fitted according to the midpoint sequence, and the back region is corrected based on the back midline so that the corrected back midline coincides with the longitudinal central axis of the corrected image. The corrected back region is divided into a left half and a right half along the longitudinal central axis.

3. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, First mirror response Second mirror response The calculation formula is: ； ； Where s represents the scale; L S and R S These are the left and right feature matrices, respectively, after channel projection and spatial size alignment, and flattened into N spatial locations, where N is the number of spatial locations in the aligned feature map; Q(·), K(·), and V(·) represent the linear projections of the query, key, and value, respectively, with the two attention directions sharing the corresponding projection parameters; d is the dimension of the key vector, T represents the matrix transpose, and softmax normalizes along all key positions corresponding to each query position.

4. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, The common characteristics of the mirror Absolute difference representation and The formula for calculating the directional difference is as follows: ； ； ； in, , and These are the learnable projection parameters corresponding to commonality representation, absolute difference representation, and directional difference representation, respectively; the symbol ";" indicates concatenation in the channel dimension, ⊙ indicates element-wise multiplication, and |·| indicates taking the absolute value of each element; To correspond to spatial location and be encoded in the left and right directions that can be broadcast along the channel dimension, the left and right sides use encoding with opposite signs to preserve the source direction of the difference.

5. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, The first set of gated aggregation and the second set of gated aggregation are performed on the mirror commonality representation, the absolute difference representation, and the directional difference representation with left and right direction indicators to form descriptive spatial fusion features and classification global fusion features, including: The dual-task conditional routing layer in a multi-scale mirror interactive routing network stores the query vector describing the task. and classification task query vector The representation components are the components of the mirror commonality representation, the absolute difference representation, and the directional difference representation with left and right direction indicators at each scale. Route scoring first converts spatial representation components into channel description vectors, defined as follows: ； Where GAP(·) represents global average pooling over the spatial dimension; the routing weights are normalized over all scale-representation component combinations (s,k) under a fixed task t, and the ergodices in the denominator are s′ and k′, which are calculated using the following formula: ； Where t∈{cap,cls} represent the description task and the classification task, respectively; Let be the learnable query vector for task t. and These are used to describe task query vectors and classification task query vectors, respectively. Let k represent the k-th type of representation component at scale s, where k is taken from the mirror commonality representation, absolute difference representation, and directional difference representation. Let be the learnable route projection parameters for task t; s′ and k′ are the traversal indices in the normalized summation, and the denominator is summed over all candidate scale-representation components to make the sum of all route weights for the corresponding task equal to 1. The dual-task conditional routing layer queries the task description vector. The generated routing weights are weighted and aggregated across the representation components at each scale while maintaining spatial resolution, forming spatial fusion features that describe the route. : ； The dual-task conditional routing layer queries the categorized task vector. The generated routing weights are weighted and aggregated across the representation components at each scale, then pooled to form a global fusion feature for classification. : ； The first set of gated aggregations corresponds to the description task, and the second set of gated aggregations corresponds to the classification task. The routing projection parameters and task query vectors of the two sets are not shared. Pool(·) indicates that global average pooling or global attention pooling is performed on the spatial dimension, making... Spatial location is preserved for description generation. A global vector is generated for multi-label classification.

6. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, The process of inputting the fused features of the description space into the Transformer description generation module, inputting the global fused features of classification into the multi-label classification module, and performing bidirectional semantic backflow of classification and description to obtain the description text of the symptoms and the probability of physical constitution includes: The Transformer description generation module converts the description space fusion features into a visual memory sequence, which serves as the key and value in the Transformer cross attention. It then queries the visual memory sequence based on the generated word sequence to generate a description text of the sha phenomenon. The description text of the sha phenomenon includes at least three types of information: the location of the sha phenomenon, the color of the sha phenomenon, the shape of the sha phenomenon, the degree of the sha phenomenon, the left-right distribution characteristics of the sha phenomenon, and the special signs of the sha phenomenon. The multi-label classification module maps the global fusion features of the classification into a multi-dimensional output through a fully connected layer, and obtains multiple constitution tendency probabilities through a Sigmoid activation function; wherein, the multiple constitution tendency probabilities include one or more of the following: balanced constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, phlegm-dampness constitution, damp-heat constitution, blood stasis constitution, qi stagnation constitution, and special constitution; In the first direction of the bidirectional semantic backflow, the multi-label classification module weights and sums the constitution tendency probability with the corresponding learnable constitution embedding to form a constitution semantic vector, and inputs the constitution semantic vector as an additional condition token into the Transformer description generation module to adjust the attention of the Transformer description generation module to different Sha attributes when generating the Sha phenomenon description text, so as to obtain a corrected Sha phenomenon description text constrained by the constitution tendency probability; In the second direction of the bidirectional semantic backflow, during the process of generating the description text of the sha phenomenon, the Transformer description generation module extracts the probability of the location, color, shape, degree and left-right distribution attributes of the sha phenomenon based on the word probability of each time step of the generated word sequence. The attribute probabilities are then used to form a classification correction vector through a learnable attribute-constitution association matrix. The classification correction vector is then gated and fused with the initial classification logits of the multi-label classification module to obtain the corrected constitution tendency probability.

7. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, The step of aligning the left and right halves of the image, inputting the aligned left and right halves into a parameter-shared feature extraction network to obtain left-side multi-scale features and right-side multi-scale features, and then performing channel projection and spatial size alignment on the left and right multi-scale features includes: Perform a horizontal flip on the right half of the image so that the corresponding anatomical regions in the left half and the flipped right half have the same spatial arrangement direction; or perform a horizontal flip on the left half of the image so that the corresponding anatomical regions in the right half and the flipped left half have the same spatial arrangement direction. The left and right halves of the image after orientation alignment are respectively input into the parameter-shared ResNet101 feature extraction network, and the left and right multi-scale features are obtained from at least two residual stages from the second to the fifth residual stages of the ResNet101 feature extraction network. After obtaining the left-side multi-scale features and the right-side multi-scale features, the scale alignment layer in the multi-scale mirror interactive routing network converts the left-side multi-scale features and the right-side multi-scale features into the same channel dimension and preset spatial size through channel projection and upsampling or downsampling, respectively, and adds scale position encoding to the left-side multi-scale features and the right-side multi-scale features.

8. The multi-task analysis method for Sha phenomena according to claim 1, characterized in that, Multi-task joint training is performed on the Transformer description generation module, the multi-label classification module, and the multi-scale mirror interaction routing network. The total loss of the multi-task joint training is formed by the weighted sum of description generation loss, multi-label classification loss, mirror response consistency loss, commonality-difference orthogonality loss, dual-task routing separation loss, and bidirectional semantic consistency loss.

9. A multi-task analysis system for Sha symptoms, characterized in that, include: The image generation module is used to input the acquired original back bruise image into the segmentation network to obtain a back region mask; The back centerline is determined based on the back region mask, and the left and right halves of the image are formed. The feature extraction and alignment module is used to perform directional alignment processing on the left half image and the right half image, and input the directional aligned left half image and the right half image into the parameter-shared feature extraction network to obtain left multi-scale features and right multi-scale features; and perform channel projection and spatial size alignment on the left multi-scale features and the right multi-scale features to obtain aligned left multi-scale features and aligned right multi-scale features. The bidirectional mirror cross-attention module is used to calculate the first mirror response using the aligned left multi-scale features as the query and the aligned right multi-scale features as the key and value, and to calculate the second mirror response using the aligned right multi-scale features as the query and the aligned left multi-scale features as the key and value. The commonality-difference decoupling module is used to generate a commonality representation of the mirror image, an absolute difference representation, and a directional difference representation with left and right direction indicators based on the first mirror image response and the second mirror image response. The dual-task conditional routing module is used to perform a first set of gated aggregation and a second set of gated aggregation on the mirror common representation, the absolute difference representation, and the directional difference representation with left and right direction indicators to form a description space fusion feature and a classification global fusion feature. The result output module is used to input the description space fusion features into the Transformer description generation module, input the classification global fusion features into the multi-label classification module, and perform bidirectional semantic backflow of classification and description to obtain the description text of the sha phenomenon and the probability of physical constitution tendency.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images