A dual-mode dynamic evaluation system for breast cancer HER2 status

Abstract

This invention discloses a dual-modal dynamic assessment system for HER2 status in breast cancer, relating to the field of medical diagnostic technology. The system includes: an acquisition module that acquires dual-modal images of the patient's clinical information and breast lesions, constructs a multi-scale tumor microenvironment region, and extracts multi-dimensional features of the tissue's inherent biological state; an analysis module that integrates the multi-dimensional features of the biological state with clinical information to perform a three-level classification assessment and dual uncertainty decomposition of HER2 expression status in breast cancer, outputting stratified assessment results; a mapping module that uses the stratified assessment results to synthesize virtual pathological staining images spatially aligned with the dual-modal images and performs spatial inconsistency labeling, outputting virtual pathological visualization images; and a decision-making module that combines the virtual pathological visualization images with the stratified assessment results to generate personalized clinical pathway suggestions and targeted resampling region recommendations. This invention improves the accuracy and clinical operability of HER2 expression status assessment.

Description

Technical Field

[0001] This invention relates to the field of medical diagnostic technology, and in particular to a dual-modal dynamic assessment system for HER2 status in breast cancer. Background Technology

[0002] In recent years, the expression status of human epidermal growth factor receptor 2 (HER2) has become a core molecular indicator for precision diagnosis and prognostic assessment of breast cancer. As clinical understanding deepens regarding the benefits of antibody-drug conjugate (ADC) therapy for HER2-low expression, the traditional binary assessment system based on immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH) is evolving towards a more refined three-tiered classification paradigm. Against this backdrop, medical radiomics technology, with its non-invasive nature and high-throughput characterization capabilities for tissue heterogeneity, has shown great potential in the field of auxiliary molecular subtyping prediction. Current mainstream technologies often employ multimodal ultrasound or photoacoustic imaging to extract morphological, functional blood flow, and physiological metabolic characteristics of specific regions within and around the tumor, and then construct machine learning classifiers to achieve predictive assessment of HER2 expression status on a single examination slice.

[0003] However, most existing technologies are limited to static, single-stage classification prediction. Their assessment logic mainly relies on linear mapping or single numerical matching between imaging features and the gold standard, failing to fully consider the biological evolution characteristics of breast lesions at different treatment cycles or observation points (e.g., regression of neovascularization within the tumor during neoadjuvant therapy, real-time fluctuations in blood oxygen saturation, and acoustic property evolution due to changes in cell density). This assessment mechanism, lacking continuous monitoring in spatiotemporal dimensions, makes it difficult for the system to identify confidence biases in assessment results caused by fluctuations in the microstructure of lesion tissue when facing complex cases with high intratumoral heterogeneity or at the classification boundary. Consequently, it cannot provide evolutionary trend analysis with clinical decision-making reference value for the dynamic evolution of the tumor microenvironment. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a dual-modal dynamic assessment system for HER2 status in breast cancer to solve the problem of difficulty in identifying fluctuations in assessment results in complex cases due to the lack of continuous monitoring of the evolution of the biological characteristics of lesions.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a dual-modal dynamic assessment system for HER2 status in breast cancer, comprising: The acquisition module obtains dual-modal images of the patient's clinical information and breast lesions, and constructs a multi-scale tumor microenvironment region to extract multi-dimensional features of the tissue's inherent biological state. The analysis module integrates multidimensional features of biological status with clinical information to perform a three-level classification assessment and dual uncertainty decomposition of HER2 expression status in breast cancer, and outputs stratified assessment results. The mapping module uses the hierarchical evaluation results to synthesize virtual pathological staining images that are spatially aligned with the bimodal image and marks them for spatial inconsistency, outputting virtual pathological visualization images. The decision-making module combines virtual pathology visualization images with stratified assessment results to generate personalized clinical pathway suggestions and targeted resampling area recommendations. The tracking module compares personalized clinical pathway recommendations with targeted resampling area recommendations at different time points, identifies evolution trends, and outputs multi-time-point dynamic assessment reports.

[0007] Preferably, the method for acquiring the dual-modal images of the subject's clinical information and breast lesions includes: The clinical information of the subjects is obtained, and the non-numerical variables in the clinical information are mapped into multi-dimensional one-hot encoded vectors to obtain the clinical information features. Simultaneously acquire raw ultrasound images, raw photoacoustic images, physiological signals, and corresponding acquisition parameter files of the breast lesion area, and combine them with clinical information features to form a dual-modal image data package.

[0008] Preferably, the method for constructing the multi-scale tumor microenvironment region includes: The original ultrasound images in the dual-modal image data package are used to identify lesion boundaries and extract the section with the maximum diameter feature as the target layer. The region inside the lesion boundary at the target level is constructed as the main tumor region, generating a main tumor region mask; Based on the edge of the tumor main area mask, expand outward at equal intervals, and construct the area outside the tumor main area formed by the expansion as the peritumor transition zone, thus generating the peritumor transition zone mask; In the target layer, a normal tissue area located outside the mask of the peritumor transition zone is selected to construct a background reference area, and a background reference area mask is generated. By integrating the tumor main region mask, the surrounding tumor transition region mask, and the background reference region mask, a multi-scale tumor microenvironment region is obtained.

[0009] Preferably, the method for extracting the multidimensional features of the tissue's inherent biological state includes: Based on the multi-scale tumor microenvironment region, ultrasound features are extracted from the original ultrasound image, and photoacoustic features are extracted from the original photoacoustic image, and combined to obtain the initial dual-modal image information; By using the acquired parameter files and physiological signals, signal variation components introduced by differences in equipment parameters and physiological fluctuations are separated from the initial bimodal image information to obtain interference-free bimodal image information; In the de-interference dual-modal imaging information, the functional and structural states of lesions and microenvironment tissues are used as multidimensional features of the tissue's inherent biological state.

[0010] Preferably, the method for performing the three-level classification assessment of HER2 expression status in breast cancer includes: The multidimensional features of biological states and clinical information features are structurally aligned according to a preset coordinate system to obtain a multimodal joint representation set; Based on the assessment needs in the clinical information features, the contribution ratio of imaging components and clinical components in the multimodal joint representation set is dynamically adjusted and fused to output comprehensive diagnostic features; The comprehensive diagnostic features are mapped to the preset pathological feature mapping benchmark to generate evidence quality indicators supporting the three levels of HER2-zero, HER2-low, and HER2-positive. The HER2 expression level of the breast lesion is determined based on the proportion of evidence quality indicators, and the three-level classification assessment tendency is output.

[0011] Preferably, the decomposition of the dual uncertainty includes:

[0012] Extract the distribution of evidence quality indicators across different levels and identify the matching deviation between comprehensive diagnostic features and the pre-defined pathological feature mapping benchmark;

[0013] Based on the matching bias, the deviation portion of the comprehensive diagnostic features that exceeds the coverage range of the preset pathological feature mapping benchmark is identified as the cognitive uncertainty component; the fuzzy fluctuation portion of the evidence quality index caused by the comprehensive diagnostic features being at the classification boundary is identified as the accidental uncertainty component.

[0014] It integrates three-level classification assessment of tendency, cognitive uncertainty component and accidental uncertainty component, and outputs hierarchical assessment results.

[0015] Preferably, the method of synthesizing the virtual pathological staining image aligned with the bimodal image space includes:

[0016] The hierarchical evaluation results are correlated with the original ultrasound image and the original photoacoustic image at the pixel level to establish colorimetric constraints that include the anatomical spatial structure of the lesion in the original ultrasound image and the tissue metabolic functional state in the original photoacoustic image, forming a set of spatial functional constraints.

[0017] Based on the spatial functional constraint set, the three-level classification assessment tendency is converted into corresponding color codes in the multi-scale tumor microenvironment region, and the immunohistochemical staining distribution is simulated pixel by pixel to generate preliminary pathological staining images.

[0018] Using the lesion boundary in the original ultrasound image as a reference, the edge of the color-coded distribution area in the preliminary pathological staining image is corrected, and a virtual pathological staining image aligned with the dual-modal image space is output.

[0019] Preferably, the method of outputting virtual pathological visualization images includes:

[0020] The virtual pathological staining image is divided into several sub-regions. The pixel proportion of color coding in each sub-region is counted, and the color coding with the highest proportion is taken as the staining judgment category corresponding to the sub-region. The staining judgment category is compared with the highest probability category in the three-level classification assessment, and the sub-regions where the two judgments conflict are identified as inconsistent sub-regions.

[0021] The boundary contours of inconsistent sub-regions are extracted to generate inconsistency region masks. The positions corresponding to the inconsistency region masks are visually enhanced in the virtual pathological staining image to obtain a virtual pathological staining image with spatial inconsistency markers. This image is then fused with the original ultrasound image and the original photoacoustic image to output a virtual pathological visualization image.

[0022] Preferably, the methods for generating personalized clinical pathway suggestions and targeted resampling area recommendations include:

[0023] Based on the proportion of evidence quality indicators at each level of the three-level classification assessment, and matching the preset treatment decision logic, personalized clinical pathway suggestions are generated.

[0024] The inconsistent region mask and the high-value region of the random uncertainty component are spatially intersected to determine the overlapping sub-regions; the boundary contours of the overlapping sub-regions are extracted and spatial coordinates are generated, and the spatial coordinates are mapped to the corresponding anatomical positions of the original ultrasound images to generate targeted resampling region recommendations.

[0025] Preferably, the method for outputting the multi-time-point dynamic evaluation report includes:

[0026] Spatial registration is performed on the targeted resampling area recommendations at the current time point and at historical time points, and personalized clinical pathway recommendations at different time points are compared to identify recommendations for adjustment records;

[0027] The overlap and position offset vectors of the target resampling areas recommended at different time points after registration are extracted, and the evolution trend is identified by combining the numerical changes of the cognitive uncertainty component and the accidental uncertainty component; the suggested adjustment records and evolution trend are integrated to output a multi-time point dynamic evaluation report.

[0028] The beneficial effects of this invention are as follows: By performing dual uncertainty decomposition, it achieves refined measurement and attribution of the reliability of diagnostic conclusions, providing a transparent benchmark for clinical decision-making to distinguish between knowledge limitations and data ambiguities, thereby ensuring the safety of stratified assessment; on this basis, by performing spatial inconsistency labeling and region intersection, it achieves precise spatial localization and visual guidance of heterogeneous conflict points within lesions, providing deterministic physical coordinate references for targeted resampling; synergistically, it transforms "black-box" image assessment into "interpretable, localizable, and verifiable" precision medical decision-making, improving the accuracy and clinical operability of HER2 expression status assessment in non-invasive dynamic monitoring. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of a dual-modal dynamic assessment system for HER2 status in breast cancer according to the present invention.

[0031] Figure 2 This is a flowchart illustrating the process of obtaining multi-scale tumor microenvironment regions in this invention.

[0032] Figure 3 This is a flowchart for outputting the hierarchical evaluation results in this invention.

[0033] Figure 4 This is a flowchart illustrating the generation of personalized clinical pathway suggestions and targeted resampling region recommendations in this invention. Detailed Implementation

[0034] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0036] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0037] Reference Figure 1 , Figure 2 , Figure 3 and Figure 4 As one embodiment of the present invention, this embodiment provides a dual-modal dynamic assessment system for HER2 status in breast cancer, comprising the following steps:

[0038] Methods for obtaining dual-modal imaging of the patient's clinical information and breast lesions include:

[0039] The clinical information of the subjects is obtained, and the non-numerical variables in the clinical information are mapped into multidimensional one-hot encoded vectors to obtain the clinical information features.

[0040] Specifically, before conducting a breast examination, the system extracts raw data such as the examinee's age (example value: 45 years old), clinical T stage (example value: T2 stage), hormone receptor status (example value: ER positive), and past treatment history through an interactive interface. For non-numerical variables in the clinical information, a classification index mapping method is used to convert textual descriptions into discrete numerical values. Then, a multidimensional one-hot encoding algorithm is used to convert each discrete numerical value into a binary vector where only the corresponding index position is 1 and the rest are 0. For example, "ER positive" in hormone receptor status is mapped to a vector (example value: [1,0]), and "ER negative" is mapped to a vector (example value: [0,1]). Finally, all the generated binary vectors are concatenated in a preset order (example concatenation order: age vector first, clinical T stage vector second, and hormone receptor status vector last) to obtain clinical information features that characterize the individual differences of the examinee.

[0041] Simultaneously acquire raw ultrasound images, raw photoacoustic images, physiological signals, and corresponding acquisition parameter files of the breast lesion area, and combine them with clinical information features to form a dual-modal image data package.

[0042] Specifically, the integrated photoacoustic-ultrasound imaging device is used to simultaneously scan the breast lesion area of ​​the examinee. The dual mode refers to the ultrasound imaging mode and the photoacoustic imaging mode. The ultrasound transducer is triggered under the same trigger clock to receive the acoustic echo reflecting the tissue anatomical structure to generate the original ultrasound image.

[0043] Synchronously triggered laser emits pulsed laser light that penetrates the tissue, causing the breast lesion to absorb energy and generate a thermoelastic effect that excites ultrasound waves. These waves are received by an ultrasound transducer and generate a raw photoacoustic image that reflects the tissue's metabolic state.

[0044] Simultaneously, sensors are used to monitor the subject's pulse (75 beats per minute for example) and blood pressure (120 / 80 mmHg for example) in real time to form physiological signals. The current laser wavelength (800 nm for example), probe frequency (7.5 MHz for example), and gain parameter (40 dB for example) are automatically recorded to form an acquisition parameter file. Finally, the original ultrasound image, original photoacoustic image, physiological signal, acquisition parameter file, and clinical information features are packaged into a folder with the same timestamp index to form a dual-modal image data package.

[0045] Methods for constructing multi-scale tumor microenvironment regions include:

[0046] The original ultrasound images in the dual-modal image data package are used to identify lesion boundaries, and the section with the maximum diameter feature is extracted as the target layer.

[0047] It should be noted that the original ultrasound image is extracted from the dual-modal image data package, and the original ultrasound image is divided into overlapping small regions. The pixel gray values ​​in each small region are weighted and summed. The weight coefficients for the weighted summation are values ​​learned in advance from a large number of breast ultrasound images (exemplary values ​​are: the first row of the weight coefficient matrix for extracting horizontal edges is [-1,-2,-1], the second row is [0,0,0], and the third row is [1,2,1], and the first row of the weight coefficient matrix for extracting vertical edges is [-1,0,1], the second row is [-2,0,2], and the third row is [-1,0,1]). The summation result is mapped by a nonlinear function to obtain the feature response value at the center of the small region. The magnitude of the feature response value reflects the intensity of the gray level change at that position in the original ultrasound image. The set of pixels with higher feature response values ​​(exemplary values ​​higher than 80) and forming a continuous closed contour in space are identified as the lesion boundary.

[0048] Then, the Euclidean distance between any two points within the lesion boundary is calculated, and the straight line segment with the largest physical distance within the lesion boundary is searched and extracted as the maximum diameter. By comparing the maximum diameter values ​​of multiple sections in the dual-modal image data package, the section image with the largest maximum diameter value (exemplarily a section with a maximum diameter of 2.5cm) is selected as the target layer.

[0049] The region inside the lesion boundary at the target level is constructed as the main tumor region, generating a main tumor region mask;

[0050] Based on the edge of the tumor main area mask, the area is expanded outward at equal intervals, and the area formed by the expansion and located outside the tumor main area is constructed as the peritumor transition zone, thus generating the peritumor transition zone mask.

[0051] Specifically, at the target level, all pixels surrounded by the lesion boundary are filled with a high brightness value (exemplary value is 255), and pixels outside the lesion boundary are filled with a background value (exemplary value is 0), thus generating a tumor main area mask. The space defined inside the tumor main area mask is the tumor main area.

[0052] Then, starting from the edge coordinates of the tumor main area mask, a circular window with the edge coordinate point as the center and the radius as the expansion width (the example value is 5mm) is used to move the circular window outward pixel by pixel along the edge normal direction. After each movement, it is determined whether the circular window overlaps with the tumor main area mask. If there is an overlap, all the external pixels covered by the circular window are marked as valid. After traversing all edge coordinate points, the expanded area is obtained.

[0053] The tumor main area mask is subtracted from the expanded region. That is, pixels in the expanded region that overlap with the tumor main area mask are removed, and pixels that are only in the expanded region and not in the tumor main area are retained. The retained surrounding region is defined as the peritumor transition region, and the peritumor transition region mask is generated.

[0054] In the target layer, a normal tissue area located outside the mask of the peritumor transition zone is selected to construct a background reference area, and a background reference area mask is generated.

[0055] Specifically, the outer boundary of the peritumor transition zone mask is located at the target level, avoiding areas in the original ultrasound image of the dual-modal image data package where the gray value is significantly lower than the surrounding tissue and the area is tubular (i.e., large blood vessel area), as well as isolated high-brightness spot-like areas (i.e., photoacoustic artifact areas) in the original photoacoustic image. Normal breast glandular tissue areas that are far from the lesion and have uniform echoes are selected. Multiple spatially non-overlapping sampling areas are extracted within the normal breast tissue using rectangular or circular sampling frames (for example, three circular areas with an area of ​​10 square millimeters are selected). The pixel positions within the above sampling areas are integrated and assigned specific label values ​​to construct a background reference area, and a background reference area mask is generated accordingly.

[0056] By integrating the tumor main region mask, the surrounding tumor transition region mask, and the background reference region mask, a multi-scale tumor microenvironment region is obtained.

[0057] Specifically, the generated tumor main area mask, peritumor transition area mask, and background reference area mask are superimposed in multiple channels according to the original spatial coordinates of the target level. Different color channels are used to color-code the three regions (for example, the tumor main area is red, the peritumor transition area is green, and the background reference area is blue), so that the lesion and its surrounding microenvironment form a hierarchical logical topological relationship in the anatomical space. Finally, the tumor main area mask, peritumor transition area mask, and background reference area mask are integrated to obtain a multi-scale tumor microenvironment region.

[0058] Methods for extracting multidimensional features of the inherent biological state of tissues include:

[0059] Based on the multi-scale tumor microenvironment region, ultrasound features are extracted from the original ultrasound image, and photoacoustic features are extracted from the original photoacoustic image, and the two are combined to obtain the initial bimodal image information.

[0060] Specifically, based on the tumor body area mask, the peritumor transition area mask, and the background reference area mask in the multi-scale tumor microenvironment region, the pixels covered by the three regions are located in the original ultrasound image and the original photoacoustic image, respectively.

[0061] For ultrasound features, all pixels covered by the mask in each region are traversed in the original ultrasound image. The gray values ​​are read and statistically analyzed pixel by pixel. The gray value distribution is statistically analyzed from the main tumor area and the area, aspect ratio and edge irregularity are calculated. The gray value change rate along the radial direction of the lesion is statistically analyzed from the surrounding tumor transition area and the echo attenuation gradient and boundary sharpness index are calculated. The gray values ​​are statistically analyzed from the background reference area and the average gray value is calculated as the ultrasound echo reference value.

[0062] For photoacoustic features, all pixels covered by the mask in each region are traversed in the original photoacoustic image. The relative concentration values ​​of oxyhemoglobin and deoxyhemoglobin are read pixel by pixel and statistically analyzed. The mean relative concentration values ​​of oxyhemoglobin and deoxyhemoglobin are calculated from the main tumor region. The spatial variation coefficient of oxygen saturation is calculated from the peritumor transition region. The mean relative concentration value of oxyhemoglobin is calculated from the background reference region as the reference value for photoacoustic function.

[0063] Subsequently, following a fixed order of the main tumor region, the surrounding transition region, and the background reference region, the ultrasound features and photoacoustic features are joined end-to-end to form a long vector of fixed dimension, which is then combined to obtain the initial bimodal image information.

[0064] By using the acquired parameter files and physiological signals, signal variation components introduced by differences in equipment parameters and physiological fluctuations are separated from the initial bimodal image information to obtain interference-free bimodal image information.

[0065] Specifically, the acquisition parameter file and physiological signals are extracted from the dual-modal image data package. The acquisition parameter file records the laser wavelength (exemplary value 800nm) and probe frequency (exemplary value 7.5MHz), and the physiological signals record the pulse (exemplary value 75 beats per minute) and blood pressure (exemplary value 120 / 80mmHg).

[0066] Laser wavelength and probe frequency are used as equipment-related variables, and pulse and blood pressure are used as physiological state variables. The equipment-related variables and physiological state variables together constitute the external factors affecting the amplitude of photoacoustic signals.

[0067] Subsequently, in the initial bimodal image information, the values ​​of the photoacoustic feature are decomposed into amplitude scaling components and offset components that are linearly related to external factors. The amplitude scaling component shows a monotonic trend with the increase of laser wavelength, while the offset component shows periodic fluctuations with the pulse. By substituting the values ​​of external factors into the signal correction relationship, the amplitude scaling component related to the difference in equipment parameters and the offset component related to physiological fluctuations are subtracted from the initial bimodal image information to separate the signal variation component. The remaining part is the de-interference bimodal image information.

[0068] It should be noted that the signal correction relationship is established by repeatedly acquiring photoacoustic signals from hemoglobin phantoms of different concentrations under the same imaging conditions, recording the variation curves of photoacoustic signal amplitude under different combinations of laser wavelength and probe frequency, and simultaneously recording the offset of the photoacoustic signal baseline under different pulse and blood pressure states. The correspondence between laser wavelength and photoacoustic signal amplitude is fitted to a linear scaling function, and the correspondence between pulse and photoacoustic signal baseline is fitted to a periodic offset function. The two functions are superimposed to form the signal correction relationship.

[0069] In the de-interference dual-modal imaging information, the functional and structural states of lesions and microenvironment tissues are used as multidimensional features of the tissue's inherent biological state.

[0070] Specifically, the de-interference dual-modal imaging information retains the ultrasound and photoacoustic features of the main tumor area, the surrounding transition area, and the background reference area, which together reflect the functional and structural state of the lesion and the microenvironment tissue. All ultrasound and photoacoustic features of the main tumor area and the surrounding transition area in the de-interference dual-modal imaging information are retained, while the ultrasound and photoacoustic features of the background reference area are retained as reference baseline values. After integration, the inherent biological state multidimensional features of the tissue are obtained.

[0071] Traditional breast cancer imaging diagnostic methods typically output definitive HER2 stratification conclusions directly. However, due to the high heterogeneity of breast cancer tissue and the susceptibility of image acquisition to interference from equipment parameters and individual physiological differences, a single classification result often masks the inherent ambiguity of the data and the limitations of the model's understanding. This makes it difficult for physicians to judge the reliability of AI conclusions, especially in the HER2-low classification boundary areas, which can easily lead to misdiagnosis. Therefore, this invention integrates multidimensional clinical information and biological characteristics, and introduces evidence theory to refine the uncertainties in the diagnostic process. The specific steps are as follows:

[0072] The three-tiered classification assessment method for HER2 expression status in breast cancer includes:

[0073] By aligning the multidimensional features of biological states with clinical information features according to a preset coordinate system, a multimodal joint representation set is obtained.

[0074] It should be noted that, based on the multidimensional characteristics of the inherent biological state of the tissue, these multidimensional characteristics include the area of ​​the main tumor region (example value 3.2 square centimeters), aspect ratio (example value 1.5), edge irregularity (example value 0.8), echo attenuation gradient (example value -2.1 dB / mm), boundary sharpness index (example value 0.6), mean relative concentration of oxyhemoglobin in the main tumor region (example value 0.45), mean relative concentration of deoxyhemoglobin (example value 0.32), and spatial variation coefficient of oxygen saturation in the peritumoral transition zone (example value 0.25), which reflect the functional and structural state of the lesion and its microenvironment.

[0075] Simultaneously, clinical information features are extracted from the dual-modal image data package. The clinical information features include an age encoding vector (an example vector mapped from the value of 45 years old), a clinical T stage encoding vector (an example vector mapped from the value of T2 stage), and a hormone receptor status encoding vector (an example vector [1,0] corresponding to ER positive).

[0076] Each feature value in the multidimensional features of the organization's inherent biological state is used as the coordinate of the image dimension, and each encoded vector in the clinical information features is used as the coordinate of the clinical dimension. They are concatenated in a unified order with the image dimension first and the clinical dimension last. The long vector formed after concatenation is the multimodal joint representation set.

[0077] Based on the assessment needs in clinical information features, the contribution ratio of imaging components and clinical components in the multimodal joint representation set is dynamically adjusted and fused to output comprehensive diagnostic features.

[0078] It should be noted that the clinical purpose of this examination is derived from the clinical information characteristics. The clinical purpose includes three types: initial diagnosis stratification, neoadjuvant efficacy assessment, and relapse monitoring.

[0079] The adjustment method for image components and clinical components is determined according to clinical purpose. When stratifying at the initial diagnosis, the adjustment method for image components and clinical components is to keep the original values ​​of image features and multiply the clinical features by a scaling factor (0.8 for example).

[0080] When assessing neoadjuvant therapy efficacy, the adjustment method for imaging and clinical components is as follows: imaging features are multiplied by a scaling factor (example value 1.2), while clinical features retain their original values.

[0081] When monitoring recurrence, the adjustment method for image components and clinical components is to multiply the image features by a scaling factor (exemplary value 1.0) and the clinical features by a scaling factor (exemplary value 1.0). According to the determined adjustment method, the feature values ​​belonging to the image components and the feature values ​​belonging to the clinical components in the multimodal joint representation set are weighted respectively. The weighted image components and clinical components are then reassembled to obtain the comprehensive diagnostic features.

[0082] The comprehensive diagnostic features are mapped to the preset pathological feature mapping benchmark to generate evidence quality indicators supporting the three levels of HER2-zero, HER2-low, and HER2-positive. The HER2 expression level of the breast lesion is determined based on the proportion of evidence quality indicators, and the three-level classification assessment tendency is output.

[0083] It should be noted that a pathological feature mapping benchmark is pre-constructed. The pathological feature mapping benchmark contains three sets of typical feature vectors. The first set of typical feature vectors corresponds to the mean vector of the comprehensive diagnostic features of multiple samples that have been pathologically diagnosed as HER2-zero.

[0084] The second set of typical feature vectors corresponds to the mean vector of comprehensive diagnostic features of multiple samples pathologically diagnosed as HER2-low;

[0085] The third set of typical feature vectors corresponds to the mean vector of comprehensive diagnostic features of multiple samples that were pathologically diagnosed as HER2-positive;

[0086] The comprehensive diagnostic features are compared with the three sets of typical feature vectors in a dimension-wise distance. The dimension-wise distance comparison is performed by calculating the absolute value of the difference between the feature value at each position in the comprehensive diagnostic features and the feature value at the same position in the corresponding typical feature vector.

[0087] For each level, the absolute values ​​of the differences between the comprehensive diagnostic features and the typical feature vectors of that level at all positions are summed to obtain a total distance value. The three levels yield the first, second, and third total distance values, respectively. The smaller the total distance value, the higher the degree of matching between the comprehensive diagnostic features and the typical feature vectors of that level.

[0088] The three total distance values ​​are converted into evidence quality indicators by taking the reciprocal of the total distance values. The reciprocal of the first total distance value is used as the evidence quality indicator supporting the HER2-zero level (exemplary value of 0.2), the reciprocal of the second total distance value is used as the evidence quality indicator supporting the HER2-low level (exemplary value of 0.6), and the reciprocal of the third total distance value is used as the evidence quality indicator supporting the HER2-positive level (exemplary value of 0.2).

[0089] Calculate the sum of the three evidence quality indicators (example value 1.0), calculate the proportion of each evidence quality indicator to the sum, and obtain the evidence quality indicator proportion; compare the evidence quality indicator proportions of the three levels, select the level with the largest proportion as the HER2 expression level of the breast lesion (example value HER2-low), and output the three-level classification assessment tendency.

[0090] The decomposition methods for dual uncertainty include:

[0091] Extract the distribution of evidence quality indicators across different levels and identify the matching deviation between comprehensive diagnostic features and the pre-defined pathological feature mapping benchmark.

[0092] Specifically, the three evidence quality indicators are arranged in the order of HER2-zero, HER2-low, and HER2-positive, and the degree of difference in the values ​​of the three evidence quality indicators is observed. When the values ​​of the three evidence quality indicators are close to each other (e.g., the three evidence quality indicators are 0.33, 0.34, and 0.33 respectively), it indicates that the distribution of evidence quality indicators among different levels is relatively uniform.

[0093] When one of the three evidence quality indicators is significantly higher than the other two (e.g., the three evidence quality indicators are 0.6, 0.2, and 0.2 respectively), it indicates that the distribution of evidence quality indicators is relatively concentrated across different levels.

[0094] Meanwhile, after comparing the comprehensive diagnostic features with the three sets of typical feature vectors in the pathological feature mapping benchmark in a dimension-by-dimensional distance manner, the first total distance value, the second total distance value, and the third total distance value are selected. The level corresponding to the smallest total distance value is taken as the best matching level, and the level corresponding to the second smallest total distance value is taken as the second best matching level. When the difference between the total distance value of the best matching level and the total distance value of the second best matching level is small (e.g., less than 0.4), it is determined that there is a matching deviation between the comprehensive diagnostic features and the pathological feature mapping benchmark.

[0095] Based on the matching bias, the deviation portion of the comprehensive diagnostic features that exceeds the coverage range of the preset pathological feature mapping benchmark is identified as the cognitive uncertainty component; the fuzzy fluctuation portion of the evidence quality index caused by the comprehensive diagnostic features being at the classification boundary is identified as the accidental uncertainty component.

[0096] Specifically, if the total distance value of all three sets of typical feature vectors in the mapping benchmark between comprehensive diagnostic features and pathological features is greater than the pre-calibrated maximum tolerance distance threshold (the maximum tolerance distance threshold is 5.0 for example), it is determined that the numerical pattern of the comprehensive diagnostic features exceeds the range of typical features that the pathological feature mapping benchmark can cover. The excess part is quantified as a cognitive uncertainty component. The magnitude of the cognitive uncertainty component is proportional to the difference between the minimum value of the three total distance values ​​and the maximum tolerance distance threshold.

[0097] If the difference between the total distance value of the best matching level and the total distance value of the second best matching level is small, and the proportion of the evidence quality index corresponding to the best matching level does not exceed 50% (for example, the proportion of the evidence quality index of the best matching level is 0.45, the proportion of the evidence quality index of the second best matching level is 0.35, and the absolute difference is 0.10), it is determined that the comprehensive diagnostic feature is in the classification boundary region of the HER2 expression level. The numerical competition and uncertainty of the evidence quality index in this classification boundary region is the accidental uncertainty component. The magnitude of the accidental uncertainty component is inversely proportional to the absolute difference between the proportion of the evidence quality index of the best matching level and the proportion of the evidence quality index of the second best matching level.

[0098] It integrates three-level classification assessment of tendency, cognitive uncertainty component and accidental uncertainty component, and outputs hierarchical assessment results.

[0099] Specifically, the HER2 expression determination result is determined by selecting the tier with the largest proportion among the three-tier classification assessment tendencies (exemplary value: HER2-low). The values ​​of the cognitive uncertainty component and the random uncertainty component are appended to the HER2 expression determination result. The cognitive uncertainty component is defined as the degree to which the comprehensive diagnostic features exceed the coverage of the pathological feature mapping benchmark, and the random uncertainty component is defined as the degree of fuzzy fluctuation caused by the comprehensive diagnostic features at the classification boundary. The HER2 expression determination result, the values ​​of the cognitive uncertainty component, the values ​​of the random uncertainty component, and their respective meanings are encapsulated together to form a stratified assessment result.

[0100] By deeply attributing the degree of matching between comprehensive diagnostic features and pathological benchmarks, a "transparent" expression of diagnostic risk is achieved. This design not only promptly identifies rare cases beyond the model's knowledge base through cognitive uncertainty components, preventing blind predictions, but also accurately identifies ambiguous regions at classification boundaries through accidental uncertainty components, providing auxiliary risk coordinates for clinical decision-making in addition to the judgment conclusion. Ultimately, this improves the safety and reliability of assisted diagnosis, providing a scientific and objective quantitative basis for the subsequent development of personalized clinical pathways and the identification of targeted resampling areas.

[0101] Methods for synthesizing virtual pathological staining images aligned with bimodal image space include:

[0102] The hierarchical assessment results are correlated with the original ultrasound image and the original photoacoustic image at the pixel level to establish colorimetric constraints that include the anatomical spatial structure of the lesion in the original ultrasound image and the tissue metabolic functional state in the original photoacoustic image, thus forming a set of spatial functional constraints.

[0103] It should be noted that the HER2 expression determination results in the stratified assessment results will be used as the overall basis for color coding, with HER2-positive corresponding to brown, HER2-low corresponding to light brown, and HER2-zero corresponding to blue;

[0104] Each pixel location within the lesion boundary and the surrounding transition zone in the original ultrasound image is taken as a pixel to be stained. The relative concentration values ​​of oxyhemoglobin and deoxyhemoglobin at the same pixel location in the original photoacoustic image are used as functional factors to adjust the color intensity. The higher the relative concentration value of oxyhemoglobin, the higher the color saturation (an example value of 0.45 corresponds to 70% saturation), and the higher the relative concentration value of deoxyhemoglobin, the lower the color brightness (an example value of 0.32 corresponds to 60% brightness).

[0105] A mapping relationship is established between the spatial coordinates of each pixel, the hue corresponding to the HER2 expression level, the saturation corresponding to the relative concentration of oxyhemoglobin, and the brightness corresponding to the relative concentration of deoxyhemoglobin. The mapping relationships of all pixels together constitute a set of spatial functional constraints.

[0106] Based on the spatial functional constraint set, the three-level classification assessment tendency is converted into corresponding color codes within the multi-scale tumor microenvironment region, and the immunohistochemical staining distribution is simulated pixel by pixel to generate preliminary pathological staining images.

[0107] It should be noted that, based on the set of spatial functional constraints, within the tumor body area and the peritumoral transition area covered by the multi-scale tumor microenvironment region, each pixel is traversed, and the hue value, saturation value, and brightness value are read from the set of spatial functional constraints and merged into a composite color value.

[0108] HER2-positive pixels are merged into a brown composite color value (exemplary values: 70% saturation, 60% brightness); HER2-low pixels are merged into a light brown composite color value (exemplary values: 50% saturation, 65% brightness); and HER2-zero pixels are merged into a blue composite color value (exemplary values: 40% saturation, 70% brightness). The composite color values ​​of all pixels are arranged according to their original spatial positions to generate a preliminary pathological staining image.

[0109] Using the lesion boundary in the original ultrasound image as a reference, the edge of the color-coded distribution area in the preliminary pathological staining image is corrected, and a virtual pathological staining image aligned with the dual-modal image space is output.

[0110] It should be noted that the coordinate point sequence of the lesion boundary is extracted from the original ultrasound image as the boundary between the main tumor area and the surrounding transition area. The pixel positions where the color changes in the preliminary pathological staining image are compared point by point with the coordinate point sequence of the lesion boundary. When the color-coded transition edge appears inside the lesion boundary, it is pushed outward to coincide with the lesion boundary. When the color-coded transition edge appears outside the lesion boundary, it is pulled inward to coincide with the lesion boundary. The edge of the corrected color-coded distribution area is spatially aligned with the lesion boundary, and a virtual pathological staining image that is spatially aligned with the dual-modal image is output.

[0111] Existing imaging assessment techniques typically only provide overall pathological grading predictions for lesions, failing to effectively identify and reflect the complex heterogeneity of HER2 expression within breast cancer. Due to differences in biological characteristics at different lesion locations, relying solely on probabilistic conclusions often leads to missed diagnoses of highly invasive areas, and physicians cannot intuitively locate specific sites where predicted findings conflict with actual imaging manifestations from traditional fused images. Therefore, this invention achieves precise localization of spatial conflicts by performing fine-grained sub-region comparisons on virtual pathological images and combining this with visual enhancement of inconsistent areas. The specific steps are as follows:

[0112] The methods for outputting virtual pathology visualization images include:

[0113] The virtual pathological staining image is divided into several sub-regions. The percentage of pixels with color codes in each sub-region is counted, and the color code with the highest percentage is taken as the staining judgment category corresponding to that sub-region. The staining judgment category is compared with the category with the highest probability in the three-level classification assessment tendency, and the sub-regions with conflicting judgments are identified as inconsistent sub-regions.

[0114] It should be noted that the virtual pathological staining image is uniformly divided into a grid of fixed size, with each grid cell being a sub-region (for example, the size of each sub-region is 10 pixels × 10 pixels). Within each sub-region, the proportion of brown pixels, light brown pixels, and blue pixels to the total number of pixels in that sub-region is counted, and the color code with the highest proportion is selected as the staining classification of that sub-region (for example, if light brown pixels account for 55% in a certain sub-region, then the staining classification of that sub-region is HER2-low).

[0115] The staining classification of each sub-region is compared with the highest probability category, which has the largest proportion of evidence quality indicators in the three-level classification assessment tendency, and the sub-region is determined to be inconsistent when the staining classification of a sub-region is inconsistent with the highest probability category.

[0116] The boundary contours of inconsistent sub-regions are extracted to generate inconsistency region masks. The positions corresponding to the inconsistency region masks are visually enhanced in the virtual pathological staining image to obtain a virtual pathological staining image with spatial inconsistency markers. This image is then fused with the original ultrasound image and the original photoacoustic image to output a virtual pathological visualization image.

[0117] It should be noted that all inconsistent sub-regions are traversed, and inconsistent sub-regions that are spatially adjacent (i.e., share a boundary or vertex) are grouped into the same connected domain. Each connected domain is assigned a unique identifier number, and all sub-regions belonging to the same connected domain are merged into a whole region. Adjacent inconsistent sub-regions are merged, and the outer boundary pixel sequence of the merged connected domain is extracted as the boundary contour of the inconsistent region.

[0118] The boundary contour is filled with a marker value (255 pixels for example) inside and with a background value (0 pixels for example) outside to generate an inconsistent region mask;

[0119] In the virtual pathological staining image, locate the pixel position corresponding to the marked value in the mask of the inconsistent region, replace the color of the pixel position with a bright warning color (fluorescent yellow for example), and obtain the virtual pathological staining image with spatial inconsistency marking; the virtual pathological staining image with spatial inconsistency marking, the original ultrasound image and the original photoacoustic image are semi-transparently overlaid according to the same spatial coordinates, and the virtual pathological visualization image is output after fusion and display.

[0120] The methods for generating personalized clinical pathway recommendations and targeted resampling area recommendations include:

[0121] Based on the proportion of evidence quality indicators at each level of the three-tier classification assessment, and matching the pre-set treatment decision logic, personalized clinical pathway recommendations are generated.

[0122] It should be noted that the proportion of evidence quality indicators for the three levels of HER2-zero, HER2-low, and HER2-positive in the three-level classification assessment is read, and the size of the three is compared. If the proportion of HER2-positive is the largest, the personalized clinical pathway recommendation is to give priority to the traditional anti-HER2 treatment pathway; if the proportion of HER2-low is the largest, the personalized clinical pathway recommendation is to enter the ADC potential benefit assessment pathway; if the proportion of HER2-zero is the largest, the personalized clinical pathway recommendation is to continue the evaluation of conventional non-anti-HER2 treatment regimens.

[0123] Simultaneously read the values ​​of the cognitive uncertainty component and the random uncertainty component. When the value of the cognitive uncertainty component is greater than the value of the random uncertainty component and exceeds the pre-defined cognitive uncertainty warning line (exemplary value of 0.5), add a suggestion for dynamic re-examination in the short term to the personalized clinical pathway recommendation; when the value of the random uncertainty component is greater than the value of the cognitive uncertainty component and exceeds the pre-defined random uncertainty warning line (exemplary value of 0.5), add a suggestion to pay attention to the pathological boundary to the personalized clinical pathway recommendation.

[0124] The inconsistent region mask and the high-value region of the random uncertainty component are spatially intersected to determine the overlapping sub-regions; the boundary contours of the overlapping sub-regions are extracted and spatial coordinates are generated, and the spatial coordinates are mapped to the corresponding anatomical positions of the original ultrasound images to generate targeted resampling region recommendations.

[0125] It should be noted that in the spatial distribution map of the random uncertainty components, pixels whose random uncertainty component values ​​are higher than the pre-defined random uncertainty high threshold (exemplary value is 0.3) are selected to form the random uncertainty high value region;

[0126] Perform a pixel-by-pixel logical AND operation between the pixel region marked with a value of 255 in the inconsistent region mask and the region with a high value of accidental uncertainty. The pixel positions covered by both regions are the overlapping sub-regions.

[0127] Extract the outer boundary pixel sequence of the overlapping sub-region as the boundary contour, calculate the centroid coordinates of the boundary contour, and map the centroid coordinates to the corresponding anatomical position in the original ultrasound image after converting them according to the spatial resolution of the original ultrasound image (exemplary values ​​are 3.5cm horizontally and 2.8cm vertically from the nipple). Mark the anatomical position with a bright dashed circle on the original ultrasound image to generate a targeted resampling region recommendation.

[0128] By cross-validating the "logical discrimination" of classification probabilities with the "spatial distribution" of virtual staining, a conflict-driven secondary verification mechanism was constructed. This design can keenly capture key areas within lesions that present conflicting and highly uncertain judgments, transforming abstract algorithmic biases into physically meaningful anatomical coordinates. This guides clinicians to quickly locate the most pathologically representative sampling points in the complex tumor microenvironment. It effectively reduces the misdiagnosis rate caused by inaccurate sampling, improves the accuracy of heterogeneous lesion identification, and provides clinicians with a highly practical targeted resampling guidance scheme.

[0129] The methods for generating multi-time-point dynamic evaluation reports include:

[0130] Spatial registration is performed on the targeted resampling region recommendations at the current time point and at historical time points, and personalized clinical pathway recommendations at different time points are compared to identify recommendations for adjustment records.

[0131] It should be noted that the recommended targeted resampling regions for the current time point are extracted from the structured assessment report at the current time point. The recommended targeted resampling regions include the centroid coordinates (exemplary values ​​are 3.5 cm horizontally and 2.8 cm vertically from the nipple) and boundary contours of the overlapping sub-regions at the current time point; the recommended targeted resampling regions for historical time points are extracted from the structured assessment report at historical time points. The recommended targeted resampling regions for historical time points include the centroid coordinates (exemplary values ​​are 3.2 cm horizontally and 2.6 cm vertically from the nipple) and boundary contours of the overlapping sub-regions at historical time points.

[0132] Using the nipple position and the chest wall fascia line as registration reference points, the translation of the nipple reference point and the rotation angle of the chest wall fascia line reference line between the historical time point and the current time point are calculated. The spatial coordinates of the recommended targeted resampling area at the historical time point are translated and rotated according to the translation and rotation angle, so that the coordinates of the historical time point are mapped to the original ultrasound image coordinate system at the current time point, so that the recommended targeted resampling areas at the two time points are under the same spatial reference system.

[0133] Personalized clinical pathway recommendations for the current time point are extracted from the structured assessment report at the current time point, and personalized clinical pathway recommendations for the historical time point are extracted from the structured assessment report at the historical time point. The personalized clinical pathway recommendations for the current time point and the personalized clinical pathway recommendations for the historical time point are compared word by word. When the two contents are inconsistent (for example, the historical time point recommendation is to enter the potential benefit assessment path of ADC, and the current time point recommendation is to prioritize the traditional anti-HER2 treatment path), the content of the recommendation adjustment, the direction of the adjustment, and the time point when the adjustment occurs are recorded to form a recommendation adjustment record.

[0134] The overlap and position offset vectors of the target resampling areas recommended at different time points after registration are extracted, and the evolution trend is identified by combining the numerical changes of the cognitive uncertainty component and the accidental uncertainty component; the suggested adjustment records and evolution trend are integrated to output a multi-time point dynamic evaluation report.

[0135] It should be noted that, under the same coordinate system after registration, the overlap area of ​​the recommended boundary contour of the target resampling area at the current time point and the historical time point is calculated as the ratio of the area of ​​the overlap to the area of ​​the union (65% for example), and the Euclidean distance between the centroid coordinates of the two time points is calculated as the position offset vector (4mm for example).

[0136] The cognitive uncertainty component and the random uncertainty component are extracted from the stratified assessment results at two time points, and the difference in change is calculated. When the degree of overlap decreases, the positional offset vector increases, and the value of the cognitive uncertainty component increases, it is identified as an uncertainty expansion trend. When the degree of overlap increases, the positional offset vector decreases, and the value of the random uncertainty component decreases, it is identified as an uncertainty decay trend. When the degree of overlap and the positional offset vector remain stable and the values ​​of the two uncertainty components fluctuate little (for example, the fluctuation range of the cognitive uncertainty component value is less than 0.1 and the fluctuation range of the random uncertainty component value is less than 0.1), it is identified as an uncertainty stabilization trend.

[0137] The proposed adjustments, including the content, direction, and timing, along with the identified types of uncertainty evolution trends, overlap values, positional offset vector values, differences in cognitive uncertainty components, and differences in accidental uncertainty components, are integrated into a single report in chronological order to form a multi-time-point dynamic assessment report.

[0138] In summary, this invention achieves refined measurement and attribution of diagnostic reliability by performing dual uncertainty decomposition, providing a transparent benchmark for clinical decision-making to distinguish between knowledge limitations and data ambiguities, thereby ensuring the safety of stratified assessment. Furthermore, by performing spatial inconsistency labeling and region intersection, it achieves precise spatial localization and visual guidance of heterogeneous conflict points within lesions, providing deterministic physical coordinate references for targeted resampling. Synergistically, it transforms "black-box" image assessment into "interpretable, localizable, and verifiable" precision medical decision-making, improving the accuracy and clinical operability of HER2 expression status assessment in non-invasive dynamic monitoring.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dual-modal dynamic assessment system for HER2 status in breast cancer, characterized in that, include: The acquisition module obtains dual-modal images of the patient's clinical information and breast lesions, and constructs a multi-scale tumor microenvironment region to extract multi-dimensional features of the tissue's inherent biological state. The analysis module integrates multidimensional features of biological status with clinical information to perform a three-level classification assessment and dual uncertainty decomposition of HER2 expression status in breast cancer, and outputs stratified assessment results. The mapping module uses the hierarchical evaluation results to synthesize virtual pathological staining images that are spatially aligned with the bimodal image and marks them for spatial inconsistency, outputting virtual pathological visualization images. The decision-making module combines virtual pathology visualization images with stratified assessment results to generate personalized clinical pathway suggestions and targeted resampling area recommendations. The tracking module compares personalized clinical pathway recommendations with targeted resampling area recommendations at different time points, identifies evolution trends, and outputs multi-time-point dynamic assessment reports.

2. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 1, characterized in that, The methods for obtaining dual-modal images of the subject's clinical information and breast lesions include: The clinical information of the subjects is obtained, and the non-numerical variables in the clinical information are mapped into multi-dimensional one-hot encoded vectors to obtain the clinical information features. Simultaneously acquire raw ultrasound images, raw photoacoustic images, physiological signals, and corresponding acquisition parameter files of the breast lesion area, and combine them with clinical information features to form a dual-modal image data package.

3. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 2, characterized in that, The methods for constructing multi-scale tumor microenvironment regions include: The original ultrasound images in the dual-modal image data package are used to identify lesion boundaries and extract the section with the maximum diameter feature as the target layer. The region inside the lesion boundary at the target level is constructed as the main tumor region, generating a main tumor region mask; Based on the edge of the tumor main area mask, expand outward at equal intervals, and construct the area outside the tumor main area formed by the expansion as the peritumor transition zone, thus generating the peritumor transition zone mask; In the target layer, a normal tissue area located outside the mask of the peritumor transition zone is selected to construct a background reference area, and a background reference area mask is generated. By integrating the tumor main region mask, the surrounding tumor transition region mask, and the background reference region mask, a multi-scale tumor microenvironment region is obtained.

4. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 3, characterized in that, The methods for extracting multidimensional features of the inherent biological state of tissues include: Based on the multi-scale tumor microenvironment region, ultrasound features are extracted from the original ultrasound image, and photoacoustic features are extracted from the original photoacoustic image, and combined to obtain the initial dual-modal image information; By using the acquired parameter files and physiological signals, signal variation components introduced by differences in equipment parameters and physiological fluctuations are separated from the initial bimodal image information to obtain interference-free bimodal image information; In the de-interference dual-modal imaging information, the functional and structural states of lesions and microenvironment tissues are used as multidimensional features of the tissue's inherent biological state.

5. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 4, characterized in that, The three-tiered classification assessment method for HER2 expression status in breast cancer includes: The multidimensional features of biological states and clinical information features are structurally aligned according to a preset coordinate system to obtain a multimodal joint representation set; Based on the assessment needs in the clinical information features, the contribution ratio of imaging components and clinical components in the multimodal joint representation set is dynamically adjusted and fused to output comprehensive diagnostic features; The comprehensive diagnostic features are mapped to the preset pathological feature mapping benchmark to generate evidence quality indicators supporting the three levels of HER2-zero, HER2-low, and HER2-positive. The HER2 expression level of the breast lesion is determined based on the proportion of evidence quality indicators, and the three-level classification assessment tendency is output.

6. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 5, characterized in that, The decomposition methods for the dual uncertainty include: Extract the distribution of evidence quality indicators across different levels and identify the matching deviation between comprehensive diagnostic features and the pre-defined pathological feature mapping benchmark; Based on the matching bias, the deviation portion of the comprehensive diagnostic features that exceeds the coverage range of the preset pathological feature mapping benchmark is identified as the cognitive uncertainty component; the fuzzy fluctuation portion of the evidence quality index caused by the comprehensive diagnostic features being at the classification boundary is identified as the accidental uncertainty component. It integrates three-level classification assessment of tendency, cognitive uncertainty component and accidental uncertainty component, and outputs hierarchical assessment results.

7. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 6, characterized in that, The method of synthesizing virtual pathological staining images aligned with the bimodal image space includes: The hierarchical evaluation results are correlated with the original ultrasound image and the original photoacoustic image at the pixel level to establish colorimetric constraints that include the anatomical spatial structure of the lesion in the original ultrasound image and the tissue metabolic functional state in the original photoacoustic image, forming a set of spatial functional constraints. Based on the spatial functional constraint set, the three-level classification assessment tendency is converted into corresponding color codes in the multi-scale tumor microenvironment region, and the immunohistochemical staining distribution is simulated pixel by pixel to generate preliminary pathological staining images. Using the lesion boundary in the original ultrasound image as a reference, the edge of the color-coded distribution area in the preliminary pathological staining image is corrected, and a virtual pathological staining image aligned with the dual-modal image space is output.

8. The dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 7, characterized in that, The methods for outputting virtual pathology visualization images include: The virtual pathological staining image is divided into several sub-regions. The pixel proportion of color coding in each sub-region is counted, and the color coding with the highest proportion is taken as the staining judgment category corresponding to the sub-region. The staining judgment category is compared with the highest probability category in the three-level classification assessment, and the sub-regions where the two judgments conflict are identified as inconsistent sub-regions. The boundary contours of inconsistent sub-regions are extracted to generate inconsistency region masks. The positions corresponding to the inconsistency region masks are visually enhanced in the virtual pathological staining image to obtain a virtual pathological staining image with spatial inconsistency markers. This image is then fused with the original ultrasound image and the original photoacoustic image to output a virtual pathological visualization image.

9. A dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 8, characterized in that, The methods for generating personalized clinical pathway suggestions and targeted resampling region recommendations include: Based on the proportion of evidence quality indicators at each level of the three-level classification assessment, and matching the preset treatment decision logic, personalized clinical pathway suggestions are generated. The inconsistent region mask and the high-value region of the random uncertainty component are spatially intersected to determine the overlapping sub-regions; the boundary contours of the overlapping sub-regions are extracted and spatial coordinates are generated, and the spatial coordinates are mapped to the corresponding anatomical positions of the original ultrasound images to generate targeted resampling region recommendations.

10. A dual-modal dynamic assessment system for HER2 status in breast cancer as described in claim 9, characterized in that, The methods for outputting multi-time-point dynamic evaluation reports include: Spatial registration is performed on the targeted resampling area recommendations at the current time point and at historical time points, and personalized clinical pathway recommendations at different time points are compared to identify recommendations for adjustment records; The overlap and position offset vectors of the target resampling areas recommended at different time points after registration are extracted, and the evolution trend is identified by combining the numerical changes of the cognitive uncertainty component and the accidental uncertainty component; the suggested adjustment records and evolution trend are integrated to output a multi-time point dynamic evaluation report.

Images