Multi-modal fusion anorectal data visualization analysis method and system

Abstract

The invention relates to the technical field of data visualization, in particular to a multi-modal fusion anorectal data visualization analysis method and system, and the method comprises the following steps: synchronously collecting an anorectal image and a pressure signal, aligning a timestamp, carrying out the preprocessing, outputting a standardized image and a coded physiological signal, and extracting edge and texture features. After LSTM coding, splicing with image features, and fusing into a joint feature map; generating a multi-scale heat map by the feature pyramid network, and coarsely positioning a lesion; after boundary optimization, U-Net recovers details, and a pixel-level lesion probability graph is output; superposing the Jet color gradation to the original image in a semitransparent manner, and marking a pressure abnormal time period; and performing weighted scoring to generate a fourth-level clinical suggestion. According to the anorectal disease diagnosis method, through multi-modal space-time alignment, multi-scale feature fusion, boundary optimization and a quantitative scoring system, the problems of data splitting, insufficient precision and low efficiency of a traditional method are solved, a high-precision and high-robustness intelligent diagnosis tool is provided for anorectal diseases, and the scientificity and efficiency of clinical decision making are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data visualization technology, and in particular to a multimodal fusion anorectal data visualization analysis method and system. Background Art

[0002] In the diagnosis and treatment of anorectal diseases, accurate and timely diagnosis plays a crucial role in patient prognosis. Clinical diagnosis relies heavily on the comprehensive analysis of endoscopic images and physiological signals (especially anal canal pressure). Unfortunately, traditional diagnostic methods have numerous insurmountable limitations, severely restricting diagnostic accuracy and efficiency.

[0003] First, traditional models suffer from a single-scale feature extraction dilemma. They often rely solely on features at a fixed resolution, which makes them inadequate for real-world conditions. For one thing, the model struggles to discern subtle changes in extremely small lesions, such as those less than 3 mm, due to resolution limitations, making it difficult for the model to discern these subtle changes, leading to a high risk of missed detections. Furthermore, when encountering large-scale abnormalities, such as tumor infiltration, which affects a wide range of tissue, a single resolution cannot fully capture the entire lesion, resulting in a high false positive rate. This is like using a camera with a fixed focal length to capture scenes of varying distances and sizes: either nearby details are blurred, or key information is missing from the distant panorama.

[0004] Secondly, the traditional diagnostic process is characterized by excessive reliance on manual labor and subjectivity. Doctors are required to manually annotate the affected areas and correlate these areas with physiological signals. This process not only consumes considerable time and effort, resulting in extremely low diagnostic efficiency, but also inevitably introduces subjective bias due to differences in experience and judgment standards among different doctors. More critically, the industry lacks a standardized, quantitative evaluation system, resulting in significant fluctuations in the accuracy and reliability of diagnostic results across different medical institutions and doctors, making it difficult to establish a unified, precise standard.

[0005] Furthermore, traditional methods struggle to handle complex anatomical structures, especially those with specialized structures like intestinal folds. Fixed convolution kernels act like rigid molds, unable to flexibly adapt to the complex and changing morphology of the intestine. This leads to deviations in the positioning of lesion boundaries, making it difficult to accurately determine the extent of the lesion. Furthermore, during the decoding of images, detail can be lost. For example, subtle textures on the surface of erosions, which contain crucial diagnostic information, often become blurred or even disappear completely with traditional methods. This undoubtedly has a significant negative impact on diagnostic accuracy, making it difficult for doctors to make the most accurate judgments based on images.

[0006] In view of this, a multimodal fusion anorectal data visualization analysis method and system are proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a multimodal fusion anorectal data visualization analysis method and system to solve the problem that traditional methods cannot provide refined models and standardized diagnoses.

[0008] To solve the above technical problems, the present invention provides a multimodal fusion anorectal data visualization analysis method, comprising the following steps: S1. Synchronously collect anorectal images and physiological signals, preprocess the collected images and physiological signals, output aligned multimodal data, and record them as original images and encoded physiological signals respectively; S2. Preprocess the original image and convert it into a spatially aligned standardized image based on the affine transformation principle of anatomical landmarks; S3. Use the network model to perform edge extraction and texture extraction on the standardized image to obtain edge feature maps and texture feature maps. Cache the output of the first convolutional layer of the network model as a shallow intermediate feature map. Perform LSTM encoding on the physiological signal encoded in S1 to generate a physiological signal vector. Broadcast the physiological signal vector, edge feature map, and texture feature map to the same spatial dimension, then splice the channels and compress them into a joint feature map through convolution. S4 generates multi-scale candidate regions through the feature pyramid network model, thereby filtering the joint feature map to obtain a coarse-grained heat map. After sampling the coarse-grained heat map, it is channel-joined with the edge feature map of S3 and input into the deformable convolutional network model to optimize the boundary and generate a refined mask. The refined mask is input into the U-Net decoder and fused with the shallow intermediate feature map of S3 through jump connections to obtain a pixel-level anomaly probability heat map. S5: Use Jet color scale mapping to semi-transparently overlay the pixel-level abnormality probability heat map and the original image of S1 as a visualization of the lesion area, and mark the abnormal time period of the physiological signal in S1. Based on the abnormal probability of the lesion area visualization map and the abnormal degree of the physiological signal, calculate the weighted score and map it to the clinical recommendation grade.

[0009] As a further improvement of the present technical solution, in S1, the anorectal image is an anorectal endoscope image, and the physiological signal is an anal canal pressure signal collected in real time by a pressure sensor; the preprocessing is to align the image frame and the physiological signal using timestamp technology, and then extract the mean, variance, and peak features of the physiological signal through sliding window technology.

[0010] As a further improvement of this technical solution, the transfer station sorting rule is to give priority to the printer that has been idle the longest after the task is completed. If the printer is in the state of receiving files or working, it is allocated from short to long according to the current task printer processing time.

[0011] As a further improvement of the present technical solution, in S2, the preprocessing of the original image includes denoising the original image through Gaussian filtering and contrast enhancement of the denoised original image through CLAHE technology; then the enhanced original image is subjected to an affine transformation based on anatomical landmarks to unify the resolution and output as a standardized image.

[0012] As a further improvement of this technical solution, in S3, the network model includes an edge extraction network model, a texture extraction network model and an LSTM encoder. The edge extraction network model includes the first convolution layer and the second convolution layer. The normalized image is output as a shallow intermediate feature map through the first convolution layer, and the shallow intermediate feature map is output as an edge feature map through the second convolution layer. The texture extraction network model includes a Gabor filter bank and an SE attention module. The normalized image is passed through the Gabor filter bank to output a multi-scale texture response map, which is then output as a texture feature map through the SE attention module. The LSTM encoder re-encodes the physiological signal encoded in S1 to generate a physiological signal vector, and broadcasts the generated physiological signal vector using the broadcast principle until the physiological signal vector and the edge feature map and texture feature map are broadcast to the same spatial dimension.

[0013] As a further improvement of this technical solution, in S4, the feature pyramid network model is constructed based on the feature map output by the backbone network model. The input standardized image is gradually extracted from the image features through the five stages of the backbone network model. Each layer outputs a feature map corresponding to a different semantic level, specifically: The standardized image outputs a C1 feature map containing edge or texture distribution through convolution and pooling of the first stage layer. The C1 feature map outputs a C2 feature map containing organ contours through convolution of the second stage layer. The C2 feature map outputs a C3 feature map containing lesion areas through convolution of the third stage layer. The C3 feature map outputs a C4 feature map containing tissue abnormalities or texture variations through convolution of the fourth stage layer. The C5 feature map outputs a C5 feature map containing global context information through convolution of the fifth stage layer. The C3 feature map, C4 feature map, and C5 feature map are input into the feature pyramid network model for sampling, horizontal connection, and feature fusion to generate the final feature pyramid level, specifically: The feature C5 map is output through convolution to detect the P5 feature map of large targets; the P5 feature map is sampled through convolution to output the P6 feature map for auxiliary detection of oversized or blurred targets; the P6 feature map is sampled through convolution to output the P7 feature map of extreme scale; the feature C4 map is aligned through the convolution channel and then added element-by-element with the bilinear sampling result of the P5 feature map to output the P4 feature map detected as a medium target; the feature C3 map is aligned through the convolution channel and then added element-by-element with the bilinear sampling result of P4 to output the P3 feature map detected as a small target.

[0014] As a further improvement to this technical solution, after the P3 feature map is convolved, the output probability map is calculated through the activation function. The probability map is binarized, and a confidence probability threshold α is preset. Only areas with a confidence probability higher than α are retained to generate a coarse-grained heat map. The coarse-grained heat map is bilinearly sampled to the same resolution as the edge feature map of S3, and then spliced ​​with the edge feature map of S3 along the channel dimension. After fusion, the boundary is optimized by the deformable convolutional network model to obtain a refined mask with a boundary that fits the actual anatomical structure. The refined mask is input into the U-Net decoder, and the shallow intermediate feature map output by the first layer of the edge extraction network model in S3 is fused through jump connections. The pixel-level abnormality probability heat map is obtained by bilinear interpolation technology. In the pixel-level abnormality probability heat map, each pixel is marked with a lesion probability value between 0 and 1. The closer the probability value is to 0, the smaller the lesion probability of the pixel, and the closer the probability value is to 1, the greater the lesion probability of the pixel.

[0015] As a further improvement of the present technical solution, in S5, the Jet color scale mapping maps the probability of lesion labeling for each pixel in the pixel-level abnormality probability heat map to the corresponding color, wherein a probability value of 0 corresponds to dark blue, and a probability value of 1 corresponds to red; when semi-transparently superimposed, the original image of S1 and the pixel-level abnormality probability heat map are mixed according to the transparency β, where β is a preset transparency.

[0016] As a further improvement of the present technical solution, in S5, the method for marking the abnormal period of the physiological signal is as follows: input the timestamp corresponding to the abnormal area in the physiological signal and the pixel-level abnormality probability heat map synchronized in S1, map the abnormal area in the pixel-level abnormality probability heat map to the synchronized physiological signal time axis according to the timestamp, the abnormal area is the area with a preset threshold value δ of the probability value in the pixel-level abnormality probability heat map, and mark the period on the signal waveform diagram of the physiological signal that is greater than the preset pressure threshold γ with red shadow, and obtain the physiological signal waveform diagram marked by red shadow.

[0017] As a further improvement of the present technical solution, in S5, a weighted score is calculated based on the abnormal probability of the lesion area visualization image and the abnormal degree of the physiological signal, specifically: S = A × number of high-probability pixels ÷ total number of pixels × 100 + B × peak value of the physiological signal ÷ preset pressure threshold γ × 100, where A and B are preset weights, S is the weighted score, and the number of high-probability pixels is the number of pixels with a pixel-labeled lesion probability value greater than 0.9; The clinical recommendation is divided into four levels. The weighted score range of level one is 0 to 40, and the clinical recommendation is regular follow-up without intervention; the weighted score range of level two is 40 to 70, and the clinical recommendation is drug treatment and review after 3 months; the weighted score range of level three is 70 to 90, and the clinical recommendation is endoscopic biopsy to rule out malignant lesions; the weighted score range of level four is 90 to 100, and the clinical recommendation is immediate surgical intervention and pathological confirmation.

[0018] A multimodal fusion anorectal data visualization analysis system, which is used to implement the above-mentioned multimodal fusion anorectal data visualization analysis method, is characterized by comprising: A signal acquisition module, which is used to synchronously acquire anorectal images and physiological signals, preprocess the acquired images and physiological signals, output aligned multimodal data, and record them as original images and encoded physiological signals respectively; An image processing module, configured to pre-process the original image and process the original image into a spatially aligned standardized image based on the affine transformation principle of anatomical landmarks; A feature extraction module, comprising an edge extraction network model, a texture extraction network model, and an LSTM encoder, wherein the edge extraction network model is used to extract edges from the standardized image and output a shallow intermediate feature map and an edge feature map; the texture extraction network model is used to extract texture from the standardized image and output a texture feature map; the LSTM encoder is used to re-encode the encoded physiological signal to generate a physiological signal vector, and broadcast the generated physiological signal vector using the broadcast principle until the physiological signal vector and the edge feature map and the texture feature map are broadcast to the same spatial dimension and compressed into a joint feature map through convolution; Anomaly probability heat map output module, which includes a feature pyramid network model, a deformable convolutional network model and a U-Net decoder, wherein the feature pyramid network model is used to filter the joint feature map to obtain a coarse-grained heat map, and after sampling the coarse-grained heat map, channel-join the coarse-grained heat map with the edge feature map of S3; the deformable convolutional network model is used to optimize the boundary of the coarse-grained heat map after channel-joining with the edge feature map of S3 to obtain a refined mask; the U-Net decoder is used to fuse the refined mask with the shallow intermediate feature map of S3 through a jump connection to obtain a pixel-level anomaly probability heat map; The result output module uses Jet color scale mapping to semi-transparently overlay the pixel-level abnormality probability heat map and the original image of S1 as a visualization of the lesion area, and marks the abnormal period of the physiological signal in S1. Based on the abnormal probability of the lesion area visualization map and the abnormal degree of the physiological signal, a weighted score is calculated, mapped to the clinical recommendation grade, and a graded diagnosis report is generated.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. This multimodal fusion anorectal data visualization analysis method and system uses a feature pyramid network to integrate the high-resolution details of the C3 feature map, the moderate-scale information of the C4 feature map, and the global semantic connotation of the C5 feature map to build a comprehensive lesion detection system, achieving comprehensive coverage of lesions of different sizes and spans.

[0020] 2. This multimodal fusion anorectal data visualization analysis method and system uses a deformable convolutional network to adjust the convolution kernel sampling points according to complex and irregular anatomical structures such as intestinal folds and vascular bifurcations, making them fit perfectly, greatly improving the accuracy of lesion range definition.

[0021] 3. This multimodal fusion anorectal data visualization analysis method and system uses pixel-level probability heat maps to precisely quantify lesion risks with continuous probability values ​​(0-1). This not only accurately supports the calculation of lesion area, but also provides a strong basis for personalized threshold setting. For example, it can accurately set the surgical threshold to 0.8 based on the condition of the patient, providing a key quantitative reference for clinical treatment decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all secondary embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0024] Currently, the clinical diagnosis of anorectal diseases (such as hemorrhoids, anal fissures, and tumors) relies heavily on the combined analysis of endoscopic images and physiological signals (such as anal canal pressure). However, traditional methods are limited by single-scale features, manual reliance and subjectivity during diagnosis, and blurred boundaries and loss of detail during model development, which affect diagnostic accuracy. For this reason, see Figure 1 As shown, one of the purposes of the present invention is to provide a multimodal fusion anorectal data visualization analysis method, the multimodal fusion anorectal data visualization analysis method comprising the following steps: S1. Synchronously collect anorectal images and physiological signals, preprocess the collected images and physiological signals, output aligned multimodal data, and record them as original images and encoded physiological signals respectively; S2. Preprocess the original image and transform it into a spatially aligned standardized image based on the affine transformation principle of anatomical landmarks to eliminate deformation interference introduced by instrument operation; S3. Use the network model to perform edge extraction and texture extraction on the standardized image to obtain edge feature maps and texture feature maps. Cache the output of the first convolutional layer of the network model as a shallow intermediate feature map. Perform LSTM encoding on the physiological signal encoded in S1 to generate a physiological signal vector. Broadcast the physiological signal vector, edge feature map, and texture feature map to the same spatial dimension, then splice the channels and compress them into a joint feature map through convolution. S4 generates multi-scale candidate regions through the feature pyramid network model, thereby filtering the joint feature map to obtain a coarse-grained heat map. After sampling the coarse-grained heat map, it is channel-joined with the edge feature map of S3 and input into the deformable convolutional network model to optimize the boundary and generate a refined mask. The refined mask is input into the U-Net decoder and fused with the shallow intermediate feature map of S3 through jump connections to obtain a pixel-level anomaly probability heat map. S5: Use Jet color scale mapping to semi-transparently overlay the pixel-level abnormality probability heat map and the original image of S1 as a visualization of the lesion area, and mark the abnormal time period of the physiological signal in S1. Based on the probability of the lesion area map and the abnormality degree of the physiological signal, calculate the weighted score and map it to the clinical recommendation grade.

[0025] In this multimodal fusion anorectal data visualization analysis method and system, a feature pyramid network is used to integrate the high-resolution details of the C3 feature map, the moderate-scale information of the C4 feature map, and the global semantic connotation of the C5 feature map to build a comprehensive lesion detection system, achieving no-dead-angle coverage of lesions of different sizes and spans; through the deformable convolutional network, the convolution kernel sampling points can be adjusted according to complex and irregular anatomical structures such as intestinal folds and vascular bifurcations to make them fit perfectly, greatly improving the accuracy of lesion range definition; through pixel-level probability heat maps, the lesion risk is finely quantified with continuous probability values ​​(0-1), which can not only accurately support the accurate calculation of lesion area, but also provide a strong basis for personalized threshold setting, such as accurately setting the surgical threshold to 0.8 based on the condition of the disease, providing a key quantitative reference for clinical treatment decision-making.

[0026] Since anorectal endoscope images and anal canal pressure signals are usually collected by different devices, there may be a time deviation between the two. Therefore, in step S1, the anorectal image is the anorectal endoscope image, and the physiological signal is the anal canal pressure signal collected in real time by the pressure sensor; the preprocessing is to align the image frame and the physiological signal using a timestamp to ensure that each image frame corresponds to the physiological signal at the same moment, avoiding misassociation caused by time deviation (such as incorrectly matching the high-pressure signal to the normal image), and providing a basis for spatiotemporal alignment for subsequent multimodal fusion. For example, the mucosal pressure area can be located in the image at the peak pressure moment; since the original anal canal pressure signal is continuous Directly inputting time series data into the model will result in excessive computational complexity and difficulty in extracting key patterns. Doctors usually focus on the statistical characteristics of the pressure signal (such as mean, fluctuation, and peak) rather than the details of the original waveform. Therefore, the continuous signal is divided into windows of fixed duration, and the following features are calculated for each window: the mean value that reflects the average pressure level, determines whether the baseline pressure is abnormal, measures the amplitude of pressure fluctuations, identifies the variance of unstable contractions (such as spasms), and captures the peak of abnormal high-pressure events (such as a sudden increase in pressure during defecation). If the variance of a window increases significantly, it may indicate anal canal muscle spasm; peak values ​​that continue to exceed the threshold may be associated with hemorrhoids or anal fissures.

[0027] Since endoscopic images are easily affected by factors such as electronic noise of the equipment, uneven illumination, and reflection of the mucosal surface, granular or speckle noise may appear in the image. At the same time, endoscopic images often have low local contrast due to reflection of the mucosal surface, secretion coverage, or uneven illumination, making it difficult to distinguish between lesion areas (such as ulcers and erosions) and normal tissues. Therefore, in step S2, the original image is preprocessed, including denoising the original image through Gaussian filtering, contrast enhancement of the denoised original image through CLAHE technology, reducing noise interference through Gaussian filtering, so that subsequent edge and texture features are more focused on the real anatomical structure, and using CLAHE technology to highlight the texture differences of the mucosal surface (such as the rough surface of the ulcer and the smooth surface of the normal tissue), providing highly discriminative input for the subsequent texture extraction network. Since the movement of the endoscope probe or the change of the patient's body position will cause the position, angle and scaling of the same anatomical structure in different images to be inconsistent, the misaligned images will interfere with the model's positioning and quantification of the lesion (such as misjudgment of the lesion size in images at different angles). Therefore, the enhanced original image is transformed based on the affine transformation of the anatomical landmarks, and the dentate line and anal sinus are selected as anatomical landmarks. The reason is that the dentate line is the anatomical dividing line between the anal canal and the rectum, and its position is fixed and easy to identify. The anal sinus is a concave area of ​​the mucosal folds and can be used as a reference point for spatial alignment. The key points such as the dentate line and anal sinus are located through algorithms or manual annotation, and the images are translated, rotated, and scaled to align the anatomical landmarks in all images. Align to a standard coordinate system to correct deformation caused by probe movement, ensure spatial consistency of different images, and unify the size and position description of the lesion area (for example, the quantitative calculation of ulcer diameter is more accurate). Since the subsequent network model requires a fixed input size, unified resolution can avoid performance fluctuations caused by size differences. Unified resolution and output as standardized images, a resolution of 512×512 is preferred here. The reason is that high resolution (such as 1024×1024) contains more details but has a high computational cost, while low resolution (such as 256×256) will lose information about tiny lesions. 512×512 is a common compromise choice in medical image processing, taking into account both computational efficiency and information retention.

[0028] Since anorectal lesions (such as hemorrhoids and anal fissures) are often accompanied by mucosal fold deformation or vascular abnormalities, edge detection can locate these structural changes. Endoscopic images have uneven illumination and motion blur, and hierarchical convolution can gradually enhance effective edges and suppress noise. Therefore, the edge extraction network model includes a first convolution layer (512×512×16) and a second convolution layer (512×512×64). The standardized image is output as a shallow intermediate feature map through the first convolution layer. The shallow features retain high-resolution edge details (such as tiny mucosal folds). The shallow intermediate feature map is output as an edge feature map through the second convolution layer. The deep features of the edge feature map are fused with multi-scale edge responses (such as vascular bifurcations and lesion boundaries). The edge extraction network model can clearly distinguish normal tissue from lesion edges (such as ulcer contours and healthy mucosal boundaries). The lightweight convolution layer avoids excessive downsampling and retains spatial details. Lesions (such as erosions and polyps) often exhibit variations in texture roughness and granularity. The texture scales of different lesions vary greatly, such as from micro-ulcers to extensive inflammation. Therefore, the texture extraction network model includes a Gabor filter bank (4 directions × 3 scales) and an SE attention module. The four directions of the Gabor filter bank (4 directions × 3 scales) are 0°, 45°, 90°, and 135°, covering the main directions of mucosal wrinkles. The three scales are small, medium, and large kernels, capturing local granularity (small kernels) and global roughness (large kernels). The Gabor filter bank simulates the physician's visual sensitivity to texture and enhances model interpretability. The SE attention module performs channel-wise weighting on multi-scale texture features, focusing on pathologically significant areas (such as the ulcer surface). The SE module suppresses irrelevant background (such as reflective areas) and improves the signal-to-noise ratio of texture features. The standardized image is passed through the Gabor filter bank to output a multi-scale texture response map, which is then output as a texture feature map through the SE attention module. Because anal canal pressure signals exhibit temporal continuity (e.g., contraction-relaxation cycles), the time-series signal must be fused with image features to achieve cross-modal association. Therefore, an LSTM encoder (32-dimensional vector) was employed. The 32-dimensional vector balances information retention with computational efficiency and avoids high-dimensional signal redundancy. The LSTM encoder re-encodes the physiological signal encoded in S1, compressing the time-series signal into a compact vector while retaining key patterns (e.g., pressure peak frequency). This generates a physiological signal vector, which is then copied to the image spatial dimension and broadcast to generate a feature map (512 × 512 × 32) for multimodal alignment. Furthermore, periods of abnormal pressure can be dynamically associated with areas of mucosal deformation in the image (e.g., high pressure corresponds to vascular dilation). Through the collaborative design of an edge extraction network (morphological localization), a texture extraction network (surface analysis), and an LSTM encoder (functional association), this solution achieves joint modeling of morphology, texture, and function, avoiding the limitations of a single modality. The Gabor filter and edge features align with physicians' visual diagnostic logic, enhancing the credibility of the results. Precise alignment of pressure signals with image changes supports analysis of pathological mechanisms.

[0029] Since anorectal lesions vary significantly in size and morphology (e.g., small ulcers versus extensive tumors), it is difficult for a single scale feature to fully cover them. Therefore, in step S4, the feature pyramid network model is constructed based on the feature map output by the backbone network model. The input standardized image is gradually extracted from the image features through the five stages of the backbone network model. Each layer outputs a feature map corresponding to a different semantic level, specifically: The standardized image (512×512×3) outputs a C1 feature map (128×128×64) containing edge or texture distribution through convolution and pooling of the first stage layer, thereby extracting low-level features such as edges and textures, similar to a "sketch draft". The C1 feature map outputs a C2 feature map (64×64×256) containing organ contours through convolution of the second stage layer, thereby capturing organ contours, similar to "outlining organ boundaries". The C2 feature map outputs a C3 feature map (32×32×512) containing the lesion area through convolution of the third stage layer, while high resolution retains details and is suitable for locating small lesions (such as ulcer points <5mm). The C3 feature map outputs a C4 feature map (16×16×1024) containing tissue abnormalities or texture variations through convolution of the fourth stage layer, while medium resolution captures tissue abnormalities (such as texture changes in inflammatory areas). The C5 feature map outputs a C5 feature map (8×8×2048) containing global context information through convolution of the fifth stage layer, while low resolution contains global semantics and identifies large-scale lesions (such as the overall distribution of tumors). Since the backbone network is suitable for image classification and simple detection, it is insufficient for multi-scale feature extraction of complex lesions, especially for small target detection, there will be a problem of losing details. The feature pyramid network upsamples layer by layer through a top-down path, combines low-level details with high-level semantics, and generates multi-scale feature maps. Therefore, the C3 feature map, C4 feature map, and C5 feature map are input into the feature pyramid network model for sampling, horizontal connection, and feature fusion to generate the final feature pyramid level, specifically: The feature C5 map is aligned with the number of channels of C5 (2048→256) through 1×1 convolution to simplify the calculation and retain the global semantics, generating the P5 feature map; the P6 feature map and the P7 feature map are gradually reduced in resolution (P5→P6→P7) through convolution with a step size of 2 to cover extremely large targets or blurred lesions (such as diffuse inflammation); the feature C4 map is convolved with 1×1 and then added element-by-element with the bilinear upsampling result of the P5 feature map to output the P4 feature map detected as a medium target; the feature C3 map is convolved with 1×1 and then added element-by-element with the bilinear upsampling result of P4 to output the P3 feature map detected as a small target. The high resolution of the P3 feature map (32×32) captures the sticky The P4 feature map (16×16) accurately identifies small membrane abnormalities (such as early cancer), and the P5 to P7 feature maps use global features to identify the range of tumor infiltration. Through this top-down path, the high-level features (low resolution, high semantics) of the P4 to P7 feature maps are transferred layer by layer to the P3 feature map through upsampling, and fused with the shallow features (high resolution, low semantics) of the P3 feature map. The deep semantic information (such as "this is a tumor area") of the P4 to P7 feature maps is injected into the P3 feature map, so that the P3 feature map can not only capture details (such as edges), but also understand its semantic meaning (such as "the edge belongs to the tumor"). The high-level semantics help P3 distinguish between real lesions (such as ulcers) and noise (such as reflective spots), reducing the possibility of false detection.

[0030] Since the P3 feature map (32×32) comes from the shallow layer of the feature pyramid network model, it has high spatial resolution and is suitable for locating small lesions. However, it has a large number of channels (256 dimensions), and direct processing requires a large amount of computation. Therefore, the P3 feature map is subjected to a 1×1 convolution to compress the number of channels to 1, reducing the complexity of subsequent computations. At the same time, the possibility of the existence of lesions needs to be quantified. Therefore, the output probability map is calculated through the activation function. The activation function selects the Sigmoid function, binarizes the probability map, and presets the confidence probability threshold α. Only areas with confidence probabilities higher than α are retained. The default value of α is 0.5. A coarse-grained heat map is generated to quickly screen high-confidence areas and reduce redundant calculations (such as filtering 80% low-probability background areas). The default value of α is 0.5, which can balance sensitivity and specificity and adapt to different clinical needs (such as cancer screening is lowered to 0.3). Since the coarse-grained heat map (32×32) needs to be restored to its original resolution (512×512) and spatially aligned with the edge feature map (512×512×64), considering the resolution matching, the coarse-grained heat map is bilinearly upsampled. The smooth interpolation of bilinear upsampling avoids the mosaic effect and retains the continuity of the heat map probability distribution. After the coarse-grained heat map and the edge feature map of S3 are made to have the same resolution, the coarse-grained heat map (1 channel) and the edge feature map (64 channels) are merged along the channel dimension to output a comprehensive image with a shape of 512×512×65. The coarse-grained heat map provides semantic information ("where is the lesion"), and the edge feature map provides anatomical details ("what does the boundary look like"). The fusion of semantics and details realizes multimodal complementarity and provides rich input for subsequent deformable convolution. Since anorectal mucosal folds, vascular bifurcations, etc. naturally have irregular shapes, traditional fixed convolution kernels are difficult to adapt. The ulcer edge may be jagged and the tumor boundary may be blurred. The fused comprehensive image is input into the deformable convolution network model to optimize the boundary. Among them, the deformable convolution network model includes three layers of deformable convolution. The first layer is used to predict the global offset (such as the overall bending direction), the second layer is used to refine the local deformation (such as the concave and convex edge of the ulcer), and the third layer is used to fine-tune the pixel-level offset (such as the end bifurcation point of the blood vessel). The offset is dynamically set for each layer, so that the convolution kernel adapts to the anatomical structure, thereby adapting to the anatomical variations of different patients (such as differences in the morphology of intestinal folds), and obtaining a refined mask with a boundary that fits the actual anatomical structure; more precise positioning improves the accuracy of lesion detection.

[0031] Considering that the refined mask (512×512) output by the deformable convolution may lose tiny details (such as erosion points <1mm), while the shallow intermediate feature map of S3 (512×512×16) retains high-resolution details (such as tiny bumps on the mucosal surface), the refined mask is input into the U-Net decoder, and the shallow intermediate feature map output by the first layer of the edge extraction network model in S3 is fused through jump connections. The shallow features are spliced ​​with the decoder features layer by layer to supplement the detailed information. The bilinear interpolation technology is used to gradually upsample and restore the resolution to avoid blurring of details, thereby obtaining a pixel-level abnormality probability heat map. Considering that doctors need the lesion probability of each pixel rather than a binary result (such as evaluating the extent of lesion infiltration), each pixel in the pixel-level abnormality probability heat map is labeled with a lesion probability value. The probability value is between 0 and 1. The closer the probability value is to 0, the smaller the lesion probability of the pixel, and the closer the probability value is to 1, the larger the lesion probability of the pixel. The probability value can be used for precise quantitative assessment and support lesion area calculation. At the same time, clinicians can adjust diagnostic criteria based on the probability threshold (such as conservative treatment threshold = 0.4, surgical threshold = 0.8); Because human vision is sensitive to color changes, the Jet color scale (dark blue → green → red) can quickly distinguish low-, medium-, and high-risk areas, which is consistent with doctors' cognitive habits of "hot and cold images." Therefore, Jet color scale mapping maps the lesion probability of each pixel in the pixel-level abnormality probability heat map to the corresponding color. A probability value of 0 corresponds to dark blue, and a probability value of 1 corresponds to red. Red areas (probability = 1) directly prompt doctors to pay priority attention, while dark blue areas (probability = 0) can be ignored. Semi-transparent overlay not only highlights the lesion area, but also preserves the anatomical details of the original image (such as mucosal folds and vascular orientation), preventing the pixel-level abnormality probability heat map from completely obscuring the background. During semi-transparent overlay, the original image of S1 and the pixel-level abnormality probability heat map are blended according to the transparency. β is the preset transparency, usually 0.6. Through Jet color scale mapping and semi-transparency, doctors can locate high-probability lesions at a glance without switching between the original image and the pixel-level abnormality probability heat map. Combining the image and the pixel-level abnormality probability heat map, the spatial relationship between the lesion range and surrounding tissue (such as whether the ulcer is near a blood vessel) is intuitively displayed. Since anorectal diseases often manifest as the coexistence of functional abnormalities (such as increased pressure) and morphological abnormalities (such as mucosal damage), the temporal and spatial correlation between the two needs to be displayed synchronously. For example, images during peak pressure periods (such as defecation) may show mucosal congestion or fissures. Therefore, it is necessary to annotate the abnormal periods of physiological signals. The annotation method for the abnormal periods of physiological signals is as follows: input the timestamps corresponding to the abnormal areas in the physiological signals synchronized in S1 and the pixel-level abnormality probability heat map, and map the abnormal areas in the pixel-level abnormality probability heat map to the synchronized physiological signal time axis according to the timestamps. The abnormal areas are areas with a preset probability threshold δ in the pixel-level abnormality probability heat map. δ is usually set to 0.8. The periods on the signal waveform of the physiological signal that are greater than the preset pressure threshold γ are marked with red shadows to enhance visual contrast. Based on clinical consensus, the pressure threshold γ is set to 30 mmHg. Exceeding this value often indicates a pathological state (such as abnormal resting pressure in patients with anal fissures), and a physiological signal waveform marked by red shadows is obtained. To facilitate doctors to quickly obtain clinical recommendations for patients based on the lesion area visualization and physiological signal waveforms, a weighted score is calculated based on the abnormal probability of the lesion area visualization and the degree of abnormality of the physiological signal. Specifically, S = A × number of high-probability pixels ÷ total number of pixels × 100 + B × peak value of the physiological signal ÷ preset pressure threshold γ × 100, where A and B are preset weights, S is the weighted score, and the number of high-probability pixels refers to the number of pixels with a pixel-labeled lesion probability value greater than 0.9. Considering that image abnormalities are the core basis for the diagnosis of anorectal diseases (such as morphological changes in ulcers and tumors), and pressure abnormalities are a manifestation of functional compensation or pathological exacerbation, A is taken as 0.6 and B as 0.4. The clinical recommendation is divided into four levels: the first level has a weighted score range of 0 to 40, and the clinical recommendation is regular follow-up without intervention; the second level has a weighted score range of 40 to 70, and the clinical recommendation is drug treatment and a review after 3 months; the third level has a weighted score range of 70 to 90, and the clinical recommendation is endoscopic biopsy to exclude malignant lesions; the fourth level has a weighted score range of 90 to 100, and the clinical recommendation is immediate surgical intervention and pathological confirmation. By setting standardized clinical recommendations, subjective differences among different doctors are reduced, ensuring consistent treatment of similar cases, and the grading of clinical recommendations clarifies the allocation of medical resources.

[0032] A second object of the present invention is to provide a multimodal fusion anorectal data visualization analysis system, which is used to implement the above-mentioned multimodal fusion anorectal data visualization analysis method, including: A signal acquisition module, which is used to synchronously acquire anorectal images and physiological signals, preprocess the acquired images and physiological signals, output aligned multimodal data, and record them as original images and encoded physiological signals respectively; An image processing module, configured to pre-process the original image and process the original image into a spatially aligned standardized image based on the affine transformation principle of anatomical landmarks; A feature extraction module, comprising an edge extraction network model, a texture extraction network model, and an LSTM encoder, wherein the edge extraction network model is used to extract edges from the standardized image and output a shallow intermediate feature map and an edge feature map; the texture extraction network model is used to extract texture from the standardized image and output a texture feature map; the LSTM encoder is used to re-encode the encoded physiological signal to generate a physiological signal vector, and broadcast the generated physiological signal vector using the broadcast principle until the physiological signal vector and the edge feature map and the texture feature map are broadcast to the same spatial dimension and compressed into a joint feature map through convolution; Anomaly probability heat map output module, which includes a feature pyramid network model, a deformable convolutional network model and a U-Net decoder, wherein the feature pyramid network model is used to filter the joint feature map to obtain a coarse-grained heat map, and after sampling the coarse-grained heat map, channel-join the coarse-grained heat map with the edge feature map of S3; the deformable convolutional network model is used to optimize the boundary of the coarse-grained heat map after channel-joining with the edge feature map of S3 to obtain a refined mask; the U-Net decoder is used to fuse the refined mask with the shallow intermediate feature map of S3 through a jump connection to obtain a pixel-level anomaly probability heat map; The result output module uses Jet color scale mapping to semi-transparently overlay the pixel-level abnormality probability heat map and the original image of S1 as a visualization of the lesion area, and marks the abnormal period of the physiological signal in S1. Based on the abnormal probability of the lesion area visualization map and the abnormal degree of the physiological signal, a weighted score is calculated, mapped to the clinical recommendation grade, and a graded diagnosis report is generated.

[0033] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal fusion anorectal data visualization analysis method, characterized in that: The following steps are involved: S1. Synchronously collect anorectal images and physiological signals, preprocess the collected images and physiological signals, output aligned multimodal data, and record them as original images and encoded physiological signals respectively; S2. Preprocess the original image and convert it into a spatially aligned standardized image based on the affine transformation principle of anatomical landmarks; S3. Use the network model to perform edge extraction and texture extraction on the standardized image to obtain edge feature maps and texture feature maps. Cache the output of the first convolutional layer of the network model as a shallow intermediate feature map. Perform LSTM encoding on the physiological signal encoded in S1 to generate a physiological signal vector. Broadcast the physiological signal vector, edge feature map, and texture feature map to the same spatial dimension, then splice the channels and compress them into a joint feature map through convolution. S4 generates multi-scale candidate regions through the feature pyramid network model, thereby filtering the joint feature map to obtain a coarse-grained heat map. After sampling the coarse-grained heat map, it is channel-joined with the edge feature map of S3 and input into the deformable convolutional network model to optimize the boundary and generate a refined mask. The refined mask is input into the U-Net decoder and fused with the shallow intermediate feature map of S3 through jump connections to obtain a pixel-level anomaly probability heat map. S5: Use Jet color scale mapping to semi-transparently overlay the pixel-level abnormality probability heat map and the original image of S1 as a visualization of the lesion area, and mark the abnormal time period of the physiological signal in S1. Based on the abnormal probability of the lesion area visualization map and the abnormal degree of the physiological signal, calculate the weighted score and map it to the clinical recommendation grade.

2. The multimodal fusion anorectal data visualization analysis method according to claim 1, characterized in that: In S1, the anorectal image is an anorectal endoscope image, and the physiological signal is an anal canal pressure signal collected in real time by a pressure sensor; the preprocessing is to align the image frame and the physiological signal using a timestamp technology, and then extract the mean, variance, and peak features of the physiological signal using a sliding window technology.

3. The multimodal fusion anorectal data visualization analysis method according to claim 1, characterized in that: In S2, the preprocessing of the original image includes denoising the original image through Gaussian filtering and contrast enhancement of the denoised original image through CLAHE technology; then the enhanced original image is subjected to affine transformation based on anatomical landmarks to unify the resolution and output as a standardized image.

4. The multimodal fusion anorectal data visualization analysis method according to claim 1, characterized in that: In S3, the network model includes an edge extraction network model, a texture extraction network model and an LSTM encoder. The edge extraction network model includes the first convolution layer and the second convolution layer. The normalized image is output as a shallow intermediate feature map through the first convolution layer, and the shallow intermediate feature map is output as an edge feature map through the second convolution layer. The texture extraction network model includes a Gabor filter bank and an SE attention module. The normalized image is passed through the Gabor filter bank to output a multi-scale texture response map, which is then output as a texture feature map through the SE attention module. The LSTM encoder re-encodes the physiological signal encoded in S1 to generate a physiological signal vector, and broadcasts the generated physiological signal vector using the broadcast principle until the physiological signal vector and the edge feature map and texture feature map are broadcast to the same spatial dimension.

5. The multimodal fusion anorectal data visualization analysis method according to claim 1, characterized in that: In S4, the feature pyramid network model is constructed based on the feature map output by the backbone network model. The input standardized image is gradually extracted from the image features through the five stages of the backbone network model. Each layer outputs a feature map corresponding to a different semantic level, specifically: The standardized image outputs a C1 feature map containing edge or texture distribution through convolution and pooling of the first stage layer. The C1 feature map outputs a C2 feature map containing organ contours through convolution of the second stage layer. The C2 feature map outputs a C3 feature map containing lesion areas through convolution of the third stage layer. The C3 feature map outputs a C4 feature map containing tissue abnormalities or texture variations through convolution of the fourth stage layer. The C5 feature map outputs a C5 feature map containing global context information through convolution of the fifth stage layer. The C3 feature map, C4 feature map, and C5 feature map are input into the feature pyramid network model for sampling, horizontal connection, and feature fusion to generate the final feature pyramid level, specifically: The feature C5 map is output through convolution to detect the P5 feature map of large targets; the P5 feature map is sampled through convolution to output the P6 feature map for auxiliary detection of oversized or blurred targets; the P6 feature map is sampled through convolution to output the P7 feature map of extreme scale; the feature C4 map is aligned through the convolution channel and then added element-by-element with the bilinear sampling result of the P5 feature map to output the P4 feature map detected as a medium target; the feature C3 map is aligned through the convolution channel and then added element-by-element with the bilinear sampling result of P4 to output the P3 feature map detected as a small target.

6. The multimodal fusion anorectal data visualization analysis method according to claim 5, characterized in that: After convolution of the P3 feature map, the output probability map is calculated through an activation function. The probability map is binarized and a confidence probability threshold α is preset. Only areas with a confidence probability higher than α are retained to generate a coarse-grained heat map. The coarse-grained heat map is bilinearly sampled to the same resolution as the edge feature map of S3 and then concatenated with the edge feature map of S3 along the channel dimension. After fusion, the boundary is optimized by the deformable convolutional network model to obtain a refined mask with a boundary that fits the actual anatomical structure. The refined mask is input into the U-Net decoder, and the shallow intermediate feature map output by the first layer of the edge extraction network model in S3 is fused through jump connections. The pixel-level abnormality probability heat map is obtained by bilinear interpolation technology. In the pixel-level abnormality probability heat map, each pixel is marked with a lesion probability value between 0 and 1. The closer the probability value is to 0, the smaller the lesion probability of the pixel, and the closer the probability value is to 1, the greater the lesion probability of the pixel.

7. The multimodal fusion anorectal data visualization analysis method according to claim 6, characterized in that: In the S5, the Jet color scale mapping is to map the probability of lesion labeling for each pixel in the pixel-level abnormality probability heat map to the corresponding color, where the probability value of 0 corresponds to dark blue and the probability value of 1 corresponds to red; when the semi-transparent overlay is performed, the original image of S1 and the pixel-level abnormality probability heat map are mixed according to the transparency β, where β is the preset transparency.

8. The multimodal fusion anorectal data visualization analysis method according to claim 7, characterized in that: In said S5, the method for marking the abnormal period of the physiological signal is: input the timestamp corresponding to the abnormal area in the physiological signal and the pixel-level abnormality probability heat map synchronized in S1, map the abnormal area in the pixel-level abnormality probability heat map to the synchronized physiological signal time axis according to the timestamp, the abnormal area is the area with a preset threshold value δ of the probability value in the pixel-level abnormality probability heat map, and mark the period on the signal waveform of the physiological signal that is greater than the preset pressure threshold γ with red shadow, and obtain the physiological signal waveform marked by red shadow.

9. The multimodal fusion anorectal data visualization analysis method according to claim 8, characterized in that: In S5, a weighted score is calculated based on the abnormal probability of the lesion area visualization image and the abnormal degree of the physiological signal, specifically: S = A × number of high-probability pixels ÷ total number of pixels × 100 + B × peak value of the physiological signal ÷ preset pressure threshold γ × 100, where A and B are preset weights, S is the weighted score, and the number of high-probability pixels is the number of pixels with a pixel-labeled lesion probability value greater than 0.9; The clinical recommendation is divided into four levels. The weighted score range of level one is 0 to 40, and the clinical recommendation is regular follow-up without intervention; the weighted score range of level two is 40 to 70, and the clinical recommendation is drug treatment and review after 3 months; the weighted score range of level three is 70 to 90, and the clinical recommendation is endoscopic biopsy to rule out malignant lesions; the weighted score range of level four is 90 to 100, and the clinical recommendation is immediate surgical intervention and pathological confirmation.

10. A multimodal fusion anorectal data visualization analysis system, wherein the multimodal fusion anorectal data visualization analysis system is used to implement the multimodal fusion anorectal data visualization analysis method according to any one of claims 1 to 9, characterized in that: include: A signal acquisition module, which is used to synchronously acquire anorectal images and physiological signals, preprocess the acquired images and physiological signals, output aligned multimodal data, and record them as original images and encoded physiological signals respectively; An image processing module, configured to pre-process the original image and process the original image into a spatially aligned standardized image based on the affine transformation principle of anatomical landmarks; A feature extraction module, comprising an edge extraction network model, a texture extraction network model, and an LSTM encoder, wherein the edge extraction network model is used to extract edges from the standardized image and output a shallow intermediate feature map and an edge feature map; the texture extraction network model is used to extract texture from the standardized image and output a texture feature map; the LSTM encoder is used to re-encode the encoded physiological signal to generate a physiological signal vector, and broadcast the generated physiological signal vector using the broadcast principle until the physiological signal vector and the edge feature map and the texture feature map are broadcast to the same spatial dimension and compressed into a joint feature map through convolution; Anomaly probability heat map output module, which includes a feature pyramid network model, a deformable convolutional network model and a U-Net decoder, wherein the feature pyramid network model is used to filter the joint feature map to obtain a coarse-grained heat map, and after sampling the coarse-grained heat map, channel-join the coarse-grained heat map with the edge feature map of S3; the deformable convolutional network model is used to optimize the boundary of the coarse-grained heat map after channel-joining with the edge feature map of S3 to obtain a refined mask; the U-Net decoder is used to fuse the refined mask with the shallow intermediate feature map of S3 through a jump connection to obtain a pixel-level anomaly probability heat map; The result output module uses Jet color scale mapping to semi-transparently overlay the pixel-level abnormality probability heat map and the original image of S1 as a visualization of the lesion area, and marks the abnormal period of the physiological signal in S1. Based on the abnormal probability of the lesion area visualization map and the abnormal degree of the physiological signal, a weighted score is calculated, mapped to the clinical recommendation grade, and a graded diagnosis report is generated.

Images