Multi-branch collaborative medical video classification method based on heuristic sampling

Abstract

The invention discloses a heuristic sampling-based multi-branch collaborative medical video classification method, which comprises the following steps of: dynamically determining a sampling starting point according to the total frame number of a video, and obtaining a key frame set; performing feature extraction on the key frame set by using parallel network branches to obtain multi-branch features; the network branches comprise a residual space-time network branch, a pre-training large model and normalized attention branch and a wavelet transform convolution residual network branch; fusing the multi-branch features by using a multi-branch feature pyramid to obtain fused features; and inputting the fusion features into a classifier to obtain a diagnosis or classification label. According to the method, the problem of data redundancy can be solved, and feature blur and noise interference are effectively overcome, so that high-precision and high-robustness classification of medical video contents is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical video classification, and particularly relates to a multi-branch collaborative medical video classification method based on heuristic sampling. BACKGROUND

[0002] With the development of medical imaging technology, video data plays an increasingly important role in clinical diagnosis. Compared with static images, medical videos (such as ultrasound, endoscopy, etc.) can capture the dynamic changes of tissues and provide more rich spatio-temporal context information, which is crucial for early detection and accurate diagnosis of diseases. Therefore, developing computer-aided diagnosis (CAD) systems that can automatically analyze these videos has become a research hotspot in the field of medical artificial intelligence.

[0003] Although the medical video analysis technology has broad prospects, its clinical popularization and application still faces multiple challenges, which seriously affects the accuracy and reliability of diagnosis. First, medical videos have significant inter-frame continuity, resulting in high information redundancy. Direct analysis of the entire video not only has heavy computational burden, but also easily makes the model learn redundant features, thereby causing overfitting problem. Second, different diseases or normal tissues may show high inter-class similarity in vision, while the same class of lesions presents obvious intra-class variability in different patients or imaging conditions. This feature makes it extremely difficult to extract effective features with strong discriminative ability. In addition, many medical imaging modalities (such as ultrasound) are inherently limited by image quality problems, including speckle noise and artifact interference, resulting in low image signal-to-noise ratio and blurred lesion boundary, which further exacerbates the difficulty of accurately identifying clinically effective information.

[0004] Therefore, how to overcome the problems of data redundancy and feature interference is urgent to be solved. SUMMARY

[0005] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a multi-branch collaborative medical video classification method based on heuristic sampling. Through a high-efficiency heuristic frame selection strategy (HFS), the data redundancy problem is solved, and a multi-branch collaborative feature learning network is designed to extract and integrate complementary information from different dimensions (semantic, structural, and spatio-temporal), effectively overcoming feature ambiguity and noise interference, so as to realize high-precision and high-robustness classification of medical video content.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: In a first aspect, the present application provides a multi-branch collaborative medical video classification method based on heuristic sampling, comprising the following steps: dynamically determining a sampling starting point according to the total number of video frames, and obtaining a key frame set; The key frame set is feature extracted by using parallel network branches to obtain multi-branch features; the network branches include a residual space-time network branch, a pre-trained large model and a normalized attention branch, and a wavelet transform convolution residual network branch; The multi-branch features are fused by using a multi-branch feature pyramid to obtain fused features; The fused features are input into a classifier to obtain a diagnosis or classification label.

[0007] As a preferred technical solution, the sampling starting point is dynamically determined according to the total number of video frames, and the key frame set is obtained, including The video includes N segments, each segment has T frames, and the frame sampling interval is t; The video type is determined according to the total number of video frames L, and the starting point S of the i-th segment is determined according to the video type i , the j-th key frame f of the i-th segment is determined according to the sampling starting point (i,j) , and the j-th key frame f of the i-th segment is determined according to the sampling starting point (i,j) , and the j-th key frame f of the i-th segment is determined according to the sampling starting point i .

[0008] As a preferred technical solution, the sampling starting point S of the i-th segment is determined according to the total number of video frames L, and the starting point S of the i-th segment is determined according to the video type i , specifically: The threshold l 1 = N+(T-1)•t and the threshold l 2=(T-1)• t; When L≥ l 1, it is considered that the video is a long video, and the sampling interval Δ of the segment starting point is defined as follows: , wherein ⌊•⌋ is the floor function; The starting point S of the i-th segment i , as follows: S i = (i-1) •Δ+ U(0, Δ), wherein U(0, Δ) represents uniformly sampling an integer in the interval [0, Δ]; When l 2≤ L< l 1, it is considered that the video is a medium-length video, and the starting point S i is defined as: if i<L - (T-1)• t, S i =i, otherwise S i = U(L-(T-1) • t, L); When L< l 2, the circular sampling is allowed, and the starting point S iRandomly selected within the entire video length, i.e. S i = U(1, L).

[0009] As a preferred technical solution, the feature extraction of the key frame set by using parallel network branches includes: The residual space-time network branch is used as a basic feature extractor to capture the conventional space-time features in the video clip, the pre-trained large model and the normalized attention branch are used to extract deep semantic features, and the wavelet transform convolution residual network branch is used to extract clear lesion edges and structural information; the network branches include the residual space-time network branch, the pre-trained large model and the normalized attention branch, and the wavelet transform convolution residual network branch are connected in parallel, and the key frame set is processed respectively.

[0010] As a preferred technical solution, the residual space-time network branch is used as a basic feature extractor to capture the conventional space-time features, specifically: Each frame is processed by using a 2D residual network block to obtain spatial features, and 3D convolution and 3D max pooling layers are used to model the motion and time dynamic information between frames, i.e. conventional space-time features; the residual space-time network branch includes a 2D residual network, 3D convolution and 3D max pooling layers.

[0011] As a preferred technical solution, the pre-trained large model and the normalized attention branch are used to extract deep semantic features, specifically: The image encoder of the pre-trained segmentation model is used to process each frame to obtain segmentation features; The segmentation features are refined by using a normalized attention module, including: In the first stage, the segmentation features are captured by using batch normalization to obtain global space-time dependence, and attention masks are obtained; In the second stage, pixel-level normalization is calculated for the attention masks, and pixel-level attention maps, i.e. deep semantic features, are obtained by using Sigmoid activation; The pre-trained large model and the normalized attention branch include a pre-trained segmentation model and a normalized attention module, and the normalized attention module includes batch normalization attention and pixel-level normalization attention.

[0012] As a preferred technical solution, the wavelet transform convolution residual network branch is used to extract clear lesion edges and structural information, specifically: 2D Haar wavelet transform is performed on each input frame tensor X to obtain multiple frequency components, including high frequency components and low frequency components; in each decomposition level, the low frequency components are decomposed again by using 2D Haar wavelet transform to obtain the next level high frequency components; The high-frequency component of each decomposition level is processed by using a lightweight 2D convolution to obtain a high-frequency feature, and the high-frequency feature and the low-frequency component of the corresponding level are element-wise summed to obtain a fusion feature. The fusion feature is reconstructed from the deepest decomposition level to the upper level by using an inverse wavelet transform to obtain a reconstructed feature. The reconstructed feature is captured by using a residual spatio-temporal network to obtain lesion edge and structure information.

[0013] As a preferred technical solution, the 2D Haar wavelet transform uses four separable filters to realize tensor decomposition and obtain low-frequency components X LL , horizontal high-frequency components X LH , vertical high-frequency components X HL and diagonal high-frequency components X HH .

[0014] As a preferred technical solution, the multi-branch feature pyramid is used to fuse the multi-branch features to obtain a fusion feature, specifically: The multi-branch feature pyramid is used to gradually splice and element-wise add the multi-branch features, and through upsampling and downsampling operations, multi-dimensional information is integrated on different scales to obtain a unified feature representation.

[0015] As a preferred technical solution, the classifier includes a 3D convolution, a pooling layer, a fully connected layer, and a classification head.

[0016] Compared with the prior art, the present application has the following advantages and beneficial effects: (1) The present application adopts a heuristic sampling strategy, which is a general sampling method independent of specific disease types. It can adaptively adjust the sampling according to the length of any medical video to ensure efficient capture of clinically relevant dynamic information while greatly reducing redundant data processing.

[0017] (2) The three-branch collaborative network architecture of the present application can extract features from different angles to form complementary advantages and effectively cope with the complexity of various medical videos. The SAM-NAM branch uses deep semantic knowledge of large-scale medical pre-training models to effectively distinguish different tissues or lesions that are highly similar in vision; the WTCR branch uses the characteristics of wavelet transform to extract lesion edge and structure information while denoising BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0019] Figure 1 Flow chart of the multi-branch collaborative medical video classification method based on heuristic sampling in the embodiments of the present application; Figure 2 Overall connection diagram of each link in the embodiments of the present application; Figure 3 Schematic diagram of the residual spatio-temporal network branch in the embodiments of the present application; Figure 4 Schematic diagram of the pre-training large model and the normalized attention branch in the embodiments of the present application; Figure 5 Schematic diagram of the wavelet transform convolution residual network branch in the embodiments of the present application. DETAILED DESCRIPTION

[0020] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0021] In the present application, the phrase "embodiments" means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean that the same embodiments are referred to, nor does it mean that independent or alternative embodiments are mutually exclusive or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0022] Please refer to Figures 1 to 2 The present embodiment provides a multi-branch collaborative medical video classification method based on heuristic sampling, which comprises the following steps: S1, dynamically determining a sampling starting point according to the total number of video frames to obtain a key frame set.

[0023] To solve the video information redundancy and ensure that the sampling segment can cover the area containing the key diagnostic information, step S1 adopts a heuristic frame sampling strategy (HFS), provided that the video includes N segments, each segment has T frames, the frame sampling interval is t, the video type is determined according to the total number of frames L, and the starting point S of the i-th segment is determined according to the video type i , the j-th key frame f of the i-th segment according to the sampling starting point (i,j) , as follows: f (i,j) = (S i + j • t) mod L.

[0024] Wherein, the determination of the starting point S i is divided into the following three cases: Case 1: long video (L ≥ N+(T-1)•t), at this time the video length is sufficient.

[0025] First, define the sampling interval Δ of the segment starting point, as follows: , wherein, ⌊•⌋ is the floor function; The starting point S of the i-th segment i , as follows: S i = (i-1) •Δ+ U(0, Δ), wherein U(0, Δ) represents uniformly sampling an integer in the interval [0, Δ] to enhance the randomness of training.

[0026] Case 2: medium length video ((T-1)• t ≤ L<N+(T-1)• t), at this time the video length is insufficient to completely sample N segments without overlap. The starting point S i is defined as: if i<L - (T-1) • t, S i =i, otherwise S i = U(L-(T-1) •t, L).

[0027] Case 3: short video (L<(T-1) • t), at this time the short video is too short to allow cyclic sampling. The starting point S i is randomly selected within the entire video length, i.e. S i =U(1,L).

[0028] S2, using parallel network branches to extract features from the key frame set to obtain multi-branch features; the network branches include residual space-time network branches, pre-trained large model and normalized attention branches, and wavelet transform convolution residual network branches.

[0029] In step S2, three parallel network branches involved are used to extract complementary feature representations. Specifically, the residual spatio-temporal network branch is used as a basic feature extractor to capture the regular spatio-temporal features F1 in the video clip, the pre-trained large model and normalization attention branch is used to extract deep semantic features F2, and the wavelet transform convolution residual network branch is used to extract clear lesion edge and structure information F3. The network branches include the residual spatio-temporal network branch, the pre-trained large model and normalization attention branch, and the wavelet transform convolution residual network branch connected in parallel, and each branch processes the key frame set.

[0030] wherein, Branch (a): Residual Spatio-Temporal Network. This branch serves as a basic feature extractor, aiming to capture the regular spatio-temporal features in the video clip. It adopts a hybrid design, first using 2D residual network (ResNet) blocks to process each frame to extract spatial features, and then modeling inter-frame motion and temporal dynamics information through 3D convolution and 3D max-pooling layers, as detailed in Figure 3 .

[0031] Branch (b): Pre-trained Large Model and Normalization Attention (Segment Anything Model-Normalization based Attention Model, SAM-NAM) branch. This branch uses a large-scale pre-trained medical model to extract deep semantic features to address the problem of highly similar visual features between different tissues or lesions, such as Figure 4 . The pre-trained large model and normalization attention branch includes a pre-trained segmentation model and a normalization attention module, which includes batch normalization (Batch Normalization, BN) and pixel-level normalization attention. The pre-trained segmentation model here uses a segmentation model pre-trained on a large medical dataset (such as SAM), and the image encoder uses a Vision Transformer (ViT) architecture, which can capture long-range context information and is particularly effective for identifying diffuse or irregularly shaped lesions. During training, the weights of this encoder are frozen to preserve its strong medical prior knowledge.

[0032] At this time, the deep semantic features extracted by the pre-trained large model and normalization attention branch are as follows: S21, using the image encoder of the pre-trained segmentation model to process each frame to obtain segmentation features F SAM ; S22, using the normalization attention module to refine the segmentation features, including: S221, capture global spatio-temporal dependencies by BN and generate attention mask, which is calculated by the following formula to obtain the enhanced feature map F BN , BN = σ(BN(F SAM ))• F SAM , where σ(·) represents the Sigmoid activation function. This operation fuses the temporal context by aggregating the statistical changes in the batch dimension; S222, in the second stage, perform pixel-level normalization on F BN to refine local spatial details.

[0033] First, calculate the normalized feature F norm ; then, apply the Sigmoid activation to generate the pixel-level attention map, and calculate the final output depth semantic feature F2 of this branch as follows: F2= σ(F norm )• F BN . This module can adaptively emphasize information-rich areas while maintaining low computational and parameter overhead.

[0034] Branch (c): Wavelet Transform Convolutional Residual Network (WTCR), which utilizes the multi-scale and multi-frequency characteristics of wavelet transform, aims to extract clear lesion edges and structural information from noisy medical videos, as shown in Figure 5 .

[0035] The wavelet transform convolutional residual network has multiple decomposition levels, including 2D Haar wavelet transform WT, lightweight 2D convolution, inverse wavelet transform, and residual spatio-temporal network. Among them, the 2D Haar wavelet transform utilizes four separable filters, which are implemented through four separable filters as follows: , .

[0036] The four filters process the low-frequency component X LL , horizontal high-frequency component X LH , vertical high-frequency component X HL , and diagonal high-frequency component X HH of the corresponding level, respectively. This part is a wavelet convolutional transform on each frame tensor X, where the convolution is performed on the high-frequency subbands obtained after multi-level decomposition. Deep convolution is a specific implementation of this particular wavelet transform, and the four Haar wavelet filters are the convolution kernels (filters) used in deep convolution. For specific feature processing, see the following description: at each decomposition level , set the 2D Haar wavelet transform WT (X) to obtain the low-frequency component X LLαhorizontal high frequency component X LHα vertical high frequency component X HLα and diagonal high frequency component X HHα where the set of high frequency subbands generated here horizontal high frequency component X LHα vertical high frequency component X HLα and diagonal high frequency component X HHα A light 2D convolution Conv 2D is used to process the set of high frequency subbands to extract local discriminative features, i.e. high frequency features such as edges and textures, in the following way: .

[0037] The spatial resolution is gradually recovered by inverse wavelet transform (IWT) and the reconstruction process is as follows: .

[0038] Residual spatio-temporal network (RSTN) can capture its dynamic changes and context-dependent relationships, and output lesion edge and structure information

[0039] In specific operation, as shown in Figure 5 , the embodiment includes the following steps: S23, 2D Haar wavelet transform is performed on each input frame tensor X to obtain a plurality of frequency components, including high frequency components and low frequency components; in each decomposition level, the low frequency components are decomposed again by using 2D Haar wavelet transform to obtain high frequency components of the next level; S24, a light 2D convolution is used to process the high frequency components of each decomposition level to obtain high frequency features, and the high frequency features and the low frequency components of the corresponding level are element-wise summed to obtain fusion features; S25, inverse wavelet transform is used to reconstruct the fusion features from the deepest decomposition level to the upper level to obtain reconstructed features; S26, residual spatio-temporal network is used to capture the dynamic changes and context-dependent relationships of the reconstructed features to obtain lesion edge and structure information.

[0040] S3, the multi-branch feature pyramid is used to fuse the multi-branch features to obtain fusion features.

[0041] The three branches extract different types of features (regular spatiotemporal features, deep semantic features, and frequency structure features) and efficiently fuse them. This step integrates multi-dimensional information at different scales through a series of concatenation, element-wise addition, up-sampling, and down-sampling operations in a pyramid-like structure, generating a more comprehensive and robust unified feature representation than any single branch, as shown in Figure 2 .

[0042] S4, input the fused features into the classifier to obtain the diagnosis or classification label.

[0043] The fused final features are input into the classification head. The classification head is usually composed of 3D convolution, pooling layer, and fully connected layer, etc., responsible for mapping complex features to the final classification space. The system finally outputs the diagnosis or classification label, such as "benign" or "malignant", "lesion A" or "lesion B", etc., as shown in Figure 2 .

[0044] To verify the effectiveness and advancement of the method described in the present application, a more detailed implementation is also provided in this embodiment.

[0045] This embodiment conducts a series of experiments on a large-scale, multi-center real clinical dataset. The experimental dataset used in this embodiment is collected from 622 patients in two first-class hospitals (Shenzhen People's Hospital and the First Affiliated Hospital of Jinan University) from 2019 to 2022, totaling 1083 transrectal ultrasound (TRUS) videos. All videos are labeled as "benign" or "malignant" by biopsy pathology results, ensuring the accuracy of the data. The dataset contains 857 videos of 346 benign patients and 486 videos of 276 malignant patients. To ensure the standardization of model input, all video frames are uniformly cropped to a resolution of 224x224 pixels.

[0046] The HFS strategy designed in this embodiment is a general sampling method independent of specific disease types. It can adaptively adjust the sampling according to the length of any medical video, ensuring efficient capture of clinically relevant dynamic information while greatly reducing redundant data processing. Experiments have shown that this strategy can significantly improve the capture rate of effective frames (containing lesion information), reducing the proportion of ineffective segments from 26.1% to 2.9%.

[0047] The three-branch collaborative network architecture of the embodiment can extract features from different angles, form complementary advantages, and effectively cope with the complexity of various medical videos. Among them, the SAM-NAM branch utilizes the deep semantic knowledge of large-scale medical pre-training models, which can effectively distinguish different tissues or lesions that are highly similar in vision; the WTCR branch utilizes the characteristics of wavelet transform to extract the edge and structure information of the lesion while denoising, so that the method performs particularly well in processing low signal-to-noise ratio videos such as ultrasound.

[0048] Table 1 Comparative experiment

[0049] As shown in Table 1, the core idea and architecture of the embodiment is not limited to a specific disease, and its excellent performance is verified on real medical video datasets, achieving a classification accuracy of up to 86.35% and a precision of 94.17%, and its comprehensive performance index is significantly better than a variety of current mainstream video classification models. The method achieves an excellent balance between high precision (reducing misdiagnosis) and high sensitivity (reducing missed diagnosis), and can be used as a reliable and efficient general framework to provide strong technical support for various video-based clinical diagnosis tasks.

[0050] In order to comprehensively evaluate the performance of the application, it is compared with eleven kinds of mainstream advanced video classification algorithms in the field, including but not limited to TSN, I3D, TRN, SlowFast, TPN, etc.

[0051] The experimental results show that the method proposed by the application is significantly better than all the comparison methods in many key evaluation indicators. As shown in Table 1, the method of the application ranks first in the three core comprehensive indicators of accuracy, F1-Score and AUC (Area Under Curve), with an accuracy of 86.35%. The application achieves a precision of 94.17% and a specificity of 97.67%, which indicates that the method has very high reliability when making positive predictions, and can accurately identify negative samples, thereby effectively controlling the misdiagnosis rate. While maintaining high precision, the method also achieves a sensitivity of 66.44%, effectively identifying positive cases and reducing the risk of missed diagnosis. In contrast, some other methods have very high scores in individual indicators, but have very low sensitivity (close to 0), which indicates that they tend to predict all samples as negative, which has no practical value in clinical practice. In summary, through comprehensive comparison with more than ten existing advanced methods on real medical video datasets, the embodiment fully verifies the great advantages and effectiveness of the method of the application in classification accuracy, reliability and clinical practicability.

[0052] It should be noted that for the foregoing method embodiments, the steps recited in the claims need not be performed in the order recited, unless otherwise specified in the claims. Thus, unless specifically stated in the claims, embodiments can be performed in any order.

[0053] Any combination of the above-described technical features in the above-described embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combinations of the technical features do not exist in contradiction, they should be considered as the scope of the description.

[0054] The above-described embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above-described embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principle of the present application should be equivalent replacement modes, and should be included in the protection scope of the present application.

Claims

1. A multi-branch collaborative medical video classification method based on heuristic sampling, characterized in that, Includes the following steps: The sampling starting point is dynamically determined based on the total number of video frames to obtain the key frame set; The keyframe set is feature extracted using parallel network branches to obtain multi-branch features; the network branches include residual spatiotemporal network branch, pre-trained large model and normalized attention branch, and wavelet transform convolutional residual network branch. Multi-branch feature pyramids are used to fuse multi-branch features to obtain fused features; Input the fused features into the classifier to obtain diagnostic or classification labels.

2. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 1, characterized in that, The process of dynamically determining the sampling starting point based on the total number of video frames and obtaining a keyframe set includes... Suppose a video consists of N segments, each segment has T frames, and the frame sampling interval is t; The video type is determined based on the total number of video frames L, and the starting point S of the i-th segment is determined based on the video type. i Based on the j-th keyframe f of the i-th segment at the sampling starting point (i,j) As shown in the following formula: f (i,j) = (S i + j • t) mod L.

3. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 2, characterized in that, The video type is determined based on the total number of video frames L, and the starting point S of the i-th segment is determined based on the video type. i Specifically: Determine the threshold l 1 = N+(T-1)•t and threshold l 2 = (T-1)• t; When L≥ l If the value is 1, then the video is considered a long video, and the sampling interval Δ at the start point of the segment is defined as follows: Where ⌊•⌋ is rounded down; The starting point S of the i-th segment i As shown in the following formula: S i = (i-1) •Δ+ U(0,Δ), where U(0, Δ) represents uniformly sampling an integer within the interval [0, Δ]. when l 2≤ L < l If the value is 1, then the video is considered to be of medium length, with the starting point S. i Defined as: if i < L-(T-1)•t, S i =i, otherwise S i = U(L-(T-1) • t ,L); When L< l At time 2, cyclic sampling is allowed, starting from point S. i Randomly selected within the entire video length, i.e., S i =U(1,L).

4. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 1, characterized in that, The method of extracting features from the keyframe set using parallel network branches to obtain multi-branch features includes: The residual spatiotemporal network branch is used as the basic feature extractor to capture conventional spatiotemporal features in video clips. Deep semantic features are extracted using a pre-trained large model and a normalized attention branch. Clear lesion edges and structural information are extracted using a wavelet transform convolutional residual network branch. The network branches include a residual spatiotemporal network branch, a pre-trained large model and a normalized attention branch, and a wavelet transform convolutional residual network branch connected in parallel, and each branch processes the keyframe set.

5. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 4, characterized in that, The method of using residual spatiotemporal network branches as basic feature extractors to capture conventional spatiotemporal features in video clips specifically involves: Each frame is processed using 2D residual network blocks to obtain spatial features, and 3D convolution and 3D max pooling layers are used to model the motion and temporal dynamics between frames, i.e., conventional spatiotemporal features. The residual spatiotemporal network branch includes a 2D residual network, a 3D convolutional layer, and a 3D max pooling layer.

6. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 4, characterized in that, The method of extracting deep semantic features using a pre-trained large model and a normalized attention branch specifically involves: Each frame is processed using a pre-trained segmentation model image encoder to obtain segmentation features; The segmentation features are refined using a normalized attention module, including: In the first stage, batch normalization is used to capture the global spatiotemporal dependence of segmentation features and obtain the attention mask. In the second stage, pixel-level normalization is calculated for the attention mask, and sigmoid activation is used to obtain the pixel-level attention map, i.e., deep semantic features. The pre-trained large model and normalized attention branch include a pre-trained segmentation model and a normalized attention module, wherein the normalized attention module includes batch normalized attention and pixel-level normalized attention.

7. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 4, characterized in that, The method of extracting clear lesion edges and structural information using wavelet transform convolutional residual network branches specifically involves: Perform a 2D Haar wavelet transform on each input frame tensor X to obtain multiple frequency components, including high-frequency and low-frequency components; in each decomposition level, use the 2D Haar wavelet transform again to decompose the low-frequency components to obtain the next level of high-frequency components. Lightweight 2D convolution is used to process the high-frequency components of each decomposition level to obtain high-frequency features. The high-frequency features and the low-frequency components of the corresponding level are summed element-wise to obtain fused features. The inverse wavelet transform is used to reconstruct the fused features from the deepest decomposition level upwards, obtaining the reconstructed features. By utilizing residual spatiotemporal networks, the dynamic changes and contextual dependencies of reconstructed features are captured, thereby obtaining information on lesion edges and structures.

8. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 7, characterized in that, The 2D Haar wavelet transform utilizes four separable filters to achieve tensor decomposition, obtaining the low-frequency component X at the corresponding decomposition level. LL Horizontal high-frequency component X LH Vertical high-frequency component X HL and the diagonal high-frequency component X HH .

9. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 4, characterized in that, The method of fusing multi-branch features using a multi-branch feature pyramid to obtain fused features is as follows: By using a multi-branch feature pyramid, multi-branch features are gradually spliced ​​together and added element by element. Through upsampling and downsampling operations, multi-dimensional information is integrated at different scales to obtain a unified feature representation.

10. The multi-branch collaborative medical video classification method based on heuristic sampling according to claim 1, characterized in that, The classifier includes 3D convolutional layers, pooling layers, fully connected layers, and a classification head.

Images