An endoscope multi-modal video data processing method, system and storage medium

Abstract

The application discloses an endoscope multi-modal video data processing method and system and a storage medium, and relates to the field of artificial intelligence and medical image analysis. The method obtains white light, fluorescence pseudo-color and native fluorescence three-way video data in the same endoscopy, and divides the three-way video data into corresponding space-time data units according to a unified time index and a space grid. A white light video encoder is used as a teacher, and a fluorescence pseudo-color video encoder is used as a student to construct a cross-modal feature constraint to update the latter. Then, the fluorescence pseudo-color video encoder is used as the teacher, and the native fluorescence video encoder is used as the student. According to a fluorescence reliability weight, a feature difference is weighted to obtain a local cross-modal distillation target to update the latter. Three corresponding features are combined into a three-modal feature group, and the three-modal feature group is input into a three-modal fusion network to obtain three-modal fusion features, which are used for feature analysis of a lesion area. The method utilizes progressive correlation of the three-way video data, reduces the influence of limited fluorescence data, and enables distillation to be concentrated on a reliable fluorescence position.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image analysis technology, and in particular to a method, system and storage medium for processing multimodal video data from endoscopy. Background Technology

[0002] Endoscopic examination can directly display the tissue morphology and lesion characteristics of cavities and organs, and is an important technical means for lesion detection, boundary determination, and treatment. With the development of fluorescence endoscopic imaging technology, some endoscopic devices can simultaneously output fluorescence pseudo-color video data and native fluorescence video data in addition to white light video data. White light video data mainly reflects tissue color, surface texture, blood vessel morphology, and lesion outline; fluorescence pseudo-color video data is used to display the spatial relationship between fluorescence signals and tissue structures; and native fluorescence video data is used to record the intensity distribution of fluorescence signals and their changes over time.

[0003] Existing endoscopic video processing methods typically build analytical models for a single video modality, or extract features from different modalities and then perform simple stitching or weighting, without fully considering the differences in data distribution among white light video data, fluorescence pseudo-color video data, and native fluorescence video data. Some methods treat static images as processing objects and do not involve temporal modeling in continuous video data. The amount of white light video data available clinically is usually significantly greater than the amount of fluorescence video data, and the limited fluorescence video data is not being fully utilized.

[0004] Existing methods often treat the entire lesion area, a single lesion frame, or mutually independent local regions as processing objects, making it difficult to continuously characterize the differences between the lesion center and its edges in different directions. In continuous endoscopic video data, lens movement, changes in observation distance, tissue deformation, reflection, blood contamination, and instrument obstruction can easily cause positional drift or tracking loss of local areas, affecting feature extraction and comparison of corresponding areas in three video streams.

[0005] Different endoscopic devices, types of endoscopes, sensor modes, and imaging conditions such as exposure, gain, white balance, noise reduction, and fluorescence channel gain can affect video signals. Different types of fluorescent reagents, dosages, administration methods, and post-administration times can also affect fluorescence signals. Fluorescence information obtained by existing technologies is not comparable between different examination conditions.

[0006] Therefore, a processing method for endoscopic multimodal video data that can overcome the above-mentioned shortcomings is needed. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, and storage medium for processing multimodal endoscopic video data, in order to solve the problems in the prior art that the data distribution differences between white light video data, fluorescent pseudo-color video data, and native fluorescence video data are not fully utilized, the limited fluorescence video data is difficult to support feature extraction, the position of the local area of ​​the lesion in continuous video is easy to drift or be lost, the difference between the lesion center and the edge in different directions is difficult to continuously characterize, and the obtained fluorescence information is not comparable under different examination conditions.

[0008] To achieve the above objectives, according to one aspect of the present invention, a method for processing endoscopic multimodal video data is provided, comprising: Acquire white light video data, fluorescence pseudo-color video data, and native fluorescence video data generated during the same endoscopic examination. Divide the three video data into spatiotemporal data units according to a unified time index and spatial grid, so that the spatiotemporal data units with the same time index and spatial grid position in the three video data correspond to each other. Among them, the fluorescence pseudo-color video data is video data formed by superimposing the native fluorescence signal with the white light tissue image after color mapping. Its luminance component carries tissue structure information, and its chromaticity component carries fluorescence response information. Using a white light video encoder as the fixed teacher encoder and a fluorescent pseudo-color video encoder as the student encoder, corresponding white light spatiotemporal data units and fluorescent pseudo-color spatiotemporal data units are input into the two respectively. Cross-modal feature constraints are constructed based on the differences in the obtained features after feature projection. The fluorescent pseudo-color video encoder is updated together with the cross-modal feature constraints and the occlusion reconstruction target of the fluorescent pseudo-color spatiotemporal data units. Among them, the white light video encoder is an encoder obtained by self-supervised pre-training through white light video data occlusion reconstruction. The initial parameters of the fluorescent pseudo-color video encoder are copied from the spatiotemporal feature coding layer parameters of the white light video encoder and are configured with independent input normalization parameters and modality adaptation layers. Using the updated fluorescence pseudo-color video encoder as the fixed teacher encoder and the native fluorescence video encoder as the student encoder, fluorescence reliability weights are determined for each native fluorescence spatiotemporal data unit. The feature differences between corresponding fluorescence pseudo-color spatiotemporal data units and native fluorescence spatiotemporal data units after feature projection are weighted according to these fluorescence reliability weights to obtain local cross-modal distillation targets, which are then used to update the native fluorescence video encoder. The fluorescence reliability weights are determined based on the fluorescence signal intensity after background subtraction, the effective pixel ratio, and the saturated pixel ratio within the spatiotemporal data unit. Higher fluorescence signal intensity, a larger effective pixel ratio, and a smaller saturated pixel ratio result in a larger fluorescence reliability weight. The fluorescence reliability weight is set to zero when the effective pixel ratio is below a first threshold or the saturated pixel ratio is not lower than a second threshold. The native fluorescence video encoder is an encoder obtained through self-supervised pre-training based on native fluorescence video data occlusion reconstruction. The three corresponding spatiotemporal data units are input into the white light video encoder, the updated fluorescent pseudo-color video encoder, and the updated native fluorescent video encoder, respectively. The resulting features are combined into a three-modal feature group according to a unified time index and spatial grid and input into the three-modal fusion network to output the three-modal fusion features.

[0009] Optionally, the fluorescence reliability weight is determined as follows: the median of the lowest percentage of pixels with the lowest signal value within the video frame containing the spatiotemporal data unit is used as the background signal value of that video frame; the effective pixels are the pixels within the spatiotemporal data unit after excluding pixels obscured by instruments, pixels with missing data, invalid pixels of the device, and saturated pixels. After background subtraction of the effective pixels within the spatiotemporal data unit, the highest and lowest signal values ​​of the preset percentage are removed, and the average of the remaining signal values ​​is taken as the fluorescence signal intensity. The fluorescence signal intensity is then normalized according to the fluorescence signal intensity distribution of each spatiotemporal data unit that meets the quality conditions within the same video segment. The quality conditions are that the proportion of effective pixels is not lower than a first threshold and the proportion of saturated pixels is lower than a second threshold; pixels whose signal values ​​reach the upper limit of the device's effective signal are determined as saturated pixels, and a saturation penalty coefficient that decreases as the proportion of saturated pixels increases is determined; the product of the normalized fluorescence signal intensity, the proportion of effective pixels, and the saturation penalty coefficient is used as the initial weight. For spatiotemporal data units that meet the quality conditions, the initial weight is limited to between a preset lower limit and 1, and the fluorescence reliability weight of each spatiotemporal data unit within the same video segment is normalized.

[0010] Optionally, when updating the native fluorescence video encoder, a video segment-level cross-modal distillation target is also constructed. The video segment-level cross-modal distillation target is the difference between the teacher-side features and student-side features in the same video segment after weighted aggregation according to fluorescence reliability weights. The local cross-modal distillation target, the video segment-level cross-modal distillation target, and the occlusion reconstruction target of the native fluorescence spatiotemporal data unit are weighted and combined according to preset weights to update the native fluorescence video encoder.

[0011] Optionally, the cross-modal feature constraint is determined by the distance between the features output by the white light video encoder and the fluorescent pseudo-color video encoder in the same intermediate spatiotemporal feature coding layer after projection and normalization; the modality adaptation layer is a bottleneck residual structure set at the output end of each spatiotemporal feature coding layer, which reduces the dimension of the input features and then increases the dimension, and adds it to the original output of the spatiotemporal feature coding layer without changing the number and arrangement order of feature units; when updating the fluorescent pseudo-color video encoder, the fixed state of the parameters obtained by copying is released in stages, first only the input normalization parameters and the modality adaptation layer are updated, and then the fixed state of each spatiotemporal feature coding layer is gradually released from the output end to the input end.

[0012] Optionally, the above method further includes: determining the lesion center site and multiple edge sites located in different directions of the lesion in the target frame of the white light video data; establishing a local coordinate system with the lesion center site as the origin; recording the normalized distance, orientation angle, site identifier, and sequential relationship between each edge site relative to the lesion center site; forming a lesion center-edge topology; the site identifier remains unchanged in continuous video data; each point corresponds to a local analysis area containing that site.

[0013] Optionally, the above method further includes joint tracking of each point between adjacent frames of white light video data: generating candidate positions for each point, and determining an appearance matching score representing the similarity between the candidate position and the corresponding region in the previous frame, a motion consistency score representing the consistency between the candidate position and the preceding motion trajectory, and a topological consistency score representing the consistency between the spatial relationship of the candidate position relative to other points in the local coordinate system and the lesion center-edge topology; determining the position of each point in the current frame according to the weighted sum of the three, while maintaining the point identifier and the lesion center-edge topology; the weight of each score in the weighted sum is determined according to the tracking confidence of the corresponding point in the previous frame, and the lower the tracking confidence, the greater the weight of the topological consistency score; the tracking confidence is determined according to the weighted sum of the candidate positions used by the point in the corresponding frame.

[0014] Optionally, when the highest weighted sum of a point in the current frame is lower than a preset acceptance threshold, its candidate position is not adopted. Instead, its current position is estimated based on the local transformation of other points with tracking confidence higher than a preset reliability threshold and the preceding motion trajectory of that point, and the tracking confidence of that point in the current frame is set to a preset value. Tracking is resumed when the scores of that point meet the preset recovery conditions. According to the temporal index and spatial grid correspondence between the three video data, the local analysis region of each point is mapped to the fluorescence pseudo-color video data and the original fluorescence video data. The three corresponding local spatiotemporal data units are extracted and input into the corresponding video encoder and the three-modal fusion network respectively to obtain the local three-modal fusion features of each point.

[0015] Optionally, the above method further includes: inputting the device imaging conditions and fluorescence signal formation conditions during video data acquisition into two independent conditional coding networks to obtain device conditional features and fluorescence conditional features; combining the device conditional features with video imaging features and combining the fluorescence conditional features with fluorescence dynamic features, wherein the combination is either splicing or generating parameters from the conditional features to adjust the corresponding features; extracting fluorescence curve parameters for each point based on the change of effective pixel signals of the original fluorescence video data in the local analysis area of ​​each point over time, and arranging the differences in fluorescence curve parameters between each edge point and the lesion center point according to the site identifier, orientation angle, and order relationship of each edge point to form fluorescence dynamic difference features corresponding to the lesion center-edge topology; inputting the local three-modal fusion features, fluorescence dynamic difference features, device conditional features, fluorescence conditional features, and lesion center-edge topology of each point into a lesion-level fusion network to output a lesion-level three-modal video representation.

[0016] According to another aspect of the present invention, an endoscopic multimodal video data processing system is also provided, comprising: The data correspondence module is used to acquire white light video data, fluorescence pseudo-color video data and native fluorescence video data generated during the same endoscopic examination, and divide the three video data into corresponding spatiotemporal data units according to a unified time index and spatial grid. The first migration module is used to copy the spatiotemporal feature coding layer parameters of the white light video encoder to the initial parameters of the fluorescent pseudo-color video encoder, configure independent input normalization parameters and modal adaptation layer for the fluorescent pseudo-color video encoder, and use the white light video encoder as a fixed teacher encoder to construct cross-modal feature constraints based on the difference between the output features of the two encoders after feature projection, thereby updating the fluorescent pseudo-color video encoder. The second migration module is used to take the updated fluorescence pseudo-color video encoder as the fixed teacher encoder, and according to the fluorescence reliability weight determined by the fluorescence signal intensity after background subtraction, the effective pixel ratio and the saturated pixel ratio, weight the feature difference between the fluorescence pseudo-color spatiotemporal data unit and the original fluorescence spatiotemporal data unit after feature projection to obtain the local cross-modal distillation target, and update the original fluorescence video encoder accordingly. The feature fusion module is used to extract features from three corresponding spatiotemporal data units through the corresponding video encoders and combine them into a three-modal feature group, which is then input into the three-modal fusion network and outputs the three-modal fused features.

[0017] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the above-described method for processing endoscopic multimodal video data.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention first uses white light endoscope video data to pre-train the video encoder, and then transfers the feature extraction capability of white light video data to fluorescent pseudo-color video data through parameter initialization, modal adaptation and cross-modal feature constraints. Furthermore, it transfers the relevant features of fluorescent pseudo-color video data to native fluorescent video data through cross-modal distillation. This allows the fluorescent pseudo-color video encoder and the native fluorescent video encoder to learn from a random initialization state without having to start from a random initialization state. Instead, they inherit the spatiotemporal representation capability of the white light video encoder and the fluorescence correlation representation capability of the fluorescent pseudo-color video encoder, respectively. This utilizes the progressive correlation between the three video data to reduce the impact of the limited amount of fluorescent video data on model training and improve the feature extraction capability of the three video data.

[0019] (2) According to the fluorescence reliability weight determined by the fluorescence signal intensity after background subtraction, the effective pixel ratio and the saturated pixel ratio, the present invention weights the feature differences between the teacher side and the student side to obtain the local cross-modal distillation target, so that the distillation is concentrated in the spatiotemporal position where the fluorescence signal is reliable, avoiding the introduction of incorrect cross-modal correspondence by instrument occlusion, invalid area and saturated area, and improving the reliability of the features obtained by the native fluorescence video encoder.

[0020] (3) This invention determines multiple sites at the center of the lesion and the edges of the lesion in different directions, and uses the lesion center-edge topology to jointly track the local analysis area of ​​each site, so that the position of each site is determined by appearance matching, motion consistency and topological consistency, rather than relying solely on its own local appearance; when the appearance matching of a site fails due to lens movement, tissue deformation, local reflection or instrument occlusion, the position and site identification of the site can still be maintained based on the local transformation estimated by other reliable sites, and tracking can be resumed after the site becomes visible again, thereby reducing tracking drift and tracking loss, and avoiding site identity confusion among multiple edge sites with similar local appearances; at the same time, based on the established time synchronization relationship and spatial correspondence relationship between the three video data, the local three-modal fusion features of the lesion center site and multiple edge sites and the fluorescence curve parameters of the original fluorescence video data are extracted, improving the stability of local area localization and cross-modal correspondence analysis in continuous video data.

[0021] (4) The present invention encodes the equipment imaging conditions and fluorescence signal formation conditions respectively to adjust the processing of video imaging features and fluorescence dynamic features, and integrates global three-modal fusion features, local three-modal fusion features of lesion center site and multiple edge sites, fluorescence curve parameters, fluorescence dynamic difference features and image difference features to form a lesion-level three-modal video representation that can reflect the overall environment, local features and spatial heterogeneity of the lesion, thereby improving the comprehensive representation ability of multimodal video data under different examination conditions. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a general flowchart of the method for processing endoscopic multimodal video data provided in the embodiments of the present invention; Figure 2 This is a flowchart of the phased transfer training provided in an embodiment of the present invention; Figure 3 This is a flowchart of the establishment and joint tracking of the lesion center-edge topology provided in the embodiments of the present invention; Figure 4 This is a block diagram of the endoscopic multimodal video data processing system provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0026] It should also be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0027] The method of the present invention is performed by a computer device connected to an endoscopic device. The computer device includes a processor, a graphics processor, a memory, and a video interface. The video interface receives three video data streams output by the endoscopic device; the memory stores the three video data streams, model parameters, and a computer program; the processor and the graphics processor execute the computer program, complete the calculations in the following steps, and store the calculation results in the memory or send them out via the output interface.

[0028] In this embodiment, both model training and inference are performed on a server configured with a graphics processor; the trained model parameters can be deployed on the embedded computing unit of the endoscope host or on a workstation connected to the endoscope host to process the video data output during the examination.

[0029] The endoscopic video data used for model training and method validation in this embodiment are all clinical examination data collected after ethical review and approval by medical institutions and with informed consent from the examinees. Before entering the processing flow of this method, direct identifying information such as name, consultation number, examination number, and date of birth has been removed from the video data, and the areas where the device information displayed in the video screen is overlaid have been masked.

[0030] This method does not collect, store, or use the subject's name, facial image, or other identification information, nor does it use the subject's gender, age, ethnicity, occupation, or other individual attributes as input to any network or as a basis for differentiating processing strategies or adjusting parameter values. The conditional parameters described below are limited to equipment imaging conditions and fluorescence signal formation conditions; they are all physical quantities and equipment settings describing the imaging process itself.

[0031] The output of this method is intermediate characteristic parameters that characterize tissue structure and fluorescence response. It does not directly lead to a disease diagnosis. The final judgment is made independently by the physician in combination with other examinations.

[0032] For ease of explanation, the following terms are defined as follows.

[0033] 1. Video data: A sequence of images continuously output by the endoscopic device during a single examination. In this article, white light video data, fluorescence pseudo-color video data, and native fluorescence video data refer to the video data of their respective modalities.

[0034] 2. Video Segment: A data segment containing a preset number of frames, continuously extracted in chronological order from a single video stream. In this embodiment, each video segment contains 16 frames.

[0035] 3. Spatiotemporal Data Unit: A data block obtained by dividing a video segment along the temporal and spatial directions. Each spatiotemporal data unit consists of pixels located at the same spatial grid position in a preset number of consecutive video frames, and is the smallest input division unit of the video encoder.

[0036] 4. Spatiotemporal Feature Unit: A feature vector output by the video encoder after encoding a spatiotemporal data unit, which corresponds one-to-one with that spatiotemporal data unit.

[0037] 5. Site: A collective term for the central and peripheral sites of the lesion. Each site corresponds to a local analysis region.

[0038] 6. Occlusion Reconstruction: This method involves placing some spatiotemporal data units in a occluded state in a video segment, inputting only the unoccluded spatiotemporal data units into the video encoder, and having the reconstruction decoder predict the data of the occluded spatiotemporal data units based on the spatiotemporal feature units output by the encoder. The parameters are then updated based on the difference between the prediction results and the actual data.

[0039] 7. Fixed-Teacher Encoder vs. Student Encoder: A video encoder whose parameters remain fixed during a single transfer training iteration and whose output features serve as the source of supervisory signals is called a fixed-teacher encoder; a video encoder whose parameters are updated is called a student encoder. Gradient propagation stops at the output features of a fixed-teacher encoder.

[0040] 8. Training Objectives and Losses: Each training objective in this paper is quantified by a corresponding loss function. For example, the local cross-modal distillation objective is quantified by the local cross-modal distillation loss. When updating parameters, the direction that reduces the corresponding loss is the optimization direction of the training objective.

[0041] Figure 1 This is a flowchart of the overall process for processing endoscopic multimodal video data provided in this embodiment of the invention, as follows: Figure 1 As shown, the method includes the following steps: Step S1: Acquire white light video data, fluorescence pseudo-color video data and original fluorescence video data generated during the same endoscopic examination, and divide the three video data into spatiotemporal data units according to a unified time index and spatial grid, so that the spatiotemporal data units with the same time index and spatial grid position in the three video data correspond to each other. Step S2: Using the white light video encoder as the fixed teacher encoder and the fluorescent pseudo-color video encoder as the student encoder, the corresponding white light spatiotemporal data units and fluorescent pseudo-color spatiotemporal data units are input into the two respectively. Based on the difference of the obtained features after feature projection, a cross-modal feature constraint is constructed. The fluorescent pseudo-color video encoder is updated together with the cross-modal feature constraint and the occlusion reconstruction target of the fluorescent pseudo-color spatiotemporal data unit. Step S3: Using the updated fluorescence pseudo-color video encoder as the fixed teacher encoder and the native fluorescence video encoder as the student encoder, determine the fluorescence reliability weight for each native fluorescence spatiotemporal data unit, and weight the feature differences between the corresponding fluorescence pseudo-color spatiotemporal data units and the native fluorescence spatiotemporal data units after feature projection according to the fluorescence reliability weight to obtain the local cross-modal distillation target, and update the native fluorescence video encoder accordingly. Step S4: Input the three corresponding spatiotemporal data units into the white light video encoder, the updated fluorescent pseudo-color video encoder, and the updated native fluorescent video encoder respectively. Combine the obtained features into a three-modal feature group according to the unified time index and spatial grid and input it into the three-modal fusion network to output the three-modal fusion features.

[0042] Figure 2 This is a flowchart of the phased transfer training provided in an embodiment of the present invention, such as... Figure 2 As shown, steps S2 and S3 constitute a phased transfer training process from white light video data through fluorescent pseudo-color video data to native fluorescence video data. This phased transfer is used instead of a direct transfer between white light video data and native fluorescence video data for the following reasons: White light video data is three-channel color data, and its pixel values ​​reflect the reflectivity of tissue under white light illumination; native fluorescence video data is single-channel data, and its pixel values ​​reflect the fluorescence intensity emitted by tissue under excitation light. The two differ significantly in the number of channels, numerical distribution, and the type of information they carry. If the white light video encoder is directly used as the fixed teacher encoder and the native fluorescence video encoder as the student encoder, the tissue structure information expressed by the teacher-side features will not have corresponding signal components in the student-side input. Forced alignment will compel the native fluorescence video encoder to fit information that does not exist in its input. Fluorescent pseudo-color video data retains the organizational structure information of white light video data in the luminance component and carries the response distribution information of the original fluorescence signal in the chromaticity component. In terms of data distribution, it is in between the two, so it can be used as a transfer medium to split a large-span transfer into two smaller-span transfers: the transfer in step S2 establishes a correspondence in the organizational structure information, and the transfer in step S3 establishes a correspondence in the fluorescence response information.

[0043] The adoption of the above transfer method without training the three video encoders independently is also based on the following: the amount of clinically available white light video data is usually significantly greater than that of fluorescence video data. If the native fluorescence video encoder is trained from a completely random initial state, its available training samples are insufficient to support the spatiotemporal representation learning of the same scale as that of the white light video encoder.

[0044] The steps described above will be explained below.

[0045] In step S1 above, white light video data, fluorescence pseudo-color video data and native fluorescence video data generated during the same endoscopic examination are acquired, and the corresponding timestamps, channel calibration parameters, equipment imaging conditions and fluorescence drug administration information are acquired.

[0046] White light video data reflects tissue color, surface texture, blood vessel morphology, and lesion outlines. Native fluorescence video data records the intensity distribution of fluorescence signals and their changes over time; it is single-channel data, and its pixel value corresponds to the fluorescence intensity emitted by the corresponding tissue region under fluorescence excitation and received by the sensor. Fluorescence pseudocolor video data is used to display the spatial relationship between fluorescence signals and tissue structures.

[0047] The fluorescence pseudo-color video data is generated by superimposing white light and fluorescence images after spatial registration using an endoscopic device. The generation process is as follows: first, the intensity value of the original fluorescence signal is converted into pseudo-color components according to a preset color mapping relationship; then, the pseudo-color components are superimposed on the registered white light tissue image. During superposition, the luminance component of the white light image remains unchanged, while the color components are jointly determined by the white light luminance and fluorescence luminance. The white light luminance determines the hue of the pseudo-color, and the fluorescence luminance determines the saturation of the pseudo-color. The superposition process is adaptively adjusted according to the image statistics.

[0048] Therefore, fluorescent pseudo-color video data fully preserves the structural information of white light video data in the luminance component, while carrying the response distribution information of the native fluorescence signal in the chromaticity component. In terms of data distribution, fluorescent pseudo-color video data simultaneously contains structural components from white light video data and fluorescence response components from native fluorescence video data, placing it between the two. Fluorescent pseudo-color video data participates in subsequent feature learning as an independent video modality; these properties form the basis for its use as a transfer mediator between white light video data and native fluorescence video data in the following section.

[0049] The time correspondence between the three video data streams is established as follows.

[0050] The sensor of the endoscopic device has a nominal frame rate of 60 frames per second. When using a dual-sensor beam splitting scheme, the white light channel and fluorescence channel are simultaneously exposed at 60 frames per second, with no time difference between white light frames and fluorescence frames at the same acquisition moment. When using a single-sensor time-division scheme, the sensor alternately acquires white light frames and fluorescence frames at 60 frames per second. The actual frame rate of both white light video data and native fluorescence video data is 30 frames per second. Within the same acquisition cycle, there is a fixed time offset of approximately 16.6 milliseconds between adjacent white light frames and fluorescence frames. A frame-level correspondence is established according to this pre-calibrated time offset. Fluorescence pseudo-color video data is generated from the corresponding white light frames and fluorescence frames, and its frame rate is consistent with that of the native fluorescence video data. The resolution of the video data is 1920×1080 pixels or 3840×2160 pixels.

[0051] The three video streams are aligned at the frame or segment level based on their timestamps. When the frame rates of the three video streams are different or there is a fixed time offset, a correspondence is established using adjacent timestamp matching, time interpolation, or a pre-defined time offset.

[0052] After frame-level synchronization is completed, time sampling is performed on each video data stream to form an input video segment with an effective frame rate of 15 frames per second. When the actual frame rate of the modal video data is 30 frames per second, one frame is selected every two frames; when the actual frame rate is 60 frames per second, one frame is selected every four frames. Each input video segment consists of 16 frames, covering approximately 1.07 seconds of continuous observation time. An 8-frame overlap is set between adjacent input video segments, meaning that each time 8 frames are moved along the time direction to extract a new 16-frame video segment, corresponding to approximately 0.53 seconds. The same set of three video segments uses the same start and end times and the same time sampling position, thus frames with the same sequence number in the three video segments correspond to the same acquisition time.

[0053] The spatial correspondence between the three video data streams is established by the device calibration relationship or a pre-determined spatial transformation relationship. Based on the channel calibration parameters, pixel correspondence, or pre-determined spatial transformation relationship provided by the endoscope device, the three video frames at the same acquisition time are mapped to a unified reference coordinate system. Corresponding video segments are divided using the same time range, spatial range, input size, and spatiotemporal data unit partitioning rules.

[0054] In this embodiment, each frame of the three video clips is resized to 224×224 pixels and divided into 14×14 spatial grids of 16 pixels × 16 pixels along both the height and width directions. Spatial grids with the same row and column numbers in the three video clips correspond to the same organizational region after being mapped using the aforementioned unified reference coordinate system.

[0055] Each video segment is divided into spatiotemporal data units, with two frames grouped along the temporal direction and the aforementioned 14×14 spatial grids along the spatial direction. Each spatiotemporal data unit contains a 16-pixel × 16-pixel region located at the same spatial grid position in two consecutive frames.

[0056] Each spatiotemporal data unit consists of triples Identification. Among them, Represents the time index, with values ​​ranging from 1 to 8, the first... The time index corresponds to the first time segment in the video clip. Frame and the frame; Indicates the row number of the spatial grid. The column number represents the spatial grid; both values ​​are integers from 1 to 14. The index of the spatiotemporal data unit. The correspondence between the triples is as follows: , Where 196 represents the total number of spatial grids corresponding to one time index, i.e. ; 14 is the number of columns in the spatial grid. The value of is an integer from 1 to 1568, corresponding one-to-one with the triplet. The total number of spatiotemporal data units within a video segment is denoted as . In this embodiment .

[0057] The three video segments use the exact same division rules described above. Therefore, in the white light spatiotemporal data unit, the fluorescence pseudo-color spatiotemporal data unit, and the native fluorescence spatiotemporal data unit, all time-indexed segments... Spatial grid position Those that are identical correspond to the same organizational region within the same data acquisition period, forming three corresponding spatiotemporal data units. In the following text, "corresponding" spatiotemporal data units or spatiotemporal feature units refer to those with the same time index and spatial grid position, i.e., sequence number. same.

[0058] Because the three video encoders output the spatiotemporal feature units of the same video segment with the same sequence number. The three spatiotemporal feature units with the same sequence number are also mutually corresponding. This is the basis for the direct combination of the three video data at the feature level according to their positions.

[0059] White light video data and fluorescent pseudo-color video data are in three-channel RGB format. Pixel data is stored in 8-bit unsigned integer format, converted to floating-point data and normalized according to their respective input normalization parameters before being input to the video encoder.

[0060] The native fluorescence video data is stored in a single-channel grayscale format, using a 16-bit unsigned integer container, with an effective signal bit width of 12 or 10 bits. The native fluorescence video data is normalized according to the device's black level and the upper limit of the device's effective signal, without compressing the native fluorescence signal into an 8-bit image to avoid loss of fluorescence signal intensity resolution during the quantization process.

[0061] The three-modal video model used in performing steps S2 to S4 above is configured as follows.

[0062] A three-modal video model is constructed, comprising a white light video encoder, a fluorescent pseudo-color video encoder, a native fluorescent video encoder, a feature projection layer, a three-modal fusion network, a conditional coding network, and a lesion-level fusion network. The connections between the components are as follows: Figure 2 and Figure 4 As shown: Three video encoders receive spatiotemporal data units of their respective modalities and output spatiotemporal feature units; the output of each video encoder is connected to the corresponding feature projection layer; during the training phase, the output of the feature projection layer of the white light video encoder is connected to the output of the fluorescent pseudo-color video encoder to construct cross-modal feature constraints, and the output of the feature projection layer of the fluorescent pseudo-color video encoder is connected to the output of the feature projection layer of the native fluorescent video encoder to construct the local cross-modal distillation target and the video segment-level cross-modal distillation target described below; the outputs of the three feature projection layers are jointly connected to the input of the three-modal fusion network; the outputs of the three-modal fusion network and the conditional coding network are jointly connected to the input of the lesion-level fusion network.

[0063] The three video encoders employ the same spatiotemporal coding structure, each including a video input layer and a spatiotemporal feature coding layer. The white light video encoder and the fluorescent pseudo-color video encoder receive three-channel color video segments, while the native fluorescent video encoder receives a single-channel fluorescent video segment. The parameters of the three video encoders are independent of each other, and the phased transfer relationship is established only through parameter initialization, modal adaptation layers, and cross-modal feature constraints, thus allowing each mode to retain its own signal characteristics.

[0064] In this embodiment, all three video encoders employ a video Transformer structure based on spatiotemporal data unit embedding. Each video encoder includes a spatiotemporal data unit embedding layer, twelve sequentially connected spatiotemporal feature coding layers, a spatiotemporal position coding layer, and an output normalization layer. The spatiotemporal feature coding layers in this embodiment are Transformer coding layers, and are sequentially referred to as the first spatiotemporal feature coding layer to the twelfth spatiotemporal feature coding layer according to their connection order.

[0065] The spatiotemporal data unit embedding layer is implemented using 3D convolution, with the kernel size and stride both set to 2 frames × 16 pixels × 16 pixels, consistent with the aforementioned spatiotemporal data unit partitioning rules. This maps each spatiotemporal data unit to a 768-dimensional spatiotemporal feature unit. A video clip thus forms a feature sequence composed of 1568 768-dimensional spatiotemporal feature units.

[0066] Spatiotemporal location coding includes trainable temporal location coding and spatial location coding, which respectively represent the temporal index and spatial grid position of each spatiotemporal feature unit in the video segment. The result is added to the output of the spatiotemporal data unit embedding layer and then fed into the first spatiotemporal feature coding layer.

[0067] The hidden feature dimension of each spatiotemporal feature coding layer is 768. The multi-head self-attention module includes 12 attention heads, each with a feature dimension of 64. The intermediate hidden dimension of the feedforward network is 3072, using a Gaussian error linear unit activation function. Each spatiotemporal feature coding layer employs layer normalization, residual connections, and a random depth mechanism, with a maximum random depth rate set to 0.1. The output of the twelfth spatiotemporal feature coding layer is then normalized by an output normalization layer to serve as the output feature of the video encoder.

[0068] The spatiotemporal data unit embedding layer of the white-light video encoder and the fluorescent pseudo-color video encoder receives three input channels; the spatiotemporal data unit embedding layer of the native fluorescent video encoder receives one input channel, and the rest of the structure is the same as the other two video encoders. Including trainable temporal and spatial location encodings, and without calculating the parameters of the reconstruction decoder, feature projection layer, trimodal fusion network, and downstream task output layer, the white-light video encoder and the fluorescent pseudo-color video encoder have approximately 86.39 million parameters, the native fluorescent video encoder has approximately 85.61 million parameters, and the total number of parameters for the three video encoders is approximately 258 million.

[0069] The feature projection layer is used to map the features output by different video encoders to a common feature space of the same dimension, so that cross-modal feature constraints and cross-modal distillation only apply to information common to the relevant modes. In this embodiment, the feature projection layer is a single-layer linear mapping structure with L2 normalization, mapping 768-dimensional spatiotemporal feature units to 256 dimensions. Each spatiotemporal feature unit acts independently without changing the number or arrangement order of the spatiotemporal feature units.

[0070] It should be noted that the specific form of the feature projection layer is not limited to the single-layer linear mapping followed by L2 normalization described above. Any trainable mapping that can map the output features of different video encoders to a common feature space of the same dimension, and where each spatiotemporal feature unit acts independently without changing the number or order of the spatiotemporal feature units, can be used as the feature projection layer. For example, a multilayer perceptron including hidden layers and nonlinear activation functions can be used; layer normalization or batch normalization can be used instead of L2 normalization after linear mapping; and trainable scaling coefficients can be introduced before and after mapping. The feature projection layers described below are also applicable to the various forms mentioned above.

[0071] Before performing step S2 above, the white light video encoder is first subjected to occlusion reconstruction self-supervised pre-training.

[0072] Continuous video segments are extracted from white-light video data. Each video segment is divided into spatiotemporal data units according to the segmentation rules described above, and some of these spatiotemporal data units are randomly occluded. The unoccluded spatiotemporal data units are input into the white-light video encoder. The white-light reconstruction decoder predicts the pixel values ​​or low-dimensional image representations of the occluded spatiotemporal data units and updates the parameters based on the difference between the prediction results and the original spatiotemporal data units. The white-light reconstruction decoder can employ a lightweight decoding structure consisting of several spatiotemporal feature coding layers. The decoding prediction for occlusion reconstruction is a well-known self-supervised training method, which will not be elaborated here. After pre-training, the white-light video encoder is retained, while the white-light reconstruction decoder is removed or disabled. The white-light video encoder obtained through the above self-supervised pre-training for occlusion reconstruction is the white-light video encoder described below.

[0073] This embodiment employs a random tubular occlusion method. For each video segment forming 14×14 spatial grid locations, 176 spatial grid locations are randomly selected during each training iteration. The spatiotemporal data units corresponding to these selected spatial grid locations across all eight time indices are simultaneously occluded, resulting in the occlusion of 1408 spatiotemporal data units. Only 160 spatiotemporal data units remain as visible input, representing an occlusion ratio of approximately 89.8%. Using tubular occlusion instead of individually occluding each spatiotemporal data unit avoids the model relying solely on the content of the same spatial location in adjacent frames for reconstruction.

[0074] In this embodiment, the total duration of unlabeled white light endoscopy video data used for pre-training is approximately 300 hours. After deleting severely blurred, long-duration, frozen, and device interface-dominated video segments, approximately 500,000 consecutive video clips were randomly extracted from the valid video data as pre-training samples. Highly overlapping segments from the same original examination video data were grouped into the same data subset to avoid data leakage between the training and validation sets.

[0075] The white-light video encoder was trained using the AdamW optimizer, with a first-order moment decay factor of 0.9, a second-order moment decay factor of 0.95, and a weight decay factor of 0.05. The base learning rate was set to... The first 10 training rounds used a linear learning rate for warm-up, followed by a cosine annealing strategy to gradually reduce the learning rate. The training run consists of 100 epochs, with a global batch size of 128. When a single computing device cannot accommodate this batch size, parallel training across multiple devices or gradient accumulation is used to obtain an equivalent global batch size.

[0076] In step S2 above, the spatiotemporal representation capability learned by the white light video encoder is transferred to the fluorescent pseudo-color video encoder.

[0077] As an optional implementation, the cross-modal feature constraint is determined by the distance between the features output by the white light video encoder and the fluorescent pseudo-color video encoder in the same intermediate spatiotemporal feature coding layer after projection and normalization. The modality adaptation layer is a bottleneck-type residual structure set at the output end of each spatiotemporal feature coding layer, which reduces the dimensionality of the input features and then increases the dimensionality, and adds it to the original output of the spatiotemporal feature coding layer without changing the number and arrangement order of feature units. When updating the fluorescent pseudo-color video encoder, the fixed state of the parameters obtained by copying is released in stages. First, only the input normalization parameters and the modality adaptation layer are updated, and then the fixed state of each spatiotemporal feature coding layer is gradually released from the output end to the input end. These are explained below.

[0078] A fluorescent pseudo-color video encoder corresponding to the structure of a white light video encoder is established. The spatiotemporal feature coding layer parameters of the pre-trained white light video encoder are copied to the fluorescent pseudo-color video encoder as its initial parameters. Independent input normalization parameters and modal adaptation layers are set for the fluorescent pseudo-color video encoder.

[0079] In this embodiment, the parameters of the spatiotemporal data unit embedding layer, spatiotemporal position encoding, twelve spatiotemporal feature encoding layers, and output normalization layer of the pre-trained white light video encoder are copied to the fluorescence pseudo-color video encoder. The input normalization parameters are the mean and standard deviation used to normalize the pixel values ​​of each channel of the input video segment. The fluorescence pseudo-color video encoder uses the values ​​obtained from the statistics of the fluorescence pseudo-color video data, instead of using the values ​​from the white light video data, to adapt to the changes in the numerical distribution of each channel after the pseudo-color is superimposed.

[0080] The modality adaptation layer employs a bottleneck residual structure, which is positioned at the output of each spatiotemporal feature coding layer. For each 768-dimensional spatiotemporal feature unit, the feature dimension is first reduced from 768 to 192 dimensions through a first linear mapping layer. After activation by a Gaussian error linear unit, the feature dimension is then restored from 192 to 768 dimensions through a second linear mapping layer. The restored feature is then added to the original output of the spatiotemporal feature coding layer. The modality adaptation layer only reduces and then increases the feature dimension, without changing the number or arrangement order of the spatiotemporal feature units, thus preserving the temporal index and spatial grid correspondence established above.

[0081] The role of the modality adaptation layer is to compensate for the data distribution differences between fluorescent pseudo-color video data and white light video data while maintaining the parameters of the copied spatiotemporal feature coding layer. Its specific structure is not limited to the bottleneck residual structure described above. Any trainable structure added to the spatiotemporal feature coding layer, with fewer parameters than the spatiotemporal feature coding layer, and without changing the number and order of spatiotemporal feature units, can be used as the modality adaptation layer. For example, a bypass mapping structure connected in parallel with the spatiotemporal feature coding layer can be used, adding the bypass output to the original output of the spatiotemporal feature coding layer; a structure that applies trainable scaling coefficients and offsets to each feature channel of the spatiotemporal feature coding layer's output can also be used; a linear adaptation structure in the form of low-rank decomposition can also be used, where the adaptation transformation is represented by the product of two low-rank matrices. All of the above structures can compensate for the data distribution differences between modalities without destroying the transferred spatiotemporal representation capabilities, achieving the same technical effect.

[0082] The pre-trained white light video encoder is used as the fixed teacher encoder, and the fluorescent pseudo-color video encoder is used as the student encoder. Corresponding white light spatiotemporal data units and fluorescent pseudo-color spatiotemporal data units are input into the two encoders respectively. The features output by the two video encoders in the same intermediate spatiotemporal feature coding layer are taken, projected and normalized by their respective feature projection layers, and the distance between them is calculated according to the same time index and spatial grid position to construct cross-modal feature constraints.

[0083] In this embodiment, both video encoders select the output of the eighth spatiotemporal feature coding layer as the features for constructing cross-modal feature constraints. The intermediate layer is selected instead of the output layer because the features of the intermediate layer are still mainly local organizational structures and textures, and the proportion of information shared by the two modalities at this level is relatively high; the features of the output layer already contain more modality-specific semantic components. If constraints are applied to the output layer, it will force the fluorescent pseudo-color video encoder to suppress the fluorescence response information carried by its chroma components.

[0084] The corresponding spatiotemporal feature units output by the two video encoders are mapped from 768 dimensions to 256 dimensions through their respective linear feature projection layers and then L2 normalized. The distance includes cosine distance and mean square error, which are calculated separately and combined with a weight of 0.5:0.5 to obtain the cross-modal feature constraint loss.

[0085] Cross-modal feature constraints only require that the features of the two modalities after projection into the common feature space be close to each other, without requiring that all features of the two modalities be consistent. Therefore, while maintaining the common organizational structure information of white light video data and fluorescent pseudo-color video data, the fluorescent pseudo-color video encoder is allowed to retain its unique fluorescence response information.

[0086] The same spatiotemporal data unit occlusion method as described above is applied to the fluorescent pseudocolor video clip. The occluded spatiotemporal data units are predicted by the fluorescent pseudocolor reconstruction decoder. The reconstruction loss of the fluorescent pseudocolor spatiotemporal data units is obtained based on the difference between the prediction result and the original spatiotemporal data units, which is the occlusion reconstruction target of the fluorescent pseudocolor spatiotemporal data units.

[0087] The total training loss of the fluorescence pseudocolor video encoder consists of the fluorescence pseudocolor spatiotemporal data unit reconstruction loss and the cross-modal feature constraint loss, where the weight of the fluorescence pseudocolor spatiotemporal data unit reconstruction loss is set to 1, and the weight of the cross-modal feature constraint loss is set to 0.5. The fluorescence pseudocolor video encoder is updated using this total training loss, enabling it to adapt to the data distribution that integrates tissue structure and fluorescence display while inheriting the spatiotemporal representation capabilities of white light video data.

[0088] When updating the fluorescent pseudocolor video encoder, the fixed state of the parameters obtained by copying is removed in stages: first, only the input normalization parameters and modality adaptation layer are updated, and then the fixed state of each spatiotemporal feature coding layer is gradually removed from the output end to the input end.

[0089] In this embodiment, the fluorescence pseudocolor video encoder is trained for 50 rounds. The same optimizer and learning rate settings as the white light video encoder pre-training are used in this stage. In training rounds 1 to 10, the entire structure is fixed and copied, and only the input normalization parameters, modality adaptation layer, feature projection layer, and fluorescence pseudocolor reconstruction decoder of the fluorescence pseudocolor video data are updated. In training rounds 11 to 30, the fixed state of the seventh to twelfth spatiotemporal feature coding layers and the output normalization layer is removed, while the first to sixth spatiotemporal feature coding layers, the spatiotemporal data unit embedding layer, and the spatiotemporal position coding remain fixed. In training rounds 31 to 50, the fixed state of the first to sixth spatiotemporal feature coding layers, the spatiotemporal data unit embedding layer, and the spatiotemporal position coding is further removed, and the fluorescence pseudocolor video encoder is jointly trained as a whole.

[0090] The reason for removing the fixed state in the above order is that the features extracted by the spatiotemporal feature coding layer near the output end have strong modal correlation, while the features extracted by the spatiotemporal feature coding layer near the input end are mainly based on general spatiotemporal structures. Updating the former first and then the latter can reduce the degree to which the transferred general spatiotemporal representation ability is destroyed while completing modal adaptation.

[0091] Throughout the entire transfer training process, the white light video encoder remains fixed, and its output features stop gradient propagation.

[0092] Before performing step S3 above, the native fluorescent video encoder is first subjected to occlusion reconstruction self-supervised pre-training.

[0093] A native fluorescence video encoder is established, and independent video input layers and input normalization parameters are set for it based on the number of channels, bit depth, and effective signal range of the native fluorescence video data. The native fluorescence video encoder does not copy the video input layer parameters of the white light video encoder or the fluorescence pseudo-color video encoder, because the native fluorescence video data is single-channel and its numerical range is different from that of three-channel color data. Copying the above parameters would introduce incorrect channel correspondences.

[0094] Continuous video segments are extracted from native fluorescence video data. Spatiotemporal data unit masking is applied to these video segments. The fluorescence signal value or low-dimensional fluorescence representation of the masked spatiotemporal data units is predicted using a native fluorescence reconstruction decoder. The native fluorescence video encoder is updated based on the difference between the predicted result and the original fluorescence signal, allowing it to first learn the spatial distribution and temporal variation of the native fluorescence signal. The native fluorescence video encoder obtained through the above masking reconstruction self-supervised pre-training is the native fluorescence video encoder described below.

[0095] In this embodiment, the self-supervised pre-training of the native fluorescent video encoder adopts the same occlusion method and training settings as the pre-training of the white light video encoder, enabling the three video encoders to be trained under the same self-supervised task difficulty, which facilitates subsequent feature alignment. For each video segment forming 14×14 spatial grid positions, 176 spatial grid positions are randomly selected during each training session, and the spatiotemporal data units corresponding to the selected spatial grid positions on all 8 time indices are simultaneously occluded, that is, 1408 spatiotemporal data units are occluded, and 160 spatiotemporal data units are retained as visible inputs, with an occlusion ratio of approximately 89.8%.

[0096] The input native fluorescence video data is normalized according to the device black level and the upper limit of the device's effective signal, and is not compressed into an 8-bit image. The native fluorescence reconstruction decoder predicts the normalized fluorescence signal value of the occluded spatiotemporal data unit, and calculates the reconstruction loss of the native fluorescence spatiotemporal data unit based on the mean square error between the predicted value and the actual normalized signal value.

[0097] The native fluorescence video encoder was trained using the AdamW optimizer with a first-order moment decay factor of 0.9, a second-order moment decay factor of 0.95, and a weight decay factor of 0.05. The base learning rate was... The first 10 training rounds used a linear learning rate for warm-up, followed by a cosine annealing strategy to gradually reduce the learning rate. The training run consists of 100 epochs with a global batch size of 128. During implementation, the learning rate, number of training epochs, and other training hyperparameters can be adaptively adjusted based on the actual scale of the native fluorescent video data.

[0098] In this embodiment, the total duration of the unlabeled native fluorescence video data used for pre-training the native fluorescence video encoder is approximately 30 hours. After removing low-quality video segments, approximately 40,000 consecutive video clips were randomly extracted as pre-training samples. The data size is significantly smaller than that used for pre-training the white light video encoder. Highly overlapping segments from the same original inspection video data are grouped into the same data subset.

[0099] After completing the pre-training described above, in step S3, the updated fluorescence pseudo-color video encoder is used as the fixed teacher encoder, and the native fluorescence video encoder is used as the student encoder. Corresponding fluorescence pseudo-color video segments and native fluorescence video segments are obtained and input into the fixed teacher encoder and student encoder, respectively. The features output by the two video encoders are mapped to the fluorescence-related feature space shared by the two modalities through their respective feature projection layers.

[0100] A fluorescence reliability weight is determined for each native fluorescence spatiotemporal data unit. Based on this weight, the feature differences between corresponding fluorescence pseudocolor spatiotemporal data units and native fluorescence spatiotemporal data units after feature projection are weighted to obtain local cross-modal distillation targets, which are then used to update the native fluorescence video encoder. The process of determining the fluorescence reliability weight and constructing each training target are explained below.

[0101] In endoscopic fluorescence imaging, the spatiotemporal locations of native fluorescence video data are not equally usable: surgical instruments obstruct parts of the field of view, there are ineffective imaging areas at the edges of the field of view, and pixels in strongly reflective areas reach the upper limit of the device's effective signal and no longer change with the incident fluorescence intensity. The signals at these locations do not reflect the fluorescence response of the corresponding tissue regions, and there is no true correspondence between their characteristics and the characteristics of the same spatiotemporal location in the fluorescence pseudocolor video data. If all spatiotemporal locations are treated equally during distillation, these erroneous correspondences will be written into the native fluorescence video encoder. Therefore, a fluorescence reliability weight is first determined for each native fluorescence spatiotemporal data unit, and then the feature differences are weighted according to this weight.

[0102] As an optional implementation, the fluorescence reliability weight is determined as follows: the median of the pixels with the lowest signal value within the video frame containing the spatiotemporal data unit is used as the background signal value of that video frame; the effective pixels are the pixels within the spatiotemporal data unit after excluding pixels obstructed by instruments, pixels with missing data, invalid devices, and saturated pixels. After background subtraction of the effective pixels within the spatiotemporal data unit, the highest and lowest signal values ​​of the preset proportion are removed, and the average value of the remaining signal values ​​is taken as the fluorescence signal intensity, and the fluorescence signal intensity distribution of each spatiotemporal data unit that meets the quality conditions within the same video segment is used. The data is normalized, and the quality condition is that the effective pixel ratio is not lower than the first threshold and the saturated pixel ratio is lower than the second threshold. Pixels whose signal values ​​reach a preset proportion of the device's effective signal upper limit are defined as saturated pixels, and a saturation penalty coefficient that decreases as the saturated pixel ratio increases is determined. The product of the normalized fluorescence signal intensity, the effective pixel ratio, and the saturation penalty coefficient is used as the initial weight. For spatiotemporal data units that meet the quality condition, the initial weight is limited to between a preset lower limit and 1, and the fluorescence reliability weight of each spatiotemporal data unit within the same video segment is normalized. These are explained below.

[0103] In native fluorescence video data, the signal value of each pixel consists of two parts: one part comes from the fluorescence emitted by the observed tissue under excitation light, and the other part comes from background factors unrelated to the tissue fluorescence response, such as device dark current, excitation light leakage, and intracavity scattering. Only by subtracting the latter can the signal intensity reflecting the tissue fluorescence response be obtained.

[0104] For the first in the native fluorescence video clip frame, This indicates the frame number. First, pixels obstructed by equipment, pixels with missing data, invalid equipment, and pixels that have reached the saturation threshold are excluded. The remaining pixels are then used as candidate pixels for background signal estimation. These candidate pixels are sorted from lowest to highest signal value. A preset proportion of pixels with the lowest signal values ​​are selected, and the median of the signal values ​​from this subset is used as the background signal value for that video frame. : , in, Represents pixels The original fluorescence signal value, that is, the original readout value of the pixel without any processing; Indicates the first The set of candidate pixels for background signal estimation with the lowest preset proportion in the frame; in this embodiment, the preset proportion is 5%. This indicates that the median of the elements in the set is taken. The pixels with the lowest signal values ​​are selected because these pixels correspond to the areas with the weakest fluorescence response in the field of view, and their signals can be approximated as consisting only of background. The median is used instead of the average value to avoid individual outlier pixels affecting the estimation results.

[0105] When the number of candidate pixels for background signal estimation is less than a preset number, or when there is no low-intensity region within the video frame that can be used for background estimation, the median of the pixel signal values ​​within a pre-marked background reference region is used as the background signal value; when there is no background reference region, the pre-measured and stored device background signal value is used. In this embodiment, the preset number is 100.

[0106] For the A spatiotemporal data unit is defined as a set of pixels covered by the spatiotemporal data unit. Let the set of pixels consisting of device occlusion pixels, missing data pixels, and invalid device pixels be denoted as The set of pixels that have reached the device saturation threshold is denoted as The set of valid pixels obtained after excluding the above pixels is: , Among them, symbols This represents the difference operation between sets. The instrument-occluded pixels are pixels that are obscured by surgical instruments and not imaged on the tissue surface. The data-missing pixels are pixels that did not obtain a valid signal value due to transmission or decoding reasons. The device-invalid pixels are pixels located outside the circular field of view of the endoscope imaging or within the device information overlay area. The saturated pixels are pixels whose signal values ​​have reached the upper limit of the device's valid signal. The common feature of these four types of pixels is that their signal values ​​do not reflect the actual fluorescence response intensity of the corresponding tissue area.

[0107] In this embodiment, pixels whose signal values ​​reach a preset percentage of the device's effective signal limit are defined as saturated pixels, where the preset percentage is 98%. When the device provides saturation calibration parameters, the saturation threshold calibrated by the device is used preferentially. The specific calculation method for the saturation threshold is described in step S8 below.

[0108] For sets For each pixel in the image, background subtraction is performed based on the background signal value of the video frame in which that pixel resides. Background subtraction signal value Calculate according to the following formula: , in, Represents pixels The frame number of the video frame to which it belongs. This is the background signal value for this frame; This indicates that the larger of the two terms within the parentheses is taken to avoid negative signal values ​​when the background estimate is too high.

[0109] For the first After background subtraction, the signal values ​​of all valid pixels within a spatiotemporal data unit are sorted. The signal values ​​with the highest and lowest preset ratios are removed, and the average of the remaining signal values ​​is calculated as the fluorescence signal intensity after background subtraction for that spatiotemporal data unit. In this embodiment, both the highest preset ratio and the lowest preset ratio are set to 10%. Removing the signal values ​​at both ends is to eliminate the influence of unidentified strong reflective points and local residual background on the overall fluorescence intensity of the spatiotemporal data unit.

[0110] Within the same native fluorescence video segment, calculate the 5th percentile of fluorescence signal intensity from spatiotemporal data units that meet quality requirements. and the 95th percentile The quality conditions are: the effective pixel ratio is not lower than a first threshold and the saturated pixel ratio is lower than a second threshold. In this embodiment, the first threshold is 0.50 and the second threshold is 0.20. Normalized fluorescence signal intensity of each spatiotemporal data unit Calculate according to the following formula: , in, The value range is from 0 to 1; To prevent constants with zero denominators, this embodiment takes... ; Indicates when Less than Time to take ,when Greater than Time to take In other cases, take The truncation function, This is the independent variable of the function. The normalization is based on the fluorescence signal intensity distribution within the same video segment, thus the obtained... It represents the relative intensity of the fluorescence signal of the spatiotemporal data unit within this segment, and is not affected by differences in fluorescence reagent dosage, excitation power, and channel gain between different examinations.

[0111] No. Effective pixel percentage of each spatiotemporal data unit Calculate according to the following formula: , in, This indicates the number of valid pixels, i.e., the number of elements in the set; This indicates the total number of pixels contained in the spatiotemporal data unit. The value range is from 0 to 1.

[0112] No. Saturation pixel ratio of each spatiotemporal data unit Calculate according to the following formula: , in, Indicates the number of saturated pixels. The value range is from 0 to 1.

[0113] Determine the saturation penalty coefficient based on the proportion of saturated pixels. This embodiment uses a piecewise linear approach to reduce the weights of saturated pixels: , Where 0.20 is the aforementioned second threshold. When there are no saturated pixels in this spatiotemporal data unit, Set to 1; as the proportion of saturated pixels increases, Linear decrease; when the proportion of saturated pixels reaches or exceeds the second threshold. Take 0. The physical meaning is: the degree to which the fluorescence signal intensity information of the spatiotemporal data unit is not lost due to sensor saturation.

[0114] Based on the normalized fluorescence signal intensity, effective pixel ratio, and saturation penalty coefficient, the first... Initial weights of each spatiotemporal data unit : , Further determine the fluorescence reliability weight of this spatiotemporal data unit based on quality conditions. : , Wherein, 0.50 is the first threshold, 0.20 is the second threshold, and 0.05 is the preset lower limit. For spatiotemporal data units with an effective pixel ratio lower than the first threshold or a saturated pixel ratio not lower than the second threshold, their fluorescence reliability weight is set to zero, and these spatiotemporal data units do not participate in cross-modal distillation. For spatiotemporal data units that meet the quality conditions, their initial weight is limited to between the preset lower limit and 1. Setting the preset lower limit of 0.05 is to prevent spatiotemporal data units with low fluorescence intensity but still valid spatial correspondence from being completely excluded.

[0115] The intrinsic correlation between the above three quantities and the reliability of the fluorescence signal is as follows: The fluorescence signal intensity after background subtraction reflects the relative strength of the actual tissue fluorescence signal received at that spatiotemporal location compared to the background. The higher the intensity, the more significant the fluorescence response at that location is relative to readout noise, and the closer the fluorescence distribution expressed by its characteristics is to the true fluorescence response of the tissue. The effective pixel ratio reflects the proportion of pixels at that spatiotemporal location that are not obstructed by instruments, have not experienced data loss, and do not fall into the invalid imaging area of ​​the device. The larger the ratio, the more completely the signal at that location originates from the observed tissue, and the more consistent it is with the tissue region corresponding to the same spatiotemporal location in the fluorescence pseudocolor video data. The saturated pixel ratio reflects the proportion of pixels at that spatiotemporal location whose signal value has reached the upper limit of the device's effective signal and no longer changes with the incident fluorescence intensity. The larger the ratio, the more intensity information is lost at that location, and the less definite the correspondence between it and the fluorescence response carried by the chromaticity component of the fluorescence pseudocolor video data. Therefore, the higher the fluorescence signal intensity, the larger the effective pixel ratio, and the smaller the saturated pixel ratio, the more reliable the correspondence between teacher-side features and student-side features at that spatiotemporal location, and correspondingly, a larger fluorescence reliability weight should be assigned. Conversely, when the effective pixel ratio is lower than the first threshold or the saturated pixel ratio is not lower than the second threshold, the signal at that location is insufficient to support the cross-modal correspondence, so the weight is set to zero so that it does not participate in distillation at all, thus avoiding the introduction of erroneous supervision signals.

[0116] It should be noted that the specific functional form of the fluorescence reliability weight is not limited to the multiplication of the three components mentioned above. Any functional form that monotonically increases with the effective pixel ratio and monotonically decreases with the saturated pixel ratio after background subtraction, and returns to zero when the effective pixel ratio is below a first threshold or the saturated pixel ratio is not below a second threshold, can be used to determine the fluorescence reliability weight. For example, the normalized fluorescence signal intensity, effective pixel ratio, and saturation penalty coefficient can be weighted by a geometric mean using a preset exponent; alternatively, a monotonically increasing nonlinear transformation can be applied to the normalized fluorescence signal intensity to adjust its influence on the weight, and then multiplied by the effective pixel ratio and saturation penalty coefficient. All of the above forms can give higher weights to reliable spatiotemporal locations of fluorescence signals and lower weights to unreliable spatiotemporal locations of fluorescence signals, or exclude them from distillation, achieving the same technical effect.

[0117] When the sum of the fluorescence reliability weights of all spatiotemporal data units within the same video segment is greater than 0, normalization is performed: , in, Indicates the first Normalized fluorescence reliability weights for each spatiotemporal data unit The spatiotemporal data unit number is used for summation. The total number of spatiotemporal data units within a video segment, in this embodiment And satisfy: , The purpose of normalization is to ensure that each video segment contributes the same amount to the training, regardless of the number of reliable spatiotemporal locations within that segment, thereby preventing segments with more reliable locations from dominating the training process.

[0118] When the sum of the fluorescence reliability weights of all spatiotemporal data units within a video segment is 0, the video segment is marked as a low-quality video segment. The video segment is not used to calculate the following local transmodal distillation loss and video segment-level transmodal distillation loss, but the video segment is still used to calculate the native fluorescence spatiotemporal data unit reconstruction loss.

[0119] The following example, using a specific spatiotemporal data unit, illustrates the complete process of determining the fluorescence reliability weights from the aforementioned quantities. This spatiotemporal data unit covers a 16-pixel × 16-pixel region at the same spatial grid location in two consecutive frames of images. Therefore... Assume that there are a total of 60 pixels in this spatiotemporal data unit, including pixels with occlusion due to equipment obstruction, pixels with missing data, and pixels with invalid equipment. There are 20 saturated pixels in total, that is... Then the number of effective pixels. effective pixel ratio saturated pixel ratio .

[0120] because Below the second threshold of 0.20, the saturation penalty coefficient Suppose that the average value of the remaining signal values ​​within this spatiotemporal data unit is the result of background subtraction and removal of the highest and lowest 10% of signal values ​​from all valid pixels. The 5th percentile of fluorescence signal intensity for each spatiotemporal data unit that meets the quality criteria within the same video segment. 95th percentile Then the normalized fluorescence signal intensity .

[0121] Initial weights .because Not lower than the first threshold of 0.50 and If the value is below the second threshold of 0.20, the spatiotemporal data unit meets the quality condition, and its fluorescence reliability weight is adjusted accordingly. If the sum of the fluorescence reliability weights of the 1568 spatiotemporal data units within this video segment is 392.5, then the normalized fluorescence reliability weights... .

[0122] As a comparison, suppose another spatiotemporal data unit within the same video segment is largely obscured by surgical instruments, with a total of 300 pixels obscured by the instruments, missing data pixels, and invalid equipment pixels, and 20 saturated pixels. Then the percentage of effective pixels would be: If the value is below the first threshold of 0.50, its fluorescence reliability weight is set to zero, and this spatiotemporal data unit does not participate in cross-modal distillation. Let's assume a third spatiotemporal data unit is located in a highly reflective region, with 120 saturated pixels and no other invalid pixels; then the effective pixel percentage is... Although it is not lower than the first threshold of 0.50, the saturated pixel ratio is Since the fluorescence intensity is not lower than the second threshold of 0.20, its saturation penalty coefficient is zero, and the fluorescence reliability weight is also zero. The signals of the two spatiotemporal data units mentioned above do not reflect the actual fluorescence response of the corresponding tissue regions; excluding them can avoid establishing incorrect correspondences between teacher-side and student-side features.

[0123] The fixed teacher encoder and student encoder will be in the... The features output from the corresponding spatiotemporal locations, after feature projection and L2 normalization, are denoted as follows: and superscript Indicates the teacher's side, superscript Represents the student side. Local cross-modal distillation loss. Calculate according to the following formula: , Among them, symbols This represents the dot product of two vectors. Since... and Both have been L2 normalized, and their inner product is the cosine similarity between the two features, with a value range of [value range missing]. Up to 1; This is the cosine distance between the two features in their shared feature space, used to measure the difference between teacher-side features and student-side features at the same spatiotemporal location. A smaller value indicates that the two features are closer. The weighted summation of the above differences at each spatiotemporal location yields the characteristic difference weighted by fluorescence reliability. Its physical meaning is: the overall degree of inconsistency between teacher and student characteristics, measured by the reliability of fluorescence signals at each spatiotemporal location, over the entire video segment.

[0124] In addition to the aforementioned local cross-modal distillation targets, video segment-level cross-modal distillation targets are also constructed. Teacher-side features and student-side features within the same video segment are weighted and aggregated according to fluorescence reliability weights to obtain teacher segment-level features. and student fragment-level features : , , and These represent the overall fluorescence correlation features on the teacher's and student's sides of the video segment, respectively. Their physical meaning is: the overall characterization of the segment obtained by weighting the features of all spatiotemporal locations of the segment with the reliability of the fluorescence signal at each spatiotemporal location as the weight. Video segment-level cross-modal distillation loss. Calculate according to the following formula: , In this case, the double vertical bar notation with the subscript 2 in the denominator represents the L2 norm of the vector. To prevent constants with zero denominators, take This formula represents the cosine distance between teacher-level and student-level segment features. The purpose of this term is twofold: the local cross-modal distillation target constrains features at each spatiotemporal location, making it more sensitive to local noise; the segment-level target, on the other hand, constrains the overall fluorescence correlation representation of the two modalities across the entire video to remain consistent. Combining these two approaches can maintain local correspondence while preventing the student encoder from fitting only individual spatiotemporal locations.

[0125] The target of the occlusion reconstruction of native fluorescence spatiotemporal data units is the normalized native fluorescence signal value of the occluded spatiotemporal data unit. The native fluorescence reconstruction decoder predicts this value and calculates the reconstruction loss of the native fluorescence spatiotemporal data unit based on the mean square error between the predicted value and the actual normalized signal value. .

[0126] The local cross-modal distillation target, the video clip-level cross-modal distillation target, and the occlusion reconstruction target of the native fluorescence spatiotemporal data unit are weighted and combined according to preset weights to obtain the total training loss. In this embodiment, the weight of the native fluorescence spatiotemporal data unit reconstruction loss is 1, the weight of the local cross-modal distillation loss is 0.5, and the weight of the video segment-level cross-modal distillation loss is 0.2. , During training, the parameters of the teacher encoder are fixed, while the parameters of the native fluorescence video encoder and the corresponding feature projection layer are updated. The transfer process only constrains the fluorescence-related information shared by the fluorescence pseudo-color video data and the native fluorescence video data, and does not require the native fluorescence video encoder to reconstruct all white light tissue structure information in the fluorescence pseudo-color video data.

[0127] In step S4 above, the three corresponding spatiotemporal data units are input into the white light video encoder, the updated fluorescent pseudo-color video encoder, and the updated native fluorescent video encoder, respectively. The obtained features are converted into the same dimension through their respective feature projection layers, and then combined into a trimodal feature group according to a unified time index and spatial grid. This group is then input into the trimodal fusion network to output the trimodal fusion features.

[0128] The three-modal feature set is constructed as follows: for each sequence number... Take the spatiotemporal feature units of white light, fluorescence pseudo-color, and native fluorescence, where the serial number is the same. These form a trimodal feature set, which together form Each trimodal feature group is represented by a trimodal feature network. The trimodal fusion network outputs a trimodal fusion feature for each trimodal feature group. Therefore, the trimodal fusion features are also arranged according to time index and spatial grid, corresponding one-to-one with spatiotemporal data units.

[0129] The three spatiotemporal feature units within the same trimodal feature group originate from the same tissue region corresponding to the same acquisition time period and spatial grid location. They respectively express the tissue structure information, the spatial relationship information of the superposition of tissue structure and fluorescence response, and the intensity distribution information of fluorescence signal in that region. These three describe different aspects of the same physical object, which is the basis for directly merging them into a single feature based on their location. Correspondingly, the output trimodal fusion feature characterizes the comprehensive state of tissue structure and fluorescence response within this spatiotemporal range, and the value of each dimension is jointly determined by the three inputs mentioned above.

[0130] The three-modal fusion features are intermediate feature parameters that characterize tissue structure and fluorescence response within the corresponding spatiotemporal range. They are used for feature analysis of lesion areas and their local sites, and can also be used for identification and localization of tissue structures within the surgical field, differentiation of instruments and tissues, and other endoscopic video analysis tasks such as video quality assessment. They do not directly yield disease diagnosis results.

[0131] In this embodiment, the trimodal fusion network adopts a cross-modal attention network; in practice, a gated fusion network can also be used, that is, a network generates three gate coefficients based on the three inputs, and then the three features are weighted and summed.

[0132] During the training of the trimodal fusion network, one modality is randomly selected from white light video data, fluorescent pseudo-color video data, or native fluorescent video data each time. 30% of the spatiotemporal feature units of this modality are randomly selected and replaced with trainable mask features. The spatiotemporal feature units of the remaining two modalities and the unmasked spatiotemporal feature units in the selected modality remain unchanged. For each modality containing 1568 spatiotemporal feature units, 470 of these are masked each time. The feature prediction layer predicts the masked features based on the remaining two modalities and the unmasked features of the selected modality, and updates the trimodal fusion network based on the difference between the predicted features and the actual encoder features. Gradient propagation is stopped for the encoder features used as the prediction target.

[0133] During the main training phase of the trimodal fusion network, the parameters of the three video encoders are fixed, and only the parameters of the three feature projection layers, the trimodal fusion network, the mask features, and the feature prediction layer are updated. This keeps the input features and prediction targets stable during training and avoids changes in the feature space caused by the video encoders updating synchronously through the unmasked feature path. The learning rate during the main training phase is set to... After completing the main training, the fixed state of the three video encoders is released, and joint fine-tuning is performed on the three video encoders, the feature projection layer, and the three-modal fusion network. The learning rate during the joint fine-tuning stage is set to... This is one-tenth of the learning rate during the main training phase.

[0134] As an optional implementation, the method provided in this embodiment further includes the following steps in addition to the above steps S1 to S4: Step S5: In the target frame of the white light video data, determine the lesion center site and multiple edge sites located in different directions of the lesion, and establish the lesion center-edge topology. Step S6: Perform joint tracking of each bit between adjacent frames of the white light video data; Step S7: Estimate the location of the tracking failure point and resume tracking when the preset recovery conditions are met. Extract the three corresponding local spatiotemporal data units according to the correspondence between the three video data to obtain the local three-modal fusion features of each point. Step S8: Extract the fluorescence curve parameters of each point to form fluorescence dynamic difference features corresponding to the center-edge topology of the lesion; Step S9: Encode the device imaging conditions and fluorescence signal formation conditions respectively, and input the local three-modal fusion features of each point, the fluorescence dynamic difference features, the device condition features, the fluorescence condition features, and the lesion center-edge topology into the lesion-level fusion network to output the lesion-level three-modal video representation.

[0135] Figure 3This is a flowchart of the establishment and joint tracking of the lesion center-edge topology provided in an embodiment of the present invention, such as... Figure 3 As shown, steps S5 to S7 above constitute the process of establishing and jointly tracking the lesion center-edge topology.

[0136] In step S5 above, the lesion center site and multiple edge sites located in different directions of the lesion are determined in the target frame of the white light video data. A local coordinate system is established with the lesion center site as the origin, and the normalized distance, orientation angle, site identifier, and order relationship between each edge site relative to the lesion center site are recorded to form a lesion center-edge topology. Each point corresponds to the local analysis region containing that site. The target frame is the video frame in the white light video data used to establish the lesion center-edge topology. In this embodiment, a video frame with a complete lesion display and clear boundaries is selected as the target frame.

[0137] In this embodiment, the lesion localization model first automatically determines the lesion region, lesion center site, and edge site in the target frame of the white light video data. When the confidence level of the automatic localization result is lower than a preset threshold or the lesion boundary is not fully displayed, the operator confirms and corrects the automatic localization result, or directly marks the corresponding site. The preset threshold is 0.50.

[0138] The lesion localization model includes an object detection model, an image segmentation model, or a combination of both. When using an object detection model, the initial lesion region is determined based on the lesion detection bounding box output by the model, and the geometric center of the bounding box is determined as the lesion center point. When using an image segmentation model, the lesion boundary is determined based on the lesion segmentation region output by the model, and the centroid of the segmentation region is determined as the lesion center point. When using a combination of both, the object detection model first determines the candidate lesion region, and then the image segmentation model obtains the lesion boundary and lesion center point within the candidate region. The object detection model can be a YOLO series model, and the image segmentation model can be U-Net or its improved model.

[0139] The principal and secondary axes of the lesion are calculated based on the contour of the segmented lesion region, and the intersections of the principal and secondary axes with the lesion boundary are determined as candidate edge sites. When the lesion boundary is complete and has reliably identifiable local tissue features in all four directions, four edge sites located in the principal and secondary axes of the lesion are selected. When the lesion boundary in one direction is obstructed, reflected light, or limited by the image range, three reliably located edge sites are retained. When two directions cannot be reliably located, two edge sites located in different directions that can be continuously tracked are retained.

[0140] The number of edge sites is set to two to four. Fewer than two edge sites only form a single center-to-edge direction, making it difficult to characterize the spatial differences and sequential relationships between edges in different directions. Four edge sites can correspond to the four main directions of the lesion's principal and secondary axes, forming a relatively complete lesion center-edge topology. When the number of edge sites exceeds four, newly added edge sites are usually located between adjacent principal directions, causing significant overlap between adjacent local analysis areas and increasing the repetitiveness of extracted features. Under conditions of lens movement, tissue deformation, local reflection, and instrument occlusion, overly dense edge sites are also prone to site identity confusion and increase the computational load of candidate matching and topological constraints. Therefore, this embodiment adaptively determines the actual number of edge sites from two to four edge sites based on the integrity of the lesion boundary and the number of directions that can be stably tracked.

[0141] A local coordinate system is established with the lesion center as the origin and the lesion principal axis, the preset image direction, or the initial center-to-edge direction as the reference axis. The local scale of the lesion is determined based on the lesion frame size, the lesion segmentation region size, or the average distance from the lesion center to each edge point.

[0142] For each edge location, the following information is recorded: normalized distance, which is the distance from the edge location to the lesion center location divided by the local scale of the lesion, used to keep this value constant when the viewing distance changes and the image zooms out; orientation angle, which is the angle between the line connecting the edge location to the lesion center location and the reference axis; location identifier, which is the number assigned to the edge location based on its initial orientation relative to the lesion center location, which remains unchanged in continuous video data to avoid confusion of edge regions with different orientations due to camera rotation; and the order relationship between the edge locations, i.e., the order in which the edge locations are arranged according to their orientation angles. The above information together constitutes the lesion center-edge topology.

[0143] Square local analysis regions were established, centered on the central and peripheral sites of the lesion. The lesion scale was set as follows: When using target detection boxes, Take the smaller value between the detection frame width and height; when segmenting regions using lesions, The equivalent diameter of the segmented lesion region is obtained. The side length of the local analysis region is also determined. Determined according to the following formula: , in, The truncation function defined above, i.e., when When the pixel size is less than 32 pixels, use 32 pixels; when the pixel size is greater than 96 pixels, use 96 pixels; otherwise, use [the specified value]. The unit is pixels. A coefficient of 0.25 makes the local analysis area vary proportionally with the lesion size; a lower limit of 32 pixels is set to ensure that the local analysis area of ​​small lesions still contains sufficient texture information; an upper limit of 96 pixels is set to avoid excessive overlap between adjacent local analysis areas of large lesions. The side length... This refers to the side length of the core local analysis region; when it is necessary to retain information about adjacent tissues, the truncated side length is expanded before inputting to the video encoder. Then, they were all adjusted to 224×224 pixels.

[0144] In step S6 above, joint tracking is performed on each point, rather than tracking each point independently. If the lesion center and each edge point are tracked independently, the position of each point in the current frame is determined only by its own local appearance, and the relative positions and spatial order between points are not constrained. In endoscopic examination, lens movement, changes in observation distance, tissue deformation, reflection, blood contamination, and instrument obstruction can all change the local appearance. If the appearance matching of a point fails, the point will drift or be lost. Furthermore, the local appearance of the edge regions of the lesion in different directions is often similar, and there is a lack of spatial constraints between points, which can easily lead to confusion of point identities, causing the same point identifier to point to different directions of the lesion in different frames. Joint tracking introduces a topological consistency score in addition to appearance matching and motion consistency scores, so that the position of each point is simultaneously constrained by the spatial relationships of the other points.

[0145] Joint tracking of each point is performed between adjacent frames of white light video data. Candidate positions are generated for each point, and appearance matching score, motion consistency score and topology consistency score are determined separately. The position of each point in the current frame is determined by the weighted sum of the three scores, while maintaining the point identifier and the lesion center-edge topology.

[0146] Between adjacent white-light video frames, one or more candidate locations are generated for the lesion center and each edge location based on optical flow, local feature correlation, or a video target tracking network. For each candidate location, the following three types of scores are determined, all normalized to 0 to 1: Appearance matching score This characterizes the similarity between the candidate location and the corresponding region in the previous frame, calculated based on the feature similarity between the local region where the candidate location is located and the corresponding region in the previous frame.

[0147] Motion consistency score The similarity represents the degree of consistency between the candidate position and the preceding motion trajectory. It is calculated based on the degree of conformity between the displacement direction and displacement amplitude of the candidate position relative to the previous frame position and the preceding motion trajectory. The feature similarity can be calculated using known metrics such as normalized cross-correlation and cosine similarity, and the degree of conformity can be measured using known methods such as the ratio of the direction angle to the amplitude between displacement vectors.

[0148] Topology Consistency Score This score characterizes the degree to which the spatial relationship of the candidate location relative to other sites in the local coordinate system matches the lesion center-edge topology. During calculation, the translation, rotation, and scale changes of the lesion's local region between adjacent frames are first estimated using sites with high tracking confidence. Then, the normalized distance, orientation angle, and changes in the order of edge sites relative to the recorded topology of the candidate location in the local coordinate system are calculated. The smaller the changes, the higher the score.

[0149] Weighted sum of candidate positions Calculate according to the following formula: , in, , and Let represent the weights of appearance matching score, motion consistency score, and topology consistency score, respectively, and satisfy the following conditions: , The three weights mentioned above are based on the tracking confidence of the corresponding locus in the previous frame. Confirmed, among which This indicates that the site is in the... The tracking confidence of the frame. The weights are determined using the tracking confidence of the previous frame instead of the weighted sum of the current frame to avoid circular dependencies that could arise from using the weighted sum of the current frame to determine the weights of the current frame. In the target frame, the tracking confidence at each point is set to 1.

[0150] In this embodiment, when hour, , and Set them to 0.40, 0.25, and 0.35 respectively. When When, the three values ​​were set to 0.25, 0.20, and 0.55 respectively, to reduce the impact of local appearance changes on the tracking results and enhance the constraint effect of the lesion center-edge topology. When When the three values ​​were set to 0.30, 0.10, and 0.60 respectively, during periods when the site was occluded or tracking was interrupted, its preceding motion trajectory was determined by an estimated location rather than actual observation, thus reducing the reference value of the motion consistency score; therefore, the score was further reduced. Meanwhile, the local transformations estimated from other reliable sites remain valid, thus further improving... .

[0151] As can be seen from the above, the lower the tracking confidence, the greater the weight of the topology consistency score. Its function is that when the appearance information of a certain site is no longer reliable, its position is mainly determined by the spatial relationship between that site and other sites, so that the site identification and relative position can still be maintained even in the case of local appearance failure.

[0152] For each site, select the candidate site with the highest weighted sum, and denot its weighted sum as . .

[0153] when If the position is not lower than the preset threshold, the candidate position is directly used as the tracking position of the point in the current frame, and the point is then used in the [number]th frame. Frame tracking confidence Set as In this embodiment, the preset threshold is directly set to 0.60.

[0154] when If the position is below the preset threshold but not below the preset acceptance threshold, it is considered an uncertain tracking position. This candidate position is still used, but after correction based on the motion trajectory of the previous three frames and the local transformation estimated from other reliable positions, it is taken as the position of that position in the current frame. Set as In this embodiment, the preset acceptance threshold is set to 0.50.

[0155] The preset acceptance threshold serves as the dividing line between using candidate positions: when the highest weighted sum is not lower than this threshold, the candidate position is used, either directly as the tracking position of the point in the current frame, or after correction as the position of the point in the current frame; when the highest weighted sum is lower than this threshold, the candidate position is not used at all, and the position of the point is estimated in the manner described below. The preset direct adoption threshold is higher than the preset acceptance threshold, and is only used to distinguish between the two adoption methods, without affecting the determination of whether to use a candidate position.

[0156] Therefore, the tracking confidence of each point is determined based on the weighted sum of the candidate positions used by that point in the corresponding frame.

[0157] In step S7 above, the location of the tracking failure site is estimated and tracking is resumed when the conditions are met, and then three local spatiotemporal data units are extracted.

[0158] When a point has the highest weighted sum in the current frame When the position is below the preset acceptance threshold, its candidate position is not used. Instead, its current position is estimated based on the local transformation of the other positions with tracking confidence higher than the preset reliability threshold and the previous motion trajectory of the position. The tracking confidence of the position in the current frame is set to the preset value.

[0159] In this embodiment, the preset reliability threshold is 0.70, and the preset value is 0.40. Specifically, when... At this point, the site is considered a low-confidence site and its candidate position is not adopted. The translation, rotation, and scale transformation of the local lesion region are estimated based on the remaining reliable sites with tracking confidence higher than 0.70, and its current position is predicted by combining the motion trajectory of the site in the first 3 frames. The tracking confidence of the site in the current frame is set to 0.40, so that it continues to be calculated in subsequent video frames according to the weight when the tracking confidence is lower than 0.50.

[0160] Tracking is resumed when all scores at the location meet preset recovery conditions. In this embodiment, the preset recovery conditions are: appearance matching score not lower than 0.70, topology consistency score not lower than 0.60, and weighted sum not lower than 0.60. When these conditions are met, normal tracking of the location is resumed, and its tracking confidence is reset according to the weighted sum at the time of recovery.

[0161] If no candidate location meeting the recovery criteria is found for 8 consecutive frames, the lesion localization model is invoked to redetermine the site. If the localization model output still does not meet the requirements, the operator performs correction. After redetermining or correcting, the tracking confidence of the site in the corresponding video frame is set to 1, and tracking continues in the above manner. At an effective processing frame rate of 15 frames / second, 8 consecutive frames correspond to approximately 0.53 seconds.

[0162] The following example illustrates the complete process of tracking an edge point across several consecutive frames, using the weighted sum, dynamic weights, position estimation, and recovery tracking described above.

[0163] Let the edge site be at the th Frame tracking confidence Not less than 0.70, therefore the first The weights of the three categories of ratings in the frame are taken as follows , , This site is at the 1st The frame generates three candidate positions with appearance matching scores of 0.86, 0.74, 0.90; motion consistency scores of 0.80, 0.62, 0.55; and topology consistency scores of 0.55, 0.80, 0.70, respectively. The weighted sums calculated using the above formula are 0.844, 0.668, and 0.665, respectively. The highest weighted sum is... The value is not lower than the preset threshold of 0.60, so the first candidate position is directly used as the position of the locus in the th position. The tracking position of the frame, and Set it to 0.844.

[0164] Suppose that the site is from the th The frame is partially obscured by surgical instruments. Because... The score remains no lower than 0.70, and the weights of the three categories of scores remain unchanged. The scores of the candidate positions corresponding to the highest weighted sum in this frame are 0.32, 0.58, and 0.66, with a weighted sum of 0.504. This value is lower than the preset direct adoption threshold of 0.60 but not lower than the preset acceptance threshold of 0.50. Therefore, this position is an uncertain tracking position, and this candidate position is still used. However, after correction based on the local transformations estimated from the motion trajectories of the previous three frames and other reliable positions, it is used as the position in the [missing information - likely a specific frame or location]. The position of the frame, and Set it to 0.504.

[0165] Suppose that the site is from the th The frame is completely obscured. Because... If the score falls between 0.50 and 0.70, the weights of the three categories of scores will be adjusted. , , The candidate positions corresponding to the highest weighted sum in this frame have three categories of scores: 0.20, 0.45, and 0.62, with a weighted sum of 0.481. This value is lower than the preset acceptance threshold of 0.50, so its candidate position is not adopted. Instead, its current position is estimated based on the local transformation of other reliable positions with tracking confidence higher than 0.70 and the motion trajectory of that position in the previous three frames. Set to the default value of 0.40.

[0166] Let this site be at the The frame becomes visible again. Due to the first... The frame tracking confidence score is 0.40, which is lower than 0.50. The weights of the three categories of scores are taken as follows: , , The candidate positions corresponding to the highest weighted sum in this frame have three categories of scores: 0.74, 0.35, and 0.68, with a weighted sum of 0.665. Their appearance matching score (0.74) is no less than 0.70, their topology consistency score (0.68) is no less than 0.60, and their weighted sum (0.665) is no less than 0.60. All three criteria meet the preset recovery conditions. Therefore, normal tracking of this position is restored, and the weighted sum at the time of recovery is used to... Reset to 0.665. (From the...) Frame to the The estimated location was used for 3 frames, which did not reach 8 consecutive frames, so there was no need to call the lesion localization model to redetermine the site.

[0167] In the above-mentioned section Frame to the In the frame, the site's own appearance information is no longer available; its location is entirely determined by the local transformations estimated from other sites and its own preceding motion trajectory. Meanwhile, the site identifier and the lesion center-edge topology remain unchanged throughout this period. Therefore, this site in the [frame number missing]... The local features extracted after frame recovery tracking still originate from the same direction as the lesion.

[0168] Based on the temporal index and spatial grid correspondence established above among the three video data, the local analysis region of each point is mapped to the fluorescence pseudo-color video data and the original fluorescence video data. The three corresponding local spatiotemporal data units are extracted and input into the corresponding video encoder and three-modal fusion network respectively to obtain the local three-modal fusion features of each point.

[0169] Specifically, based on the joint tracking results in the white light video data, the local analysis region of each point in each video frame is obtained; based on the temporal synchronization and spatial transformation relationship between the three video data, the region coordinates in the white light video data are mapped to the corresponding fluorescence pseudo-color video frames and native fluorescence video frames, and local video segments are extracted according to the same time range and corresponding spatial range; the corresponding local video segments of the three channels are uniformly resized and temporally sampled, divided into local spatiotemporal data units according to the division rules above, and then input into the corresponding video encoders, and then obtained through a three-modal fusion network to obtain the local three-modal fusion features of the lesion center site and the local three-modal fusion features of each edge site.

[0170] Since the three local spatiotemporal data units are obtained from the same set of regional coordinates through a predetermined mapping relationship, each point corresponds to the same organizational region in the three video data. Therefore, the obtained three features can be combined into a three-modal feature group in the manner described above.

[0171] In step S8 above, the fluorescence curve parameters of each point are extracted, and the fluorescence dynamic difference characteristics are formed.

[0172] Based on the changes in the effective pixel signal of the native fluorescence video data within the local analysis region of each point over time, fluorescence curve parameters are extracted for each point. Before calculation, saturated pixels, missing data pixels, invalid device pixels, and instrument-occluded pixels in the native fluorescence video data are identified and excluded.

[0173] For the The first frame A local analysis region, This indicates the sequence number of the local analysis region. The local analysis regions at each point are numbered sequentially according to the lesion center first, followed by each edge site, and their effective pixel percentage is also indicated. Calculate according to the following formula: , in, Indicates the first The first frame The set of all pixels within a local analysis region This represents the set of valid pixels after excluding pixels obscured by instruments, pixels with missing data, invalid pixels from equipment, and saturated pixels. When At that time, the data of the local analysis area in the video frame is marked as low-quality data, and the fluorescence signal of the area in the frame is not used to calculate the regional fluorescence intensity and fluorescence curve parameters.

[0174] Saturated pixels are determined based on the upper limit of the device's effective signal strength in the native fluorescence video data, not on the maximum value of the data storage container. Let the upper limit of the device's effective signal strength be... saturation threshold Calculate according to the following formula: , In this formula, the parentheses on the right indicate rounding up, meaning taking the smallest integer not less than the value within the parentheses; 0.98 is the preset ratio mentioned above. When pixels raw fluorescence signal value A pixel is defined as a saturated pixel if it satisfies the following formula: , For native fluorescent video data with an effective signal bit width of 12 bits, The saturation threshold is 4014; for native fluorescent video data with an effective signal bit width of 10 bits, The saturation threshold is 1003. All the above data is stored in a 16-bit unsigned integer container. The upper limit of the effective signal value is used instead of the maximum value of the container because the maximum value of the 16-bit container (65535) is much higher than the actual signal value that the sensor can output. If the maximum value of the container were used, pixels that are already saturated by the sensor would not be recognized. When the endoscopic device provides a calibrated saturation threshold, the device calibration value is used preferentially.

[0175] The background signal value of each video frame is determined as described above, i.e., the median of the lowest percentage of pixels within the video frame, the median of pixel signal values ​​within a pre-marked background reference area, or the pre-measured and stored device background signal value is used, and background subtraction is performed on the fluorescence signal within the local analysis area. Signal normalization is performed based on available imaging parameters such as exposure time and fluorescence channel gain, or relative normalization is performed using the fluorescence signal of the reference tissue area within the same frame. For the effective pixel signal values ​​after background subtraction, the regional fluorescence intensity of each video frame is determined using the median, the average value after removing the highest and lowest signal values ​​from the preset percentage, or the average value of all effective pixel signal values. The regional fluorescence intensities are then arranged relative to the time of fluorescent reagent administration to form fluorescence curves for the lesion center and each edge site.

[0176] At least one of the following parameters was extracted from the actual observation time range: peak value, minimum value, mean value, amplitude of change, maximum rate of increase, maximum rate of decrease, area under the curve, and degree of temporal fluctuation as fluorescence curve parameters. These parameters reflect the fluorescent reagent perfusion level, perfusion rate, clearance rate, and total perfusion volume of the tissue region corresponding to the site, respectively. For cases where video data does not cover the complete fluorescence inflow or attenuation process, only parameters that can be reliably calculated within the actual observation range were extracted.

[0177] The fluorescence curve parameters mentioned above are not used as the sole result of lesion characterization. Instead, the differences in fluorescence curve parameters between each edge site and the lesion center site are arranged according to the site identifier, orientation angle, and sequence relationship of each edge site, forming a dynamic fluorescence difference feature corresponding to the lesion center-edge topology.

[0178] For any fluorescence curve parameter, the curve parameter at the center of the lesion is denoted as... , will the The corresponding curve parameters for each edge site are denoted as: , where subscript Indicates the central location of the lesion. The index represents the edge site, and values ​​are taken only within the range of each edge site. The fluorescence dynamic difference between the center and the edge is calculated using the following formula: , in, Indicates the first The difference in this fluorescence curve parameter between each marginal site and the central site of the lesion indicates whether the fluorescence level at the marginal site is higher or lower than that at the central site. The absolute difference can also be calculated. , in, This represents the absolute value of the above difference, used to characterize the magnitude of the difference without distinguishing direction. For different edge sites, the differences between their corresponding curve parameters can also be calculated.

[0179] The differences between the center and the edge, as well as the differences between each edge site, are arranged sequentially according to the fixed site identifier, orientation angle, and order of each edge site. This forms a dynamic fluorescence difference feature corresponding to the center-edge topology of the lesion. Arranging them by site identifier rather than by numerical value ensures that each position in this feature corresponds to a specific orientation of the lesion, thus making it comparable across different frames and examinations.

[0180] In addition to the aforementioned fluorescence dynamic difference features, the local trimodal fusion features of the lesion's central site and each edge site can be normalized. For each edge site, at least one of the following can be calculated: feature difference, absolute difference, feature product, cosine similarity, or feature distance between it and the lesion's central site, forming the image difference features between the center and the edge; and the local trimodal fusion feature differences between different edge sites can be calculated, forming the difference features between edges. These difference features are also arranged according to the fixed site identifier, orientation angle, and order relationship of each edge site, or the lesion's central site and each edge site can be used as graph nodes, and the relationship between the center and the edge and the relationship between edges can be used as graph edges, so that the difference features are associated with the lesion's center-edge topology.

[0181] In step S9 above, the imaging conditions and fluorescence signal formation conditions are first encoded separately. This step is necessary because the same tissue is recorded with different pixel values ​​under different equipment, different microscope types, and different exposure, gain, white balance, gamma correction, and noise reduction settings; the actual fluorescence intensity emitted by the same lesion also varies under different types of fluorescent reagents, different dosages, and at different times after administration. If these conditions are not distinguished, the signal differences caused by the imaging process and drug administration will be mixed with the signal differences caused by the tissue itself, making the features obtained from the same lesion in different examinations incomparable.

[0182] The device imaging conditions and fluorescence signal formation conditions during the acquisition of the video data are input into two independent conditional coding networks to obtain device conditional features and fluorescence conditional features. The device conditional features are combined with the video imaging features, and the fluorescence conditional features are combined with the fluorescence dynamic features; the combination is either stitching or generating parameters from the conditional features to adjust the corresponding features.

[0183] The video imaging features are features obtained from video data through a video encoder and a three-modal fusion network, including the three-modal fusion features mentioned above, the local three-modal fusion features at each bit, and the global three-modal fusion features mentioned below; the fluorescence dynamic features are features obtained based on the changes of the effective pixel signals of the original fluorescence video data over time, including the fluorescence curve parameters and fluorescence dynamic difference features mentioned above.

[0184] In this embodiment, the combination is a splicing process. Specifically, at the input of the lesion-level fusion network, device conditional features are spliced ​​with each video imaging feature, and fluorescence conditional features are spliced ​​with each fluorescence dynamic feature, before participating in the joint encoding described below. Alternatively, the combination involves generating parameters from the conditional features to adjust the corresponding features. Specifically, the device conditional features generate scaling factors and offsets for each feature channel of the video imaging features via a fully connected layer, and these scaling factors and offsets are applied to each channel of the video imaging features. Similarly, the fluorescence conditional features generate and apply scaling factors and offsets to each component of the fluorescence dynamic features in the same manner. Both combination methods ensure that the corresponding features are adjusted according to their acquisition conditions, making the features obtained under different examination conditions comparable.

[0185] The device imaging conditions include at least one of the following: device type, lens type, sensor mode, exposure, gain, white balance, gamma correction, noise reduction, and fluorescence channel gain. The fluorescence signal formation conditions include at least one of the following: fluorescent reagent type, drug dosage, drug administration method, and the time of a video frame or video segment relative to the drug administration time.

[0186] The two types of conditions have different functions: the device imaging conditions determine the numerical values ​​recorded for the same tissue in the imaging chain, and are therefore combined with video imaging characteristics; the fluorescence signal formation conditions determine how much fluorescence the same tissue actually emits at a given moment, and are therefore combined with fluorescence dynamic characteristics. They are encoded and function separately, without forcing the conversion of raw fluorescence intensities from different devices or under different drug administration conditions to a uniform physical scale.

[0187] In this embodiment, device type, microscope type, sensor mode, fluorescent reagent type, and administration method are encoded as discrete variables. Device type and microscope type are each encoded using a 32-dimensional trainable embedding vector, sensor mode using a 16-dimensional trainable embedding vector, and fluorescent reagent type and administration method using a 16-dimensional trainable embedding vector. For discrete variables with values ​​not present in the training data or with missing values, a pre-defined unknown category embedding vector is used to represent that variable.

[0188] Exposure time, device gain, white balance parameters, gamma correction parameters, noise reduction intensity, and fluorescence channel gain are encoded as continuous device variables; drug dosage and time relative to the time of fluorescent reagent administration are encoded as continuous fluorescence variables. Instead of binning lookup tables or sinusoidal positional encoding, these continuous variables are directly mapped through fully connected layers after numerical transformation and normalization to maintain the continuity and monotonicity of their values.

[0189] For non-negative continuous variables with a wide range of values, such as exposure time, device gain, fluorescence channel gain, and drug dosage, a logarithmic transformation is first performed: , in, This represents the original values ​​of the continuous variable. This represents the value after logarithmic transformation. This represents the natural logarithm. Adding 1 is to make it... When the value is 0, the transformation result is 0, and the logarithm of zero is avoided. Logarithmic transformation is used because the influence of the above parameters on the imaging result corresponds approximately to their multiple relationship rather than their difference relationship.

[0190] Then, normalization is performed based on the median and interquartile range of the corresponding variables in the training data: , in, This represents the normalized value; and These represent the median and interquartile range of the variable after logarithmic transformation in the training data, respectively. To prevent constants with zero denominators, take ; The truncation function defined above restricts the normalization result to... The values ​​are between 5 and 5. The median and interquartile range are used instead of the mean and standard deviation to reduce the impact of individual extreme values ​​on the normalization results.

[0191] For gamma correction parameters, noise reduction intensity, and white balance parameters, no logarithmic transformation is performed; the normalization method based on median and interquartile range is directly adopted. When the white balance parameter includes multiple color channel gains, each channel gain is encoded as an independent continuous variable.

[0192] Time relative to the time of administration To simultaneously represent the time positions before and after drug administration, a signed logarithmic transformation is used: , in, Indicates the value after a signed logarithmic transformation; This represents the sign function, which takes the value 1 when the independent variable is positive and 0 when it is negative. When the sum is zero, take 0; express The absolute value of the time before and after drug administration is determined. This transformation places the time before and after drug administration on opposite sides of the number axis, with the absolute value increasing the further away from the drug administration time. After the transformation, the result is normalized based on the median and interquartile range of this variable from the training data, and the result is restricted to a certain range. Between 5 and 5. When all processed video data is located after drug administration, the post-drug administration time can also be limited to a non-negative number, and a method can be used. Perform the transformation.

[0193] Each continuous variable is assigned a corresponding binary missing value flag: 0 for a variable that is available, and 1 for a missing variable, which is then filled with the median of the variable from the training data. The normalized continuous variables and their corresponding missing values ​​are then input into a conditional encoding network, enabling the network to distinguish between cases where the actual value is exactly the median and cases where missing values ​​are filled.

[0194] Equipment continuous variables and their missing identifiers are sequentially concatenated and input into the equipment continuous conditional coding network. This network comprises a 64-dimensional fully connected layer, a Gaussian error linear unit activation layer, and a 128-dimensional fully connected layer, outputting 128-dimensional equipment continuous conditional features. Fluorescence continuous variables and their missing identifiers are also sequentially concatenated and input into a fluorescence continuous conditional coding network with parameters independent of the equipment continuous conditional coding network but identical in structure, outputting 128-dimensional fluorescence continuous conditional features.

[0195] Device type embedding, mirror type embedding, sensor mode embedding, and continuous device condition features are concatenated and mapped to 128-dimensional device condition features through a fully connected layer. Fluorescent reagent type embedding, administration method embedding, and continuous fluorescence condition features are concatenated and mapped to 128-dimensional fluorescence condition features through another parameter-independent fully connected layer.

[0196] This embodiment does not require that the device imaging conditions and fluorescence signal formation conditions be completely obtained in every video frame or video segment. When all parameters in the device imaging conditions or fluorescence signal formation conditions are missing, an unknown condition vector pre-set in the corresponding condition encoding network is used, and a condition availability identifier is generated simultaneously. The condition availability identifier and the corresponding condition feature are input into the lesion-level fusion network to indicate whether the corresponding condition feature is encoded from the actually obtained condition parameters.

[0197] The local trimodal fusion features, fluorescence dynamic difference features, device condition features, fluorescence condition features, and lesion center-edge topology of each point are input into the lesion-level fusion network, and the lesion-level trimodal video representation is output.

[0198] In this embodiment, a global video segment containing the lesion and surrounding tissue is also extracted from the three corresponding video data. The global three-modal fusion features are obtained by three video encoders and a three-modal fusion network. These features, along with the image difference features between the center and the edge, the difference features between edges, and the fluorescence curve parameters, are input into the lesion-level fusion network.

[0199] The lesion-level fusion network employs attention coding, graph structure coding, or fully connected mapping after feature concatenation to jointly encode the aforementioned inputs. The relationship between each input and output is as follows: global trimodal fusion features provide overall environmental information of the surgical field in which the lesion is located; local trimodal fusion features at each point provide local tissue structure and fluorescence response information of the lesion center and edges in different directions; fluorescence dynamic difference features and imaging difference features provide difference information between different directions within the lesion; device condition features and fluorescence condition features provide information on the imaging and fluorescence formation conditions under which the above information was obtained, making features obtained under different examination conditions comparable; the lesion center-edge topology provides the spatial positional relationship between the above local information, allowing each local feature to be interpreted according to its direction rather than the order of input.

[0200] The lesion-level trimodal video representation serves as intermediate feature parameters for characterizing lesions. It is used to output local features, fluorescence curve parameters, and spatial difference parameters of the lesion's central and peripheral sites, or as a common input for subsequent lesion classification, risk prediction, efficacy assessment, and case comparison tasks. This method does not directly yield a disease diagnosis; the final judgment is made independently by the physician in conjunction with other examinations.

[0201] Figure 4 This is a block diagram of the endoscopic multimodal video data processing system provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the system includes a data mapping module, a first migration module, a second migration module, and a feature fusion module.

[0202] The data correspondence module is used to acquire white light video data, fluorescence pseudo-color video data, and native fluorescence video data generated during the same endoscopic examination, and divides the three video data into corresponding spatiotemporal data units according to a unified time index and spatial grid. This module performs the processing described in step S1 above.

[0203] The first transfer module is used to copy the spatiotemporal feature coding layer parameters of the white light video encoder to the initial parameters of the fluorescent pseudo-color video encoder, configure independent input normalization parameters and modality adaptation layers for the fluorescent pseudo-color video encoder, and use the white light video encoder as a fixed teacher encoder to construct cross-modal feature constraints based on the differences in the output features of the two encoders after feature projection, thereby updating the fluorescent pseudo-color video encoder. This module performs the self-supervised pre-training of the white light video encoder and the processing described in step S2.

[0204] The second transfer module uses the updated fluorescence pseudo-color video encoder as a fixed teacher encoder. It weights the feature differences between the fluorescence pseudo-color spatiotemporal data units and the native fluorescence spatiotemporal data units after feature projection, according to fluorescence reliability weights determined based on the fluorescence signal intensity after background subtraction, the effective pixel ratio, and the saturated pixel ratio. This yields a local cross-modal distillation target, which is then used to update the native fluorescence video encoder. This module performs the self-supervised pre-training of the native fluorescence video encoder and the processing described in step S3.

[0205] The feature fusion module is used to extract features from three corresponding spatiotemporal data units through their respective video encoders, combine them into a three-modal feature group, input the group into the three-modal fusion network, and output the three-modal fused features. This module performs the processing described in step S4 above.

[0206] The connection relationship between the above four modules is as follows: the output of the data correspondence module is connected to the input of the first migration module, the second migration module, and the feature fusion module, respectively; the output of the first migration module is connected to the input of the second migration module; and the outputs of both the first migration module and the second migration module are connected to the input of the feature fusion module.

[0207] The system may further include a lesion center-edge topology establishment module, a joint tracking module, a three-modal local video extraction module, a fluorescence dynamic information extraction module, a conditional coding module, a differential feature construction module, and a lesion-level characterization generation module, which respectively perform the processing described in steps S5 to S9 above.

[0208] The above modules can be implemented by software modules, application-specific integrated circuits or programmable logic devices, or by a combination of software and hardware.

[0209] The present invention also provides an electronic device, including a processor, a memory, and program instructions stored in the memory and executable by the processor. When the processor executes the program instructions, it implements the above-described method for processing endoscopic multimodal video data.

[0210] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for processing endoscopic multimodal video data. The computer-readable storage medium includes a read-only memory, random access memory, magnetic disk, optical disk, or flash memory, or other media capable of storing program code.

[0211] The video encoder structure described in various parts of this invention is not limited to the video Transformer structure mentioned above. Any spatiotemporal coding structure that can receive video segments, output spatiotemporal feature units corresponding one-to-one with spatiotemporal data units according to time index and spatial grid, and can be self-supervised pre-trained through occlusion reconstruction can be used as the video encoder; in this case, the above-mentioned parameter copying, modality adaptation layer setting, and phased defixation processing are performed on the spatiotemporal feature coding layer in the structure accordingly.

[0212] The training order of the three video encoders described in this invention is as follows: first, the white light video encoder is trained; then, it is migrated to the fluorescence pseudo-color video encoder; and finally, the fluorescence pseudo-color video encoder is migrated to the native fluorescence video encoder. This order is determined by the data distribution relationship between the three video data streams: the fluorescence pseudo-color video data retains the organizational structure information of the white light video data in the luminance component and carries the response distribution information of the native fluorescence signal in the chrominance component. It lies between the white light video data and the native fluorescence video data, thus serving as an intermediary between the two, eliminating the need for direct migration between white light video data and native fluorescence video data with significantly different data distributions.

[0213] The specific values ​​given in this embodiment, including the number of frames in the video segment, image resolution, size of the spatiotemporal data unit, number and dimension of the spatiotemporal feature coding layer, occlusion ratio, training rounds, learning rate, weights of various losses, thresholds, and preset ratios, are all a set of values ​​given for ease of explanation. Those skilled in the art can make adaptive adjustments according to the actual video data scale, equipment conditions, and computing resources. After adjustment, the technical solution of the present invention can still be implemented according to the above steps.

[0214] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0215] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0216] In the several embodiments provided by this invention, it should be understood that the disclosed technical solutions can be implemented in other ways. The system embodiments described above are merely illustrative; for example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0217] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0218] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit; when implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0219] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing multimodal video data from an endoscope, characterized in that, include: Acquire white light video data, fluorescence pseudo-color video data, and native fluorescence video data generated during the same endoscopic examination. Divide the three video data into spatiotemporal data units according to a unified time index and spatial grid, so that the spatiotemporal data units with the same time index and spatial grid position in the three video data correspond to each other. Among them, the fluorescence pseudo-color video data is video data formed by superimposing the native fluorescence signal with the white light tissue image after color mapping. Its luminance component carries tissue structure information, and its chromaticity component carries fluorescence response information. Using a white light video encoder as a fixed teacher encoder and a fluorescent pseudo-color video encoder as a student encoder, corresponding white light spatiotemporal data units and fluorescent pseudo-color spatiotemporal data units are input into both. Cross-modal feature constraints are constructed based on the differences in the obtained features after feature projection. The fluorescent pseudo-color video encoder is updated together with the cross-modal feature constraints and the occlusion reconstruction target of the fluorescent pseudo-color spatiotemporal data units. The white light video encoder is an encoder obtained through self-supervised pre-training of white light video data occlusion reconstruction. The initial parameters of the fluorescent pseudo-color video encoder are copied from the spatiotemporal feature coding layer parameters of the white light video encoder and are configured with independent input normalization parameters and modality adaptation layers. Using the updated fluorescence pseudo-color video encoder as the fixed teacher encoder and the native fluorescence video encoder as the student encoder, fluorescence reliability weights are determined for each native fluorescence spatiotemporal data unit. The feature differences between corresponding fluorescence pseudo-color spatiotemporal data units and native fluorescence spatiotemporal data units after feature projection are weighted according to these fluorescence reliability weights to obtain a local cross-modal distillation target, which is then used to update the native fluorescence video encoder. The fluorescence reliability weights are determined based on the fluorescence signal intensity after background subtraction, the effective pixel ratio, and the saturated pixel ratio within the spatiotemporal data unit. Higher fluorescence signal intensity, a larger effective pixel ratio, and a smaller saturated pixel ratio result in a larger fluorescence reliability weight. The fluorescence reliability weight is zero when the effective pixel ratio is below a first threshold or the saturated pixel ratio is not below a second threshold. The native fluorescence video encoder is an encoder obtained through self-supervised pre-training via native fluorescence video data occlusion reconstruction. The three corresponding spatiotemporal data units are respectively input into the white light video encoder, the updated fluorescent pseudo-color video encoder, and the updated native fluorescent video encoder. The obtained features are combined into a three-modal feature group according to the unified time index and spatial grid and input into the three-modal fusion network to output the three-modal fusion features.

2. The method for processing endoscopic multimodal video data according to claim 1, characterized in that, The fluorescence reliability weight is determined as follows: the median of the pixels with the lowest signal value within the video frame containing the spatiotemporal data unit is taken as the background signal value of that video frame; the effective pixels are the pixels within the spatiotemporal data unit after excluding pixels obscured by instruments, pixels with missing data, invalid devices, and saturated pixels. After background subtraction of the effective pixels within the spatiotemporal data unit, the highest and lowest signal values ​​of the preset proportion are removed, and the average of the remaining signal values ​​is taken as the fluorescence signal intensity. This intensity is then normalized according to the fluorescence signal intensity distribution of each spatiotemporal data unit that meets the quality conditions within the same video segment. The quality condition is that the effective pixel ratio is not lower than the first threshold and the saturated pixel ratio is lower than the second threshold. Pixels whose signal values ​​reach a preset proportion of the device's effective signal upper limit are defined as saturated pixels, and a saturation penalty coefficient that decreases as the saturated pixel ratio increases is determined. The product of the normalized fluorescence signal intensity, the effective pixel ratio, and the saturation penalty coefficient is used as the initial weight. For spatiotemporal data units that meet the quality condition, the initial weight is limited to between a preset lower limit and 1, and the fluorescence reliability weight of each spatiotemporal data unit within the same video segment is normalized.

3. The method for processing endoscopic multimodal video data according to claim 1 or 2, characterized in that, When updating the native fluorescence video encoder, a video segment-level cross-modal distillation target is also constructed. The video segment-level cross-modal distillation target is the difference between teacher-side features and student-side features within the same video segment after weighted aggregation according to the fluorescence reliability weight. The local cross-modal distillation target, the video segment-level cross-modal distillation target, and the occlusion reconstruction target of the native fluorescence spatiotemporal data unit are weighted and combined according to a preset weight to update the native fluorescence video encoder.

4. The method for processing endoscopic multimodal video data according to claim 1, characterized in that, The cross-modal feature constraint is determined by the distance between the features output by the white light video encoder and the fluorescent pseudo-color video encoder in the same intermediate spatiotemporal feature coding layer after projection and normalization. The modality adaptation layer is a bottleneck-type residual structure set at the output end of each spatiotemporal feature coding layer. It reduces the dimensionality of the input features and then increases the dimensionality, and adds it to the original output of the spatiotemporal feature coding layer without changing the number and arrangement order of the feature units. When updating the fluorescent pseudo-color video encoder, the fixed state of the parameters obtained by copying is released in stages. First, only the input normalization parameters and the modality adaptation layer are updated. Then, the fixed state of each spatiotemporal feature coding layer is gradually released from the output end to the input end.

5. The method for processing endoscopic multimodal video data according to claim 1, characterized in that, Also includes: In the target frame of white light video data, the central location of the lesion and multiple edge locations located in different directions of the lesion are determined. A local coordinate system is established with the central location of the lesion as the origin. The normalized distance, orientation angle, location identifier, and sequential relationship between each edge location relative to the central location of the lesion are recorded to form a lesion center-edge topology. The location identifier remains unchanged in continuous video data. Each point corresponds to the local analysis area containing that location.

6. The method for processing endoscopic multimodal video data according to claim 5, characterized in that, It also includes joint tracking of each point between adjacent frames of white light video data: generating candidate positions for each point, and determining an appearance matching score representing the similarity between the candidate position and the corresponding region in the previous frame, a motion consistency score representing the consistency between the candidate position and the preceding motion trajectory, and a topology consistency score representing the consistency between the spatial relationship of the candidate position relative to other points in the local coordinate system and the lesion center-edge topology; determining the position of each point in the current frame according to the weighted sum of the three, while maintaining the point identifier and the lesion center-edge topology; the weight of each score in the weighted sum is determined according to the tracking confidence of the corresponding point in the previous frame, the lower the tracking confidence, the greater the weight of the topology consistency score; the tracking confidence is determined according to the weighted sum of the candidate positions used by the point in the corresponding frame.

7. The method for processing endoscopic multimodal video data according to claim 6, characterized in that, When the highest weighted sum of a point in the current frame is lower than a preset acceptance threshold, its candidate position is not adopted. Instead, its current position is estimated based on the local transformation of other points with tracking confidence higher than a preset reliability threshold and the preceding motion trajectory of that point. The tracking confidence of that point in the current frame is set to a preset value. Tracking is resumed when the scores of that point meet the preset recovery conditions. According to the temporal index and spatial grid correspondence between the three video data, the local analysis region of each point is mapped to the fluorescence pseudo-color video data and the original fluorescence video data. The three corresponding local spatiotemporal data units are extracted and input into the corresponding video encoder and the three-modal fusion network respectively to obtain the local three-modal fusion features of each point.

8. The method for processing endoscopic multimodal video data according to claim 7, characterized in that, Also includes: The device imaging conditions and fluorescence signal formation conditions during video data acquisition are input into two independent conditional coding networks to obtain device conditional features and fluorescence conditional features. The device conditional features are combined with video imaging features, and the fluorescence conditional features are combined with fluorescence dynamic features. The combination is either splicing or generating parameters from the conditional features to adjust the corresponding features. Based on the changes in the effective pixel signals of the native fluorescence video data in the local analysis area of ​​each point over time, fluorescence curve parameters of each point are extracted. The differences in fluorescence curve parameters between each edge point and the lesion center point are arranged according to the site identifier, orientation angle, and order relationship of each edge point to form fluorescence dynamic difference features corresponding to the lesion center-edge topology. The local three-modal fusion features of each point, the fluorescence dynamic difference features, the device conditional features, the fluorescence conditional features, and the lesion center-edge topology are input into the lesion-level fusion network to output the lesion-level three-modal video representation.

9. An endoscopic multimodal video data processing system, characterized in that, include: The data correspondence module is used to acquire white light video data, fluorescence pseudo-color video data and native fluorescence video data generated during the same endoscopic examination, and divide the three video data into corresponding spatiotemporal data units according to a unified time index and spatial grid. The first migration module is used to copy the spatiotemporal feature coding layer parameters of the white light video encoder to the initial parameters of the fluorescent pseudo-color video encoder, configure independent input normalization parameters and modal adaptation layer for the fluorescent pseudo-color video encoder, and use the white light video encoder as a fixed teacher encoder to construct cross-modal feature constraints based on the difference between the output features of the two encoders after feature projection, thereby updating the fluorescent pseudo-color video encoder. The second migration module is used to take the updated fluorescence pseudo-color video encoder as a fixed teacher encoder, and according to the fluorescence reliability weight determined based on the fluorescence signal intensity after background subtraction, the effective pixel ratio and the saturated pixel ratio, weight the feature difference between the fluorescence pseudo-color spatiotemporal data unit and the original fluorescence spatiotemporal data unit after feature projection to obtain the local cross-modal distillation target, and update the original fluorescence video encoder accordingly. The feature fusion module is used to extract features from three corresponding spatiotemporal data units through the corresponding video encoders and combine them into a three-modal feature group, which is then input into the three-modal fusion network and outputs the three-modal fusion features.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the method for processing endoscopic multimodal video data according to any one of claims 1 to 8.

Images