Image data processing method and apparatus

Abstract

This invention provides an image data processing method and apparatus, belonging to the field of image data processing technology. The image data processing method of this invention introduces a signal attenuation feature evaluation and dynamic triggering mechanism, enabling electronic devices to actively determine the quality of optical signals at the data level and adaptively introduce ultrasonic compensation data, thus avoiding the loss of deep structural information caused by medium turbidity. Furthermore, through a cross-dimensional non-rigid registration network, precise spatial alignment of three-dimensional volume data and two-dimensional planar data is achieved, ensuring that the fused multimodal spatial volume data contains complementary structural information from different modalities at each spatial location. The final output target image, compared to a single-modal image, shows significant improvements in the integrity of structural information, the comprehensiveness of spatial coverage, and the clarity of deep regions.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to an image data processing method and apparatus. Background Technology

[0002] With the development of modern ophthalmology, multimodal imaging examinations such as optical coherence tomography (OCT), ultra-wide-angle fundus cameras, and ophthalmic ultrasound are playing an increasingly important role in the diagnosis of complex diseases such as fundus lesions and glaucoma. Among them, OCT can provide high-resolution retinal tomographic structures, fundus cameras can reflect a wide range of superficial vascular morphology, and ultrasound can penetrate turbid media to detect deep tissues.

[0003] However, existing optical imaging methods suffer from media-dependent limitations. When patients have refractive media opacities such as cataracts or vitreous hemorrhage, the optical signal of OCT is severely attenuated or even completely blocked. Existing systems typically passively receive low-quality OCT images, leading to the loss of effective structural information. Therefore, there is an urgent need for an image data processing method capable of acquiring image data containing complete deep structural information. Summary of the Invention

[0004] This invention provides an image data processing method and apparatus to address the deficiency of structural information in existing OCT images under specific scenarios, thereby achieving the effect of obtaining rich structural information.

[0005] This invention provides an image data processing method, comprising:

[0006] Acquire pre-collected optical coherence tomography data and synchronously acquired two-dimensional surface image data;

[0007] The optical coherence tomography (OCT) data is evaluated for signal attenuation characteristics to obtain attenuation characteristic values; when the attenuation characteristic values ​​are lower than a preset medium turbidity threshold, a high-frequency ultrasound trigger command is generated.

[0008] In response to the high-frequency ultrasound trigger command, the corresponding high-frequency ultrasound echo signal is received; the high-frequency ultrasound echo signal is coherently superimposed and time-delayed focusing is performed on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data.

[0009] The optical coherence tomography data is processed by applying a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold.

[0010] Using a pre-trained non-rigid registration network, the three-dimensional panoramic interferometric image data and the two-dimensional surface image data are spatially registered across dimensions, and the deep structure ultrasound image data is mapped to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0011] According to an image data processing method provided by the present invention, the step of applying a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to the optical coherence tomography data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold includes:

[0012] Extract consecutive adjacent scan frames from the optical coherence tomography data, and calculate the phase error matrix and pixel correlation matrix between adjacent scan frames;

[0013] Based on the pixel correlation matrix, local artifact pixel regions are identified, and pixel reconstruction is performed on the local artifact pixel regions to obtain the reconstructed scan frame;

[0014] The overlapping edge features of the multi-view volume data blocks composed of the reconstructed scan frames are extracted, and the multi-view volume data blocks are globally seamlessly stitched together to generate the three-dimensional panoramic interferometric image data.

[0015] According to an image data processing method provided by the present invention, the step of performing coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data includes:

[0016] Identify the received raw radio frequency data belonging to different channels;

[0017] The two-way flight time of each channel at different depths is calculated based on the preset medium sound velocity parameters to obtain the dynamic delay matrix;

[0018] The original radio frequency data is phase-aligned using the dynamic delay matrix, and the aligned data is coherently superimposed to reconstruct the deep structure ultrasound image data.

[0019] According to an image data processing method provided by the present invention, the step of performing cross-dimensional spatial registration of the three-dimensional panoramic interferometric image data and the two-dimensional surface image data using a pre-trained non-rigid registration network includes:

[0020] The three-dimensional panoramic interferometric image data is subjected to maximum intensity projection along the depth direction to generate a two-dimensional projection map with the same dimensional attributes as the two-dimensional surface image data.

[0021] The two-dimensional projection image and the two-dimensional surface image data are stitched together in the channel dimension to obtain a channel stitched image;

[0022] The channel mosaic is input into the non-rigid registration network, and a dense deformation field is output through the non-rigid registration network.

[0023] The three-dimensional panoramic interferometric image data is subjected to three-dimensional voxel-level spatial deformation distortion using the dense deformation field, so that the three-dimensional panoramic interferometric image data and the two-dimensional surface image data are aligned in the spatial coordinate system.

[0024] According to an image data processing method provided by the present invention, the step of constructing multimodal fusion spatial volume data to obtain a target image includes:

[0025] Image features of each modality are extracted from the multimodal fusion spatial volume data and input into a feature consistency verification network based on a contrastive learning framework to obtain the cross-modal feature consistency identifier output by the feature consistency verification network.

[0026] Based on the cross-modal feature consistency identifier, a target rendering instruction is generated for the feature difference region in the multimodal fused spatial volume data;

[0027] The target image is obtained based on the multimodal fused spatial volume data and the target rendering instructions.

[0028] According to an image data processing method provided by the present invention, the step of extracting image features of each modality from the multimodal fused spatial volume data and inputting them into a feature consistency verification network based on a contrastive learning framework to obtain a cross-modal feature consistency identifier output by the feature consistency verification network includes:

[0029] The registered interferometric mode feature vector and ultrasonic mode feature vector are extracted from the multimodal fused spatial volume data using a preset feature encoder.

[0030] The interference mode feature vector and the ultrasonic mode feature vector are input into the feature consistency verification network to determine the cosine similarity between the interference mode feature vector and the ultrasonic mode feature vector in the projection space;

[0031] If the cosine similarity is lower than a preset consistency threshold, a negative consistency identifier representing feature conflict is generated as the cross-modal feature consistency identifier.

[0032] According to an image data processing method provided by the present invention, the step of evaluating the signal attenuation characteristics of the optical coherence tomography data to obtain attenuation characteristic values ​​includes:

[0033] Calculate the signal-to-noise ratio attenuation gradient of the pixel layer below the preset pixel depth in the optical coherence tomography data;

[0034] Extract the optical signal reflection intensity parameters of the shallow pixel layer;

[0035] The attenuation characteristic value is determined based on the weighted calculation result of the signal-to-noise ratio attenuation gradient and the optical signal reflection intensity parameter.

[0036] The present invention also provides an image data processing apparatus, comprising:

[0037] The acquisition module is used to acquire pre-acquired optical coherence tomography data and synchronously acquired two-dimensional surface image data.

[0038] The first processing module is used to evaluate the signal attenuation characteristics of the optical coherence tomography data to obtain attenuation characteristic values; when the attenuation characteristic value is lower than a preset medium turbidity threshold, a high-frequency ultrasound trigger command is generated.

[0039] The response module is used to respond to the high-frequency ultrasound trigger command, receive the corresponding high-frequency ultrasound echo signal, and perform coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data.

[0040] The second processing module is used to apply a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to the optical coherence tomography data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold.

[0041] The third processing module is used to perform cross-dimensional spatial registration of the three-dimensional panoramic interferometric image data and the two-dimensional surface image data using a pre-trained non-rigid registration network, and to map the deep structure ultrasound image data to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image data processing method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image data processing method as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image data processing method as described above.

[0045] The image data processing method and apparatus provided by this invention introduce a signal attenuation feature evaluation and dynamic triggering mechanism, enabling electronic devices to actively determine the quality of optical signals at the data level and adaptively introduce ultrasonic compensation data, thus avoiding the loss of deep structural information caused by medium turbidity. Furthermore, through a cross-dimensional non-rigid registration network, precise spatial alignment between three-dimensional volume data and two-dimensional planar data is achieved. This ensures that the fused multimodal spatial volume data contains complementary structural information from different modalities at each spatial location. The final output target image, compared to a single-modal image, shows significant improvements in the integrity of structural information, the comprehensiveness of spatial coverage, and the clarity of deep regions. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the image data processing method provided by the present invention;

[0048] Figure 2 is a schematic diagram of the image data processing device provided by the present invention;

[0049] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined with Figures 1-3 This invention describes an image data processing method and apparatus. The image data processing method provided by this invention can be executed by an electronic device and addresses the problem of loss of effective structural information caused by the passive reception of low-quality optical coherence tomography (OCT) images in existing multimodal image processing systems. The image data processing method of this invention optimizes the data processing flow to generate high-quality target images containing complete structural information.

[0052] As shown in Figure 1, the image data processing method of this embodiment mainly includes steps 110, 120, 130, 140 and 150.

[0053] Step 110: Acquire pre-collected optical coherence tomography data and synchronously acquired two-dimensional surface image data.

[0054] Optical coherence tomography (OCT) data can be interference signal data pre-acquired and stored using an OCT device. OCT is an imaging technique based on the principle of low-coherence optical interference. It emits a broadband light source towards the target tissue, receives light signals reflected from different depths of the tissue, and interferes with the light signals of a reference arm to generate a tomographic image reflecting the microscopic hierarchical structure within the tissue.

[0055] In this embodiment, the optical coherence tomography (OCT) data can be digitized interferometric signal data that has been acquired and stored on a local disk or transmitted to an electronic device via a network. The data format can be raw frequency domain interferometric spectral data or voxel grayscale data after preliminary Fourier transform processing. The OCT data contains optical tomographic structural information about the internal tissues of the target object, such as the reflection intensity distribution at different depths.

[0056] Two-dimensional surface image data can be two-dimensional planar image data that is synchronously acquired and stored using a wide-angle imaging device. In this embodiment, the two-dimensional surface image data can be two-dimensional structural images covering a large field of view acquired using an ultra-wide-angle camera device, and its data format can be a standard bitmap format or a medical digital imaging and communication format. The two-dimensional surface image data contains morphological information of the surface tissue of the target object, such as the distribution and orientation of blood vessels and the contour of the surface structure.

[0057] In one possible implementation, the electronic device can read pre-acquired optical coherence tomography (OCT) data files from an external storage device or network storage system via a standard data communication interface. This data file can be frequency-domain interferometric spectral data stored in a medical digital imaging and communication standard format, or it can be the raw acquired data. Simultaneously, the electronic device acquires two-dimensional surface image data acquired in time with the OCT data via another data channel. This two-dimensional surface image data can be a high-resolution planar image captured by an ultra-wide-angle camera.

[0058] In another possible implementation, the electronic device is provided with a data receiving buffer, which can receive data streams from the acquisition device in real time streaming mode, and after frame synchronization and alignment in the buffer, it is used as input data for subsequent processing.

[0059] Step 120: Evaluate the signal attenuation characteristics of the optical coherence tomography data to obtain attenuation characteristic values; when the attenuation characteristic value is lower than the preset medium turbidity threshold, generate a high-frequency ultrasound trigger command.

[0060] The propagation characteristics of optical signals in optical coherence tomography (OCT) data can be quantitatively analyzed to obtain attenuation characteristic values. These attenuation characteristic values ​​characterize the degree of intensity loss of the optical signal as it passes through a target object. A lower attenuation characteristic value indicates more severe optical signal loss and a greater likelihood that the target object contains media components that obstruct optical signal propagation.

[0061] The preset medium turbidity threshold is pre-set and stored in the electronic device, and is used to determine whether the penetration quality of the optical signal meets the minimum requirements for subsequent image processing. When the attenuation characteristic value is lower than this threshold, it indicates that the effective structural information of the deep region in the currently acquired optical coherence tomography data is severely insufficient.

[0062] In this embodiment, the preset medium turbidity threshold can be set based on the statistical analysis results of a large amount of historical data. For example, it can be set to 0.35, which means that the compensation mechanism is triggered when the deep signal retention rate is lower than 35%.

[0063] High-frequency ultrasonic trigger commands are control command data automatically generated by electronic devices based on attenuation characteristic evaluation results. These commands can be used to notify external data transmission channels or buffer queues to send high-frequency ultrasonic acquisition data corresponding to the current target object into the processing flow.

[0064] In one possible implementation, the signal intensity of OCT data at a preset pixel depth, such as 200 micrometers below the retinal pigment epithelium, can be extracted. The ratio of this signal intensity to the signal intensity at a shallower layer, such as the nerve fiber layer, is calculated to obtain the signal-to-noise ratio attenuation gradient. Simultaneously, the reflection intensity parameter of the shallowest pixel layer is extracted. These two parameters are then weighted according to preset weights to obtain the attenuation characteristic value.

[0065] In another possible implementation, the processor of the electronic device can perform layer-by-layer analysis of the acquired optical coherence tomography (OCT) data. Specifically, the processor first selects the pixel values ​​located in the shallow layer of the data as the reference signal intensity, and then gradually scans the average signal intensity of each layer towards deeper layers. When it is found that the attenuation rate of signal intensity with increasing depth is significantly greater than the standard attenuation curve under normal tissue conditions, attenuation characteristic values ​​can be calculated.

[0066] For example, the attenuation characteristic value can be determined by a weighted calculation of the signal-to-noise ratio attenuation gradient and the shallow surface reflection intensity parameter. When the calculated attenuation characteristic value is lower than the medium turbidity threshold, such as 0.35, pre-stored in the electronic device, the processor determines that there is insufficient effective information in the deep region of the current optical coherence tomography data, and then generates a high-frequency ultrasound trigger command. The trigger command can be written into the command queue or sent to the ultrasound data processing channel via the system bus.

[0067] In another possible implementation, the electronic device can use a pre-trained signal quality assessment classifier to quickly evaluate optical coherence tomography data. The signal quality assessment classifier uses a large amount of historically acquired optical data as training samples, learns the difference between normal signal patterns and attenuated signal patterns, and can quickly output attenuation feature values ​​and compare them with a threshold.

[0068] Step 130: In response to the high-frequency ultrasound trigger command, receive the corresponding high-frequency ultrasound echo signal; perform coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data.

[0069] The high-frequency ultrasound echo signal can be radio frequency echo data pre-acquired and buffered by an ultrasound transducer array with an operating frequency of not less than 18 MHz. High-frequency ultrasound can utilize the physical penetrating properties of sound waves to pass through optically impermeable turbid media and obtain reflection information from deep tissues. In this embodiment, the high-frequency ultrasound echo signal can be the raw radio frequency data array that has already been acquired and stored in a buffer memory or data queue.

[0070] Coherent superposition and time-delay focusing are steps in signal processing based on synthetic aperture reconstruction algorithms. Synthetic aperture technology utilizes echo signals received from multiple receiving channels at different spatial locations, performs time-delay correction based on the propagation time difference of sound waves in the medium, and then superimposes the corrected multi-channel signals in terms of amplitude and phase. Mathematically, this is equivalent to using a large-aperture receiving array much larger than a single physical probe, thereby significantly improving spatial resolution and signal-to-noise ratio.

[0071] After processing by the synthetic aperture reconstruction algorithm, high-resolution ultrasound image data can be generated, which in turn can yield deep structural ultrasound image data. High-resolution ultrasound image data reflects the tissue structure information of the deep region of the target object, including the location, morphology, and reflection characteristics of deep tissue interfaces.

[0072] In one possible implementation, when the instruction queue in the electronic device receives a high-frequency ultrasonic trigger instruction, the data scheduler can read the high-frequency ultrasonic echo signal data corresponding to the current target object from the ultrasonic data buffer.

[0073] The electronic device can perform a synthetic aperture reconstruction algorithm on the multi-channel radio frequency data. Specifically, this includes: calculating the channel delay for each pixel based on the sound wave propagation velocity in the medium and the spatial relationship between the probes in each channel; performing precise time delay compensation on the data for each channel; and then coherently superimposing the compensated data from each channel to achieve a focusing effect at each pixel. After this processing, the resulting deep structural ultrasound image data can have a spatial resolution and signal-to-noise ratio far exceeding that of single-channel ultrasound imaging, clearly revealing the structural details of deep tissues.

[0074] Step 140: Apply a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching to the optical coherence tomography data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold.

[0075] It should be noted that the motion artifact removal algorithm based on inter-frame correlation feedback is an image processing algorithm that uses the pixel correlation between consecutive scan frames to detect and correct image artifacts introduced by the slight movement of the target object. By calculating the phase deviation and pixel matching degree between adjacent frames, it can automatically identify the regions affected by motion interference and use information from the undisturbed regions to compensate and reconstruct the interfered regions.

[0076] The preset wide-angle threshold is a pre-defined minimum requirement for the field of view, which can be set to 140 degrees in this embodiment. Image data from multiple local scanned areas can be spatially stitched together in the frequency domain to generate a complete image covering a wider field of view, thus achieving frequency domain panoramic stitching processing. Frequency domain stitching utilizes Fourier transform to convert the image from the spatial domain to the frequency domain. In the frequency domain, methods such as phase correlation are used to accurately calculate the spatial displacement relationship between different local images, and then seamless fusion and stitching are performed in the spatial domain. The field of view parameter is the angular range covered by the generated three-dimensional panoramic interferometric image data; the field of view parameter characterizes the spatial coverage breadth of the image data.

[0077] After motion artifact elimination and frequency domain panoramic stitching, optical interferometric data containing complete structural information in all three spatial dimensions and covering a large field of view, i.e., three-dimensional panoramic interferometric image data, can be obtained.

[0078] In one possible implementation, the electronic device can extract a sequence of consecutive scan frames from the acquired optical coherence tomography (OCT) data in chronological order of acquisition. For two consecutive adjacent scan frames, the processor can calculate the grayscale difference and phase difference at each corresponding pixel position between the two frames, generating a pixel correlation matrix. Further, by analyzing regions in the pixel correlation matrix where the correlation is significantly lower than the mean, local artifact pixel regions caused by minute movements can be identified. Then, an adaptive phase compensation algorithm is used to reconstruct the artifact regions at the pixel level, i.e., using weighted interpolation replacement based on the effective values ​​of the pixel in multiple preceding and following frames. The adaptive phase compensation algorithm can be based on motion estimation and resampling after optical flow, or on a deep learning-based denoising network; no limitation is made here.

[0079] After artifact removal, features of overlapping edge regions can be extracted from multiple processed local volume data blocks. Then, the spatial displacement and rotation parameters between each data block are accurately calculated using a frequency domain phase correlation algorithm. Finally, seamless fusion and stitching are performed in the spatial domain to generate three-dimensional panoramic interferometric image data covering a field of view greater than 140 degrees.

[0080] Step 150: Using a pre-trained non-rigid registration network, cross-dimensional spatial registration is performed between the three-dimensional panoramic interferometric image data and the two-dimensional surface image data. The deep structure ultrasound image data is then mapped to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0081] Non-rigid registration networks are deep learning-based image registration models that learn the nonlinear spatial deformation relationship between two input images and output a dense deformation field. Non-rigid registration networks can be based on the U-Net architecture or derived from a Voxel Morph model based on convolutional neural networks. Unlike rigid registration, which only involves translation and rotation, non-rigid registration allows for deformations of different directions and magnitudes in local regions of the image, thus adapting to non-uniform changes in tissue structures.

[0082] A spatial coordinate system is used as a unified reference frame to describe the spatial positions of pixels or voxels in image data of different modalities. Registration maps data from different modalities to the same spatial coordinate system, establishing the correspondence between the same spatial position in different modalities. Cross-dimensional spatial registration can establish a spatial correspondence between volume data in three-dimensional space and two-dimensional planar image data. Because the two types of data have different dimensions, techniques such as projection or dimensionality reduction are needed to convert the three-dimensional data into a form comparable to the two-dimensional data before using registration algorithms to establish the spatial mapping relationship.

[0083] Multimodal fusion spatial volume data is a comprehensive volumetric data formed by fusing registered and coordinate-mapped 3D panoramic interferometric image data, 2D surface image data, and deep structural ultrasound image data in a unified spatial coordinate system. Each spatial location in this volumetric data simultaneously contains structural information from different modalities, thus achieving complementary fusion of multi-source information. After the aforementioned processing, output image data containing complete structural information can be generated.

[0084] In one possible implementation, the electronic device first performs maximum intensity projection processing on the 3D panoramic interferometric image data along the depth direction, reducing its dimensionality to a 2D projection image with the same dimension as the 2D surface image data. Then, this 2D projection image and the 2D surface image data are stitched together along the channel dimension to form a multi-channel input image, which is fed into a pre-trained non-rigid registration network. This network outputs a dense deformation field describing the displacement required for each pixel in the 2D projection image to move to its corresponding position in the 2D surface image data. This dense deformation field is used to perform a non-rigid distortion transformation in 3D space on the original 3D panoramic interferometric image data, precisely aligning it with the 2D surface image data in a unified spatial coordinate system. Subsequently, the deep structure ultrasound image data is mapped to the same spatial coordinate system according to known acquisition geometry parameters and spatially fused with the registered 3D panoramic interferometric image data and 2D surface image data to construct multimodal fused spatial volume data. This multimodal fused spatial volume data is the target image containing complete structural information.

[0085] In some embodiments, the training data for the non-rigid registration network can be derived from publicly available multimodal medical image databases and anonymized image data provided by relevant institutions. Specifically, a large amount of optical coherence tomography (OCT) volume data and corresponding two-dimensional surface image data can be obtained from publicly available datasets. Alternatively, at least one hundred cases of multimodal image data can be collected from relevant institutions, with each case containing one to three sets of OCT data and two-dimensional surface image data, thereby covering a variety of common tissue structure types.

[0086] It should be noted that all data has undergone strict privacy desensitization before entering the training process.

[0087] In some embodiments, experienced imaging experts can use professional medical image annotation tools to annotate the training data. Specific annotations include: annotating key anatomically significant feature points, such as the optic disc center, fovea centralis, and major vascular bifurcation points, in both the two-dimensional projection image and the two-dimensional surface image data of the optical coherence tomography (OCT) scan data, thereby establishing corresponding feature point pairs between the two images.

[0088] Before inputting the data into the model, the raw optical coherence tomography (OCT) data can be normalized to linearly map the grayscale values ​​to the range of zero to one. Histogram equalization can be applied to the two-dimensional surface image data to enhance contrast, and the data can be uniformly adjusted to a fixed resolution, such as 512 pixels multiplied by 512 pixels.

[0089] In some embodiments, data augmentation can be performed on the training data, such as random rotation, random scaling, random flipping, and random addition of Gaussian noise, which can improve the generalization ability of the model.

[0090] In some embodiments, the non-rigid registration network can employ a U-Net encoder-decoder architecture. The encoder consists of four convolutional downsampling layers, each containing two 3x3 convolutional layers, followed by a 2x2 pooling layer and a linear rectified activation function layer. Spatial reduction is achieved between layers through a 2-stride downsampling operation. The number of channels in each encoder layer can be 16, 32, 64, or 128, respectively. The decoder can be symmetrical to the encoder, containing four upsampling layers. Each layer performs a 2x upsampling through bilinear interpolation, and features from the same layer as the encoder are concatenated and fused via skip connections before being processed by two convolutional layers.

[0091] The final layer of the decoder can use a convolutional layer to output a two-dimensional dense deformation field, which corresponds to the displacement components of each pixel in the input image in both the horizontal and vertical directions. The input layer receives a two-channel image after channel dimension stitching, with the two-dimensional projection map as one channel and the two-dimensional surface image data as another channel. The output layer can output a two-channel dense deformation field.

[0092] Specifically, after receiving the stitched dual-channel input image, the non-rigid registration network can extract multi-scale features layer by layer through the encoder, capture global context information at the bottleneck layer, and then restore the spatial resolution layer by layer through the decoder to generate a dense deformation field. The dense deformation field can be applied to one of the input images, and the input image is non-rigidly distorted at the pixel level through the spatial transformer layer to align it spatially with the other image.

[0093] In some embodiments, the training of the non-rigid registration network can employ a combination of end-to-end unsupervised training and supervised training strategies. The loss function may include image similarity loss, deformation field smoothness regularization loss, and key feature point matching loss.

[0094] Image similarity loss can use normalized cross-correlation coefficients to measure the similarity between the distorted image and the target image; deformation field smoothness regularization loss can calculate the gradient difference of the displacement of adjacent pixels in the deformation field to constrain the spatial smoothness of the deformation field and avoid abrupt deformations that do not conform to physical laws; key feature point matching loss can use expert-annotated corresponding feature point pairs to calculate the Euclidean distance between the distorted feature point position and the target feature point position.

[0095] It is understandable that the three loss functions in the above embodiments can be weighted and summed according to certain weights, such as weighted summation with weight ratios of 0.5, 0.2, and 0.3. After training is completed, the network parameters can be fixed and deployed.

[0096] Understandably, during the actual inference phase, the two-dimensional projection image and the two-dimensional surface image data to be registered can be stitched together along the channel dimension and then input into a non-rigid registration network. The network outputs a dense deformation field during forward propagation. The electronic device uses this dense deformation field to perform non-rigid distortion operations on each slice of the three-dimensional panoramic interferometric image data through a spatial transformer layer, thereby completing cross-dimensional spatial registration.

[0097] In this embodiment, two types of source data can be acquired first: pre-collected optical coherence tomography data and two-dimensional surface image data. Then, the signal attenuation of the optical data is actively evaluated. When the optical signal penetration quality is found to be insufficient, high-frequency ultrasound compensation data is dynamically triggered. Next, the optical data and ultrasound data are reconstructed at the bottom layer respectively. Finally, all modal data are unified into the same spatial coordinate system and fused through a deep learning non-rigid registration network to output a target image containing complete structural information.

[0098] The image data processing method provided by the embodiments of the present invention introduces a signal attenuation feature evaluation and dynamic triggering mechanism, enabling electronic devices to actively determine the quality of optical signals at the data level and adaptively introduce ultrasonic compensation data, thus avoiding the loss of deep structural information caused by medium turbidity. Furthermore, through a cross-dimensional non-rigid registration network, precise spatial alignment of three-dimensional volume data and two-dimensional planar data is achieved, ensuring that the fused multimodal spatial volume data contains complementary structural information from different modalities at each spatial location. The final output target image, compared to a single-modal image, shows significant improvements in the integrity of structural information, the comprehensiveness of spatial coverage, and the clarity of deep regions.

[0099] In some possible embodiments, motion artifact elimination algorithms based on inter-frame correlation feedback and frequency domain panoramic stitching are applied to optical coherence tomography (OCT) data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold. This includes: extracting consecutive adjacent scan frames from the OCT data and calculating the phase error matrix and pixel correlation matrix between adjacent scan frames; identifying local artifact pixel regions based on the pixel correlation matrix and reconstructing the local artifact pixel regions to obtain reconstructed scan frames; extracting the overlapping edge features of multi-view volume data blocks composed of reconstructed scan frames and performing global seamless stitching of the multi-view volume data blocks to generate three-dimensional panoramic interferometric image data.

[0100] In optical coherence tomography (OCT) data, a single tomographic image arranged chronologically by acquisition time constitutes a scan frame. Each scan frame corresponds to the optical interference signal distribution of a spatial section, and a continuous sequence of scan frames forms three-dimensional volumetric data. The phase difference between corresponding pixel positions in two consecutive adjacent scan frames forms a phase error matrix. Phase error reflects the phase shift of the interference signal caused by minute movements of the target object. Under normal circumstances, the phase change between adjacent frames should be within a predictable range; a phase difference exceeding this range indicates motion interference. The correlation coefficient of grayscale values ​​between corresponding pixel positions in two consecutive adjacent scan frames forms a pixel correlation matrix. A higher pixel correlation coefficient indicates better consistency between corresponding regions in the two frames; regions with significantly lower correlation coefficients indicate the presence of artifacts or motion interference.

[0101] In the pixel correlation matrix, connected pixel regions with correlation coefficients below a preset artifact recognition threshold are designated as local artifact pixel regions. Pixel values ​​within these local artifact pixel regions undergo unpredictable shifts due to the movement of the target object, no longer accurately reflecting tissue structure information.

[0102] Understandably, by reconstructing pixels in local artifact regions, a repaired scan frame, or reconstructed scan frame, can be obtained. The reconstruction process can utilize the effective values ​​of the same pixel location in multiple scan frames that are not affected by artifacts to perform weighted interpolation calculations, generating replacement values ​​to fill the artifact regions.

[0103] Multiple sets of reconstructed scan frames, arranged spatially according to their respective scanning angles, can form a set of 3D data blocks consisting of multi-view volumetric data blocks. Each volumetric data block corresponds to a scanning region within a local viewpoint, and there are spatially overlapping regions between multiple volumetric data blocks.

[0104] Overlapping edge features refer to image features extracted from the spatially overlapping region of adjacent volume data blocks to accurately locate the relative spatial relationship between the two data blocks. These features can include the gradient direction of the edges, texture patterns, feature point distribution, etc.

[0105] The spatial displacement parameters between multiple volume data blocks can be accurately estimated using a frequency domain phase correlation algorithm. Then, the overlapping areas are gradually fused in the spatial domain to eliminate stitching seams. Finally, all volume data blocks are merged into a three-dimensional panoramic interferometric image data covering the entire field of view, thereby achieving global seamless stitching.

[0106] In one possible implementation, the electronic device extracts consecutively numbered scan frames, such as frame n and frame (n+1), from the optical coherence tomography (OCT) data according to their timestamps. For these two scan frames, a normalized cross-correlation coefficient between the grayscale values ​​of the two frames can be calculated at each corresponding pixel coordinate position, generating a pixel correlation matrix with the same spatial size as the scan frames. Simultaneously, the phase difference at each pixel position can be calculated using the complex representation of the interference signals from the two frames, generating a phase error matrix.

[0107] Subsequently, the pixel correlation matrix can be scanned to find pixel regions with correlation coefficients lower than a preset artifact recognition threshold. These low-correlation pixels are then aggregated into several local artifact pixel regions using a connected component analysis algorithm. For each pixel in each local artifact pixel region, the median value of that pixel in the preceding and following five scan frames can be filtered, and the median value is taken as the reconstructed value to replace the original artifact value, thus obtaining the reconstructed scan frame.

[0108] After artifact removal of all scan frames, the reconstructed scan frames can be grouped into multiple volume data blocks according to the scanning angle. For two adjacent volume data blocks, edge gradient features within their spatially overlapping regions can be extracted. Two-dimensional Fourier transforms are then performed on the data in the overlapping regions, and the cross-power spectrum of the two data blocks is calculated. An inverse Fourier transform is then performed to obtain the phase correlation peak, the position of which corresponds to the precise spatial displacement between the two data blocks. Using this displacement, the volume data blocks are spatially translated and aligned. A linear weighted gradual fusion strategy is then applied to the overlapping regions to eliminate stitching seams, ultimately generating three-dimensional panoramic interferometric image data covering the entire field of view.

[0109] In this embodiment, by introducing a dual detection mechanism of pixel correlation matrix and phase error matrix, artifact regions can be accurately located without damaging normal pixels, significantly improving the accuracy of artifact recognition. Furthermore, the pixel reconstruction method using multi-frame median filtering can avoid new noise that may be introduced by single-frame interpolation.

[0110] It should be noted that, compared with traditional feature point matching and stitching methods, the frequency domain phase correlation algorithm has sub-pixel level registration accuracy and is more robust to changes in image content and noise, ensuring the spatial continuity and consistency of the stitched 3D panoramic interferometric image data.

[0111] In some possible embodiments, coherent superposition and time-delay focusing processing is performed on high-frequency ultrasound echo signals to generate deep structure ultrasound image data, including: identifying the received original radio frequency data belonging to different channels; calculating the two-way flight time of each channel at different depths according to preset medium sound velocity parameters to obtain a dynamic delay matrix; using the dynamic delay matrix to perform phase alignment on the original radio frequency data, and coherently superimposing the aligned data to reconstruct deep structure ultrasound image data.

[0112] In a high-frequency ultrasonic transducer array, each independent receiving channel can acquire unprocessed radio frequency echo signal data. Each channel corresponds to the physical location of an independent receiving element in the array, and the echo signals generated by the same acoustic pulse at the same reflecting target acquired by each channel have different arrival times.

[0113] The preset medium sound velocity parameter can be a pre-calibrated value stored in the electronic device representing the speed at which sound waves propagate in the target medium. For example, this value can be set to 1540 meters per second, corresponding to the typical propagation speed of sound waves in biological soft tissue. The two-way flight time refers to the total time it takes for a sound wave to travel from the transmitting element, through the medium, to the target reflection point, and then back to the receiving element.

[0114] The dynamic delay matrix is ​​a two-dimensional data matrix composed of delay compensation amounts calculated for each target pixel and each receiving channel in the synthetic aperture reconstruction process. The rows of the matrix correspond to different target pixels, i.e., various spatial locations in the image to be reconstructed, and the columns correspond to different receiving channels. The value of each element in the matrix is ​​the delay compensation amount required by that receiving channel for that target pixel.

[0115] In one possible implementation, the electronic device can first identify the original radio frequency data streams belonging to different channels in the received high-frequency ultrasonic echo signal. For example, for a linear array consisting of 128 receiver elements, the electronic device can separate the received data into 128 independent radio frequency data sequences according to channel number and buffer them in memory.

[0116] Next, based on the preset medium sound velocity parameters and the known spatial coordinates of each receiving primitive, the two-way flight time of each channel can be calculated for each target pixel to be reconstructed. Specifically, for a pixel to be reconstructed in the target image, the processor calculates the distance from the transmitting primitive to that pixel, adds the distance from that pixel to each receiving primitive, and then divides the total distance by the medium sound velocity to obtain the two-way flight time of that receiving channel for that pixel. The two-way flight times of all target pixels and all receiving channels are organized into a dynamic delay matrix.

[0117] Then, the electronic device uses a dynamic delay matrix to perform phase alignment processing on the raw radio frequency data of each channel. Specifically, for each target pixel, the sampled value at the corresponding time moment can be extracted from the radio frequency data of each channel according to the delay amount corresponding to that channel, so that the signals of all channels are aligned to the same equivalent reception time in time. Finally, the amplitude of the aligned sampled values ​​of each channel is superimposed. Since the effective signal from the same reflecting target is coherent after alignment while random noise is not coherent, the amplitude of the effective signal after superposition increases to several times that of the original channel, while the noise amplitude only increases to the square root of the original channel number, thereby significantly improving the signal-to-noise ratio and reconstructing high-resolution deep structure ultrasound image data.

[0118] In this embodiment, precise phase alignment of signals from each channel is achieved by using a dynamic delay matrix based on two-way flight time, and the signal-to-noise ratio and spatial resolution are significantly improved by coherent superposition, enabling clear images of deep tissue structures to be obtained even under high-frequency conditions, effectively compensating for the lack of structural information when optical signals are limited.

[0119] In some possible embodiments, a pre-trained non-rigid registration network is used to perform cross-dimensional spatial registration of 3D panoramic interferometric image data and 2D surface image data. This includes: performing maximum intensity projection processing on the 3D panoramic interferometric image data along the depth direction to generate a 2D projection map with the same dimensional attributes as the 2D surface image data; stitching the 2D projection map and the 2D surface image data in the channel dimension to obtain a channel stitched map; inputting the channel stitched map into the non-rigid registration network, and outputting a dense deformation field through the non-rigid registration network; and using the dense deformation field to perform 3D voxel-level spatial deformation distortion on the 3D panoramic interferometric image data to align the 3D panoramic interferometric image data and the 2D surface image data in the spatial coordinate system.

[0120] Maximum intensity projection (MIP) is a process that compresses 3D volumetric data into a single 2D image by taking the maximum pixel value across all depth layers at each 2D plane coordinate location along the depth dimension of the 3D panoramic interferometric image data (perpendicular to the imaging plane). MIP preserves the structural information with the strongest signal in each depth layer.

[0121] The two-dimensional planar image obtained from the three-dimensional panoramic interferometric image data through the above-described maximum intensity projection processing is the two-dimensional projection map. The two-dimensional projection map and the two-dimensional surface image data have the same dimensional attribute, i.e., both are two-dimensional planar images, allowing for direct pixel-level comparison and registration operations. Stitching the two-dimensional projection map and the two-dimensional surface image data along the image channel dimension yields a channel-stitched image. For example, if the two-dimensional projection map is a single-channel grayscale image and the two-dimensional surface image data is a three-channel color image, then the channel-stitched image is a four-channel image. In this embodiment, to simplify processing, both images can be converted to single-channel grayscale images before stitching to obtain a two-channel stitched image.

[0122] A dense deformation field is a two-dimensional vector field with the same spatial dimensions as the input image, output by a non-rigid registration network. Each pixel location stores a two-dimensional displacement vector, representing the horizontal and vertical displacements that the pixel needs to move during the registration transformation. The displacements in the dense deformation field are spatially continuous and can express complex nonlinear local deformation relationships.

[0123] By utilizing the displacement information in the dense deformation field, an independent spatial remapping operation can be performed on each voxel of each slice in the 3D panoramic interferometric image data, thereby obtaining 3D voxel-level spatial deformation distortion. Since the dense deformation field is generated on a 2D projection plane, for 3D data, the same 2D deformation field can be applied to each depth slice of the 3D data, thus achieving non-rigid spatial distortion of the entire 3D volume data in both horizontal and vertical directions.

[0124] In one possible implementation, the electronic device can first read the three-dimensional panoramic interferometric image data obtained after artifact removal and panoramic stitching. The spatial dimension of this data is width multiplied by height multiplied by depth, where the depth dimension corresponds to several tomographic slice layers. Each two-dimensional planar coordinate position can be traversed along the depth dimension, and the maximum pixel value at that position across all depth layers can be taken to generate a two-dimensional projection image with width multiplied by height.

[0125] Then, the two-dimensional projection image and the synchronously acquired two-dimensional surface image data are both adjusted to a uniform spatial resolution and stitched together along the channel dimension of the images to generate a dual-channel stitched image. Next, the electronic device can input the channel stitched image into a non-rigid registration network. The encoder part performs multi-scale feature extraction on the dual-channel input to capture the local morphological differences and global spatial relationships between the two images, and the decoder part generates a dense deformation field under the guidance of the multi-scale features.

[0126] Finally, a spatial transformer layer can be used to apply a dense deformation field to each depth slice of the 3D panoramic interferometric image data. For each voxel, bilinear interpolation resampling is performed based on the displacement vector at the corresponding position in the deformation field, resulting in spatially distorted 3D panoramic interferometric image data. The distorted 3D data and the 2D surface image data are precisely aligned in a unified spatial coordinate system.

[0127] Understandably, by employing the dimensionality reduction strategy of maximum projection, the 3D-to-2D registration problem can be transformed into a 2D-to-2D registration problem, significantly reducing computational complexity while preserving crucial structural correspondence information. Furthermore, utilizing the dense deformation field output by deep learning networks for voxel-level spatial distortion can handle complex nonlinear deformations of tissue structures, achieving registration accuracy far superior to traditional rigid registration methods based on feature point matching. This provides a spatial foundation for the accurate fusion of subsequent multimodal data.

[0128] In some possible embodiments, constructing multimodal fusion spatial volume data to obtain a target image includes: extracting image features of each modality from the multimodal fusion spatial volume data, inputting them into a feature consistency verification network based on a contrastive learning framework, obtaining a cross-modal feature consistency identifier output by the feature consistency verification network; generating target rendering instructions for feature difference regions in the multimodal fusion spatial volume data based on the cross-modal feature consistency identifier; and obtaining the target image based on the multimodal fusion spatial volume data and the target rendering instructions.

[0129] Numerical values ​​reflecting the image content characteristics of a particular modality can be extracted from different modal components of multimodal fusion spatial volume data and used as image features for each modality. For example, features extracted from interferometric modal data can reflect the clarity and continuity of interlayer boundaries, while features extracted from ultrasound modal data can reflect the reflection intensity distribution and interface morphology of deep tissues.

[0130] A feature consistency verification network can be constructed using a deep learning network trained with contrastive learning methods to determine whether features from different modalities semantically describe the same structural information. This network can learn to map features from different modalities describing the same structure to neighboring locations in the projection space, while mapping features describing different structures to distant locations in the projection space.

[0131] Cross-modal feature consistency identifiers refer to the identifier information output by the feature consistency verification network, used to characterize whether there is semantic consistency between image features from different modalities. When features from different modalities are close in the projection space, a positive consistency identifier can be output, indicating that the observation results of different modalities corroborate each other; when they are far apart, a negative consistency identifier, i.e., a feature conflict identifier, can be output, indicating that the observation results of different modalities contradict each other.

[0132] In the spatial coordinate system of multimodal fusion volumetric data, the spatial regions where different modal image features are judged to be in conflict, i.e., outputting negative consistency indicators, can be identified as feature difference regions.

[0133] Electronic devices can automatically generate target rendering instructions for controlling the image display method based on cross-modal feature consistency identifiers. The target rendering instructions can include information such as the coordinates of the spatial area to be highlighted, rendering color parameters, and transparency parameters.

[0134] In one possible implementation, the electronic device extracts features of the interferometric mode and the ultrasonic mode from the constructed multimodal fusion spatial volume data. Feature extraction can be performed using a preset feature encoder, which inputs the voxel data belonging to the interferometric mode and the voxel data belonging to the ultrasonic mode from the multimodal fusion spatial volume data into their respective feature encoders, and outputs a fixed-length feature vector.

[0135] Furthermore, the extracted feature vectors from different modalities can be input into a feature consistency verification network based on a contrastive learning framework. The feature consistency verification network can calculate the distance or similarity index between feature vectors of different modalities in the projection space, determine whether the features of the two modalities are semantically consistent based on the distance value, and output a cross-modal feature consistency identifier.

[0136] Electronic devices can locate regions of feature discrepancies based on the cross-modal feature consistency markers output. For spatial regions identified as feature conflicts, the electronic device generates a target rendering instruction that specifies that the region should be highlighted and overlaid in the final output image with a specific color, such as red. Finally, based on the multimodal fused spatial volume data and the target rendering instruction, the structural information of the multimodal fusion and the highlighted feature difference annotation information can be overlaid to generate the final target image.

[0137] In some embodiments, the feature consistency verification network can employ a Siamese network architecture, such as including two SimCLR framework branches. The two branches can share weights, and each branch contains a feature encoder based on a residual network. The feature encoder can use an 18-layer residual network as its backbone, and the encoder can output a 512-dimensional feature vector. A projection head can be connected after the encoder. The projection head can consist of two fully connected layers, specifically implementing projection from 512 dimensions to 256 dimensions and from 256 dimensions to 128 dimensions, using a linear rectified activation function in between, ultimately projecting the feature vector into a 128-dimensional projection space.

[0138] Specifically, the extracted local image patches can be subjected to size unification, grayscale value normalization, and random enhancement processing. During the inference stage, the registered interferometric modal local image patches and ultrasonic modal local image patches can be input into the two branches of the Siamese network architecture, respectively, and each branch outputs a projection vector. The cosine similarity between the two projection vectors is calculated and compared with a preset consistency threshold. Finally, a cross-modal feature consistency identifier is output.

[0139] In some embodiments, the training process of the feature consistency verification network is as follows.

[0140] The training data can include spatially registered multimodal image data pairs from the aforementioned multimodal image database. Specifically, registered interferometric and ultrasonic modal data can be selected, and local image patches at corresponding spatial locations can be extracted as training sample pairs.

[0141] Samples can be labeled as positive and negative pairs. During training, a contrastive loss function can be used. For positive pairs, the loss function constrains the cosine similarity between the projected vectors of the two branches to approach one; for negative pairs, it constrains the cosine similarity to approach negative one or zero.

[0142] In this embodiment, by introducing a feature consistency verification mechanism based on contrastive learning, it is possible not only to fuse multimodal data, but also to actively identify and visually annotate contradictory regions between different modalities, which greatly enhances the information value of the target image. When users view the target image, they can not only see the fused structural information, but also discover the feature difference regions that need attention.

[0143] In some possible implementations, image features of each modality are extracted from the multimodal fusion spatial volume data and input into a feature consistency verification network based on a contrastive learning framework to obtain a cross-modal feature consistency identifier output by the feature consistency verification network. This includes: extracting registered interferometric modal feature vectors and ultrasonic modal feature vectors from the multimodal fusion spatial volume data using a preset feature encoder; inputting the interferometric modal feature vectors and ultrasonic modal feature vectors into the feature consistency verification network to determine the cosine similarity between the interferometric modal feature vectors and ultrasonic modal feature vectors in the projection space; and generating a negative consistency identifier representing feature conflict as a cross-modal feature consistency identifier when the cosine similarity is lower than a preset consistency threshold.

[0144] The pre-defined feature encoder can be a deep learning network module, such as ResNet-50, that is pre-trained and deployed in electronic devices to convert image data into fixed-length numerical vector representations. Different modalities of data can use their own dedicated encoders or a general encoder with shared weights; there is no restriction here.

[0145] Interferometric modal eigenvectors are fixed-length numerical vectors extracted by the feature encoder from the interferometric modal data portion of multimodal fusion spatial volume data. Ultrasonic modal eigenvectors are also fixed-length numerical vectors extracted by the feature encoder from the ultrasonic modal data portion of multimodal fusion spatial volume data.

[0146] The projection space is the low-dimensional space in which feature vectors are mapped by the projection head. In the projection space, semantically similar feature vectors cluster together, while semantically dissimilar feature vectors are far apart.

[0147] The preset consistency threshold is a pre-defined boundary value for judging cosine similarity. In this embodiment, the threshold can be set to 0.5. When the cosine similarity is lower than this value, it is considered that there is a significant conflict between the two modal features.

[0148] Understandably, by setting a pre-defined consistency threshold, automatic binarization of regions with feature differences is achieved, simplifying subsequent processing. The negative consistency flag provides a clear conflict indicator, enabling subsequent highlighting and manual review to accurately locate suspicious areas, thus improving the intelligence and accuracy of the diagnostic assistance system.

[0149] In some possible embodiments, signal attenuation characteristics are evaluated on optical coherence tomography (OCT) data to obtain attenuation characteristic values, including: calculating the signal-to-noise ratio (SNR) attenuation gradient of pixel layers below a preset pixel depth in the OCT data; extracting the optical signal reflection intensity parameters of shallow pixel layers; and determining the attenuation characteristic values ​​based on the weighted calculation results of the SNR attenuation gradient and the optical signal reflection intensity parameters.

[0150] The preset pixel depth is a reference position pre-set in the axial direction of the OCT image. It can be selected near the retinal pigment epithelium or the choroidal capillary layer to evaluate the signal quality of deep layers. The pixel layer is a two-dimensional image layer composed of all pixels in the horizontal (xy plane) at a specific depth location.

[0151] The signal-to-noise ratio attenuation gradient, i.e. the rate at which the signal-to-noise ratio decreases with increasing depth, can be obtained by calculating the difference or derivative of the signal-to-noise ratio at different depths, thus revealing the degree of energy attenuation of light propagating in the medium.

[0152] The superficial pixel layer can be the shallow pixel region near the cornea or the internal limiting membrane of the retina, usually corresponding to the anterior or upper region in an OCT image.

[0153] Optical signal reflection intensity parameters refer to statistical measures of signal intensity in the superficial pixel layer, such as average intensity, maximum intensity, or a specific intensity value. High reflection intensity may indicate the presence of a highly scattering medium (cataract, vitreous opacity).

[0154] Electronic devices can perform layered analysis on acquired optical coherence tomography (OCT) data. First, a preset pixel depth is determined. Within this depth, the data is divided into multiple pixel layers, such as 50 micrometers per layer. Then, the signal-to-noise ratio (SNR) of each layer and the rate of change of SNR between adjacent layers are calculated to obtain the SNR attenuation gradient. A larger gradient value indicates that the light signal attenuates faster during penetration, suggesting the possible presence of medium turbidity.

[0155] Simultaneously, optical signal reflection intensity parameters can be extracted from superficial pixel layers, such as those above the ILM to the anterior vitreous. The average grayscale value of all pixels in this region can be calculated as the reflection intensity parameter. If this value is abnormally high, exceeding the normal physiological range (e.g., low reflection in a normal vitreous body, and high reflection in the anterior capsule of the lens in cataracts), it indicates the presence of a strong scattering medium in the superficial layer.

[0156] The signal-to-noise ratio (SNR) attenuation gradient and the optical signal reflection intensity parameter can be weighted for calculation. For example, by setting the weight of the SNR attenuation gradient to 0.7 and the weight of the reflection intensity parameter to 0.3, the final attenuation characteristic value can be obtained. The attenuation characteristic value can reflect the degree of deep signal attenuation and shallow scattering. The worse the optical imaging quality, the more necessary it is to trigger high-frequency ultrasound for compensation.

[0157] This embodiment objectively quantifies the attenuation degree of OCT signals in deep tissues by calculating the signal-to-noise ratio attenuation gradient, directly reflecting the light transmittance of the medium. By extracting the reflection intensity parameters of the shallow pixel layer, the scattering and blocking effects of the anterior medium, such as cataracts and vitreous opacities, on light are effectively detected. Through a weighted calculation method, two complementary indicators are combined to generate a stable attenuation characteristic value, avoiding misjudgments that may be caused by a single indicator.

[0158] The image data processing apparatus provided by the present invention will be described below. The image data processing apparatus described below can be referred to in correspondence with the image data processing method described above.

[0159] As shown in Figure 2, the image data processing device of this embodiment includes an acquisition module 210, a first processing module 220, a response module 230, a second processing module 240, and a third processing module 250.

[0160] The acquisition module 210 is used to acquire pre-acquired optical coherence tomography data and synchronously acquired two-dimensional surface image data;

[0161] The first processing module 220 is used to evaluate the signal attenuation characteristics of optical coherence tomography data and obtain attenuation characteristic values; when the attenuation characteristic value is lower than the preset medium turbidity threshold, a high-frequency ultrasound trigger command is generated.

[0162] The response module 230 is used to respond to the high-frequency ultrasound trigger command, receive the corresponding high-frequency ultrasound echo signal, and perform coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data.

[0163] The second processing module 240 is used to apply a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to optical coherence tomography data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold.

[0164] The third processing module 250 is used to perform cross-dimensional spatial registration of three-dimensional panoramic interferometric image data and two-dimensional surface image data using a pre-trained non-rigid registration network, and to map deep structure ultrasound image data to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0165] The image data processing apparatus provided in this embodiment of the invention introduces a signal attenuation feature evaluation and dynamic triggering mechanism, enabling the electronic device to actively determine the quality of the optical signal at the data level and adaptively introduce ultrasonic compensation data, thus avoiding the loss of deep structural information caused by medium turbidity. Furthermore, through a cross-dimensional non-rigid registration network, precise spatial alignment between three-dimensional volume data and two-dimensional planar data is achieved, ensuring that the fused multimodal spatial volume data contains complementary structural information from different modalities at each spatial location. The final output target image, compared to a single-modal image, shows significant improvements in the integrity of structural information, the comprehensiveness of spatial coverage, and the clarity of deep regions.

[0166] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logic instructions in the memory 330 to execute an image data processing method, which includes: acquiring pre-acquired optical coherence tomography (OCT) data and synchronously acquired two-dimensional surface image data; evaluating the signal attenuation characteristics of the OCT data to obtain attenuation characteristic values; generating a high-frequency ultrasound trigger command when the attenuation characteristic value is lower than a preset medium turbidity threshold; receiving the corresponding high-frequency ultrasound echo signal in response to the high-frequency ultrasound trigger command; performing coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data; applying a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to the OCT data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold; using a pre-trained non-rigid registration network to perform cross-dimensional spatial registration of the three-dimensional panoramic interferometric image data and the two-dimensional surface image data, and mapping the deep structure ultrasound image data to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0167] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image data processing method provided by the above methods. The method includes: acquiring pre-acquired optical coherence tomography (OCT) data and synchronously acquired two-dimensional surface image data; evaluating the signal attenuation characteristics of the OCT data to obtain an attenuation characteristic value; generating a high-frequency ultrasonic trigger command when the attenuation characteristic value is lower than a preset medium turbidity threshold; and responding to the high-frequency ultrasonic trigger command. The system receives corresponding high-frequency ultrasonic echo signals; performs coherent superposition and time-delay focusing on the high-frequency ultrasonic echo signals to generate deep structural ultrasonic image data; applies a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to the optical coherence tomography data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold; and uses a pre-trained non-rigid registration network to perform cross-dimensional spatial registration between the three-dimensional panoramic interferometric image data and the two-dimensional surface image data, and maps the deep structural ultrasonic image data to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0169] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the image data processing method provided by the above methods. The method includes: acquiring pre-acquired optical coherence tomography (OCT) data and synchronously acquired two-dimensional surface image data; evaluating the signal attenuation characteristics of the OCT data to obtain an attenuation characteristic value; generating a high-frequency ultrasound trigger command when the attenuation characteristic value is lower than a preset medium turbidity threshold; receiving a corresponding high-frequency ultrasound echo signal in response to the high-frequency ultrasound trigger command; performing coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data; applying a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to the OCT data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold; and using a pre-trained non-rigid registration network to perform cross-dimensional spatial registration of the three-dimensional panoramic interferometric image data and the two-dimensional surface image data, and mapping the deep structure ultrasound image data to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image data processing method, characterized in that, The method is performed by an electronic device, and the method includes: Acquire pre-collected optical coherence tomography data and synchronously acquired two-dimensional surface image data; The optical coherence tomography (OCT) data is evaluated for signal attenuation characteristics to obtain attenuation characteristic values; when the attenuation characteristic values ​​are lower than a preset medium turbidity threshold, a high-frequency ultrasound trigger command is generated. In response to the high-frequency ultrasound trigger command, the corresponding high-frequency ultrasound echo signal is received; the high-frequency ultrasound echo signal is coherently superimposed and time-delayed focusing is performed on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data. The optical coherence tomography data is processed by applying a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold. Using a pre-trained non-rigid registration network, the three-dimensional panoramic interferometric image data and the two-dimensional surface image data are spatially registered across dimensions, and the deep structure ultrasound image data is mapped to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

2. The image data processing method according to claim 1, characterized in that, The process of applying a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching to the optical coherence tomography data generates three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold, including: Extract consecutive adjacent scan frames from the optical coherence tomography data, and calculate the phase error matrix and pixel correlation matrix between adjacent scan frames; Based on the pixel correlation matrix, local artifact pixel regions are identified, and pixel reconstruction is performed on the local artifact pixel regions to obtain the reconstructed scan frame; The overlapping edge features of the multi-view volume data blocks composed of the reconstructed scan frames are extracted, and the multi-view volume data blocks are globally seamlessly stitched together to generate the three-dimensional panoramic interferometric image data.

3. The image data processing method according to claim 1, characterized in that, The process of coherently superimposing and delaying focusing the high-frequency ultrasound echo signal to generate deep structural ultrasound image data includes: Identify the received raw radio frequency data belonging to different channels; The two-way flight time of each channel at different depths is calculated based on the preset medium sound velocity parameters to obtain the dynamic delay matrix; The original radio frequency data is phase-aligned using the dynamic delay matrix, and the aligned data is coherently superimposed to reconstruct the deep structure ultrasound image data.

4. The image data processing method according to claim 1, characterized in that, The method of using a pre-trained non-rigid registration network to perform cross-dimensional spatial registration of the three-dimensional panoramic interferometric image data and the two-dimensional surface image data includes: The three-dimensional panoramic interferometric image data is subjected to maximum intensity projection along the depth direction to generate a two-dimensional projection map with the same dimensional attributes as the two-dimensional surface image data. The two-dimensional projection image and the two-dimensional surface image data are stitched together in the channel dimension to obtain a channel stitched image; The channel mosaic is input into the non-rigid registration network, and a dense deformation field is output through the non-rigid registration network. The three-dimensional panoramic interferometric image data is subjected to three-dimensional voxel-level spatial deformation distortion using the dense deformation field, so that the three-dimensional panoramic interferometric image data and the two-dimensional surface image data are aligned in the spatial coordinate system.

5. The image data processing method according to claim 1, characterized in that, The process of constructing multimodal fused spatial volume data to obtain the target image includes: Image features of each modality are extracted from the multimodal fusion spatial volume data and input into a feature consistency verification network based on a contrastive learning framework to obtain the cross-modal feature consistency identifier output by the feature consistency verification network. Based on the cross-modal feature consistency identifier, a target rendering instruction is generated for the feature difference region in the multimodal fused spatial volume data; The target image is obtained based on the multimodal fused spatial volume data and the target rendering instructions.

6. The image data processing method according to claim 5, characterized in that, The step of extracting image features of each modality from the multimodal fused spatial volume data and inputting them into a feature consistency verification network based on a contrastive learning framework to obtain the cross-modal feature consistency identifier output by the feature consistency verification network includes: The registered interferometric mode feature vector and ultrasonic mode feature vector are extracted from the multimodal fused spatial volume data using a preset feature encoder. The interference mode feature vector and the ultrasonic mode feature vector are input into the feature consistency verification network to determine the cosine similarity between the interference mode feature vector and the ultrasonic mode feature vector in the projection space; If the cosine similarity is lower than a preset consistency threshold, a negative consistency identifier representing feature conflict is generated as the cross-modal feature consistency identifier.

7. The image data processing method according to claim 1, characterized in that, The process of evaluating the signal attenuation characteristics of the optical coherence tomography data to obtain attenuation characteristic values ​​includes: Calculate the signal-to-noise ratio attenuation gradient of the pixel layer below the preset pixel depth in the optical coherence tomography data; Extract the optical signal reflection intensity parameters of the shallow pixel layer; The attenuation characteristic value is determined based on the weighted calculation result of the signal-to-noise ratio attenuation gradient and the optical signal reflection intensity parameter.

8. An image data processing apparatus, characterized in that, include: The acquisition module is used to acquire pre-acquired optical coherence tomography data and synchronously acquired two-dimensional surface image data. The first processing module is used to evaluate the signal attenuation characteristics of the optical coherence tomography data and obtain attenuation characteristic values. When the attenuation characteristic value is lower than the preset medium turbidity threshold, a high-frequency ultrasonic trigger command is generated; The response module is used to respond to the high-frequency ultrasound trigger command, receive the corresponding high-frequency ultrasound echo signal, and perform coherent superposition and time-delay focusing processing on the high-frequency ultrasound echo signal to generate deep structure ultrasound image data. The second processing module is used to apply a motion artifact elimination algorithm based on inter-frame correlation feedback and frequency domain panoramic stitching processing to the optical coherence tomography data to generate three-dimensional panoramic interferometric image data with a field of view parameter greater than a preset wide-angle threshold. The third processing module is used to perform cross-dimensional spatial registration of the three-dimensional panoramic interferometric image data and the two-dimensional surface image data using a pre-trained non-rigid registration network, and to map the deep structure ultrasound image data to the registered spatial coordinate system to construct multimodal fusion spatial volume data to obtain the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the image data processing method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image data processing method as described in any one of claims 1 to 7.

Images