Three-dimensional image data processing method, data processing method, equipment and storage medium

Abstract

The invention discloses a three-dimensional image data processing method, a data processing method, a device and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining three-dimensional images of a plurality of visual angles in three-dimensional image data, extracting the local features of the three-dimensional images of the plurality of visual angles, and updating a memory tensor through the local features of the plurality of visual angles, and aggregating the updated memory tensor to obtain the image features of the three-dimensional image, and by dynamically updating the memory tensor by using the local features of different view angles, effectively capturing and integrating the spatial information among the three-dimensional images of different view angles, thereby fully enhancing the image feature expression ability of the three-dimensional images, and improving the image feature expression efficiency. According to the technical scheme, the image feature expression comprehensiveness and accuracy are improved, the technical problem that the image feature extraction accuracy and comprehensiveness are poor in the related technology is solved, the image feature expression ability of the three-dimensional image is enhanced, and the technical effect of improving the image feature expression comprehensiveness and accuracy is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a three-dimensional image data processing method, a data processing method, a device and a storage medium. Background Art

[0002] In today's digital age, many fields rely on the efficient processing and analysis of various types of data, and 3D image data processing technology plays an important role in this. From the medical field to industrial manufacturing, virtual reality, geological exploration and many other industries, the application of 3D image data is becoming more and more extensive, and the depth and breadth of the information it contains greatly affects the accuracy of decision-making and work efficiency.

[0003] For example, in the medical field, 3D imaging data contains a wealth of space and is crucial for early diagnosis of diseases, condition assessment, and treatment plan formulation. Accurate and comprehensive analysis of these 3D medical imaging data can help doctors more accurately identify lesions and determine the stage of disease development, thereby improving treatment effectiveness and patient survival rates. For another example, in the field of industrial manufacturing, 3D imaging data is used for product testing, quality control, and reverse engineering. By analyzing the 3D images of products, surface defects and internal structural problems can be detected to ensure that product quality meets standards. In reverse engineering, 3D imaging data can be used to quickly obtain a 3D model of a product, improving design and manufacturing efficiency.

[0004] Therefore, how to efficiently process and analyze three-dimensional image data and accurately extract valuable image features has become a key issue in many fields. Summary of the invention

[0005] The present invention provides a three-dimensional image data processing method, a data processing method, a device and a storage medium, so as to at least solve the problem of poor accuracy and comprehensiveness in extracting image features in the related art.

[0006] The present invention provides a three-dimensional image data processing method, comprising the following steps: obtaining three-dimensional images of multiple perspectives in three-dimensional image data; extracting local features of the three-dimensional images of multiple perspectives, and updating a memory tensor using the local features of the multiple perspectives, wherein the memory tensor is used to store and fuse the local features of the multiple perspectives; and aggregating the updated memory tensors to obtain image features of the three-dimensional image.

[0007] The present invention also provides a data processing method, comprising the following steps: obtaining the data to be processed and task requirements input by a user, wherein the data to be processed includes at least one of three-dimensional image data and text data; inputting the data to be processed and the task requirements into a data processing model, and the data processing model outputs analysis results corresponding to the task requirements, wherein the data processing model includes an image encoding module, a text encoding module and a feature alignment module, the image encoding module processes the three-dimensional image data based on the above-mentioned three-dimensional image data processing method to obtain image features of the three-dimensional image, the text encoding module processes the text data based on a text encoder to obtain text features, and the feature alignment module is used to align image features and text features, and determine the analysis results corresponding to the task requirements based on the results of the feature alignment.

[0008] The present invention also provides a three-dimensional image data processing device, including: a first acquisition module, used to acquire three-dimensional images of multiple perspectives in three-dimensional image data; an extraction module, used to extract local features of the three-dimensional images of multiple perspectives, and use the local features of the multiple perspectives to update a memory tensor, wherein the memory tensor is used to store and fuse the local features of multiple perspectives and enhance the image feature expression of the three-dimensional image; an aggregation module, used to aggregate the updated memory tensor to obtain the image features of the three-dimensional image.

[0009] The present invention also provides a data processing device, including: a second acquisition module, used to acquire the data to be processed and task requirements input by a user, wherein the data to be processed includes at least one of three-dimensional image data and text data; an input module, used to input the data to be processed and the task requirements into a data processing model, and the data processing model outputs analysis results corresponding to the task requirements, wherein the data processing model includes an image encoding module, a text encoding module and a feature alignment module, the image encoding module processes the three-dimensional image data based on the above-mentioned three-dimensional image data processing device to obtain image features of the three-dimensional image, the text encoding module processes the text data based on the text encoder to obtain text features, and the feature alignment module is used to align image features and text features, and determine the analysis results corresponding to the task requirements based on the results of the feature alignment.

[0010] The present invention also provides an electronic device, comprising: a memory for storing a computer program; a processor for implementing the steps of the above-mentioned three-dimensional image data processing method, or the steps of the above-mentioned data processing method when executing the computer program.

[0011] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the three-dimensional image data processing method or the steps of the above-mentioned data processing method are implemented.

[0012] The present invention also provides a computer program product, including a computer program, which implements the steps of the three-dimensional image data processing method or the steps of the above-mentioned data processing method when the computer program is executed by a processor.

[0013] Through the present invention, by extracting local features of three-dimensional images from multiple perspectives, using the local features of multiple perspectives to update the memory tensor, and aggregating the updated memory tensor to obtain the image features of the three-dimensional image, by dynamically updating the memory tensor using the local features of different perspectives, the spatial information of the three-dimensional images from different perspectives can be effectively captured and integrated, thereby fully enhancing the image feature expression capability of the three-dimensional image, and improving the comprehensiveness and accuracy of the image feature expression. Therefore, technical problems such as the poor accuracy and comprehensiveness of image feature extraction in related technologies can be solved, and the technical effect of enhancing the image feature expression capability of the three-dimensional image and improving the comprehensiveness and accuracy of the image feature expression can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flowchart of a three-dimensional image data processing method provided by an embodiment of the present invention; Figure 2 A flowchart of a data processing method provided by an embodiment of the present invention; Figure 3 A flowchart of a data processing method provided for a specific embodiment of the present invention; Figure 4 A schematic diagram of a three-dimensional image data processing device provided by an embodiment of the present invention; Figure 5 A schematic diagram of a data processing device provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] Before describing the technical solution of the present invention, some technologies related to the solution of the present invention are first introduced.

[0020] In the field of traditional computer vision, multi-view image analysis mainly studies the spatial and geometric relationships between images taken from different angles. These methods are often applied to tasks such as video understanding, 3D rendering, and image segmentation, and improve scene understanding and target classification by integrating geometric features such as depth, shape, or texture. However, most of these methods focus on the analysis of spatial changes and lack attention to temporal dynamics or physiological dimensions, which are particularly important in the field of medical imaging.

[0021] It should be noted that the solution of the present invention can process three-dimensional images, three-dimensional videos or text data in multiple fields, without specific limitation. The following technical solutions of the present invention are described by taking the processing of medical data (such as three-dimensional medical imaging data and medical text data) in the medical field as an example.

[0022] In the field of medical imaging, multi-view analysis not only focuses on spatial perspectives, but also needs to combine temporal dynamics and physiological and pathological changes of biological processes, such as the distribution of contrast agents in the body. These changes bring unique challenges because normal structures and pathological structures need to be distinguished at the same time. This is significantly different from general computer vision tasks. Medical imaging needs to extract clinically meaningful insights by analyzing cross-stage and cross-modal correlations. The complexity of this problem requires a customized approach to capture these characteristics.

[0023] Diagnostic reports of medical images provide rich clinical information in text form by integrating spatial, temporal, and pathological correlations in multi-stage images. Corresponding diagnostic reports to imaging stages can deeply explore domain-specific knowledge. Visual-language models have obvious advantages in combining image and text data, and can support tasks such as classification, retrieval, and generative interpretation. In multi-stage radiological image analysis, temporal factors such as the dynamic changes of contrast agents are crucial to clinical decision-making, and this information is usually contained in diagnostic reports. By combining images with text descriptions, such models can learn the correlations between stages and extract key information of multi-stage dynamics, thereby improving the interpretation of complex imaging patterns and generating clinically meaningful conclusions.

[0024] Medical image and text alignment technology combines visual data (such as X-ray, MRI (Magnetic Resonance Imaging, CT (Computed Tomography) images) with domain-specific textual knowledge (such as clinical records, diagnostic reports) to improve the understanding and diagnosis of medical conditions. Pre-trained visual-language models (such as BiomedCLIP) have shown significant potential in large-scale biomedical image and text datasets, supporting tasks such as image retrieval, classification, and explanation generation. In addition, knowledge distillation techniques (such as self-evolving visual transformers for chest X-ray diagnosis) incorporate medical expertise into visual models, enabling them to interpret visual patterns and clinical context simultaneously, and excel in highly specialized fields. Models such as LLaVA-Med (Large Language and Vision Assistant for Medicine) further expand this concept, being able to process clinical text and medical images simultaneously, and interpreting predictions based on medical knowledge, thereby improving the interpretability of the model. While addressing the problems of data limitations and noisy annotations, recent advances in multimodal representation learning have also enhanced cross-modal retrieval and diagnostic reasoning capabilities.

[0025] For example, in the related art, medical image analysis methods fail to fully utilize the rich temporal and spatial information inherent in contrast-enhanced images. Traditional multi-view methods mainly focus on geometric correlations, but often ignore the subtle differences in time and physiological changes, which limits their performance in clinical diagnosis and treatment.

[0026] The solution of the present invention can solve at least one of the following problems in the medical field: 1. Related technologies face challenges in extracting features from high-dimensional data, especially in the deep mining of multi-dimensional spatial information and the integration of cross-view image data. Especially in medical imaging, how to effectively fuse information from different anatomical planes (such as axial, sagittal, and coronal) and extract comprehensive features on this basis is still a difficult problem to be solved. Related technologies often fail to fully capture the deep relationship between high-dimensional features in this process, which affects the comprehensiveness of feature expression and the parsing ability of the final model.

[0027] 2. Solve several technical problems in current multimodal image analysis, especially the accuracy and consistency issues when processing heterogeneous data (i.e. text and image data) and high-dimensional information. Related technologies usually rely on a single loss function for cross-modal alignment in the joint training of multimodal data such as images and text. However, this method often fails to fully balance the contributions of each modality, especially when processing multi-view and cross-stage information in medical images. The lack of fine-tuning of different modal features leads to the failure to achieve optimal learning effects of the model.

[0028] An embodiment of the present invention provides a three-dimensional image data processing method. The method is described in detail in conjunction with the execution flow of the three-dimensional image data processing method. The three-dimensional image data processing method of this embodiment is described by taking three-dimensional medical image processing in the medical field as an example.

[0029] like Figure 1 As shown, the three-dimensional image data processing method includes the following steps: In step S101 , three-dimensional images of multiple viewing angles are acquired from three-dimensional image data.

[0030] Among them, multiple perspectives may be, for example, different anatomical planes in the medical field, such as axial, coronal, sagittal, and the like.

[0031] It is understandable that the embodiment of the present invention can obtain three-dimensional images of multiple viewing angles in the three-dimensional image data so as to perform feature processing thereon subsequently.

[0032] In step S102, local features of the three-dimensional images from multiple perspectives are extracted, and the memory tensor is updated using the local features from multiple perspectives, wherein the memory tensor is used to store and fuse the local features from multiple perspectives.

[0033] Among them, the three-dimensional images of multiple perspectives include three-dimensional images of multiple stages. For example, if the current perspective is the axial perspective, the axial perspective may also include multiple stages, such as the plain scan stage, the arterial phase stage, the perfusion phase, etc. The memory tensor is a shared, dynamically updateable data structure used to store and fuse information from all perspectives. It can also be regarded as a long-term memory representation of the input data by the model, which can transmit and integrate information at different time points (i.e., different stages) or between perspectives, so as to better capture and utilize the complementary information between different perspectives and avoid the deviation or information loss that may be caused by a single perspective.

[0034] It can be understood that the embodiments of the present invention can extract local features of three-dimensional images from multiple perspectives, and use the local features of multiple perspectives to update the memory tensor to fuse the information between multiple perspectives, avoid the deviation or information loss caused by using only a single perspective, and generate a more comprehensive and accurate image feature representation.

[0035] In addition, the embodiment of the present invention can extract local features of three-dimensional images from multiple perspectives through the Transformer network. Since local features have higher robustness than global features, they can more accurately locate and describe image details, and the data processing volume is also smaller. Therefore, the embodiment of the present invention only uses local features when updating the memory tensor, thereby reducing the data processing volume, improving data processing efficiency, and generating a more comprehensive and accurate image feature representation.

[0036] In an embodiment of the present invention, the memory tensor is updated using local features of multiple perspectives, including: obtaining local features of the three-dimensional image at the current stage under the current perspective; using the local features of the current stage under the current perspective to update the memory tensor stored at the previous stage under the current perspective, until the memory tensors at multiple stages under the current perspective are updated; after updating the memory tensor using the local features of multiple stages of the current perspective, using the local features of multiple stages of the next perspective to update the memory tensor stored at the current perspective, until the memory tensors of multiple perspectives are updated, to obtain the final updated memory tensor.

[0037] It can be understood that the embodiments of the present invention can obtain the local features of the three-dimensional image at the current stage under the current perspective, and use the local features of the current perspective at the current stage to update the memory tensor stored in the previous stage of the current perspective, until the memory tensors of multiple stages under the current perspective are updated, and then use the local features of multiple stages of the next perspective to update the memory tensor stored in the current perspective, until the local features of multiple perspectives are updated to the memory tensor, and then the final updated memory tensor is obtained to integrate the spatial information and temporal information of the three-dimensional images at multiple perspectives and multiple stages, so as to improve the utilization efficiency of spatial and temporal information in the three-dimensional image processing process, the accuracy of feature extraction and the comprehensiveness of image feature expression, wherein multiple stages represent data at multiple time points.

[0038] In addition, it should be noted that in the medical field, three-dimensional medical imaging data at multiple stages contains rich information on changes in physiological characteristics. Taking the medical imaging of the heart as an example, the heart is imaged at different stages, and physiological activities such as the heart's beating cycle, myocardial contraction and relaxation will be reflected in the image. By obtaining the local features of the images at different stages and updating the memory tensor, it is possible to record the changes in the physiological state of the heart at different times, such as the periodic changes in myocardial thickness, changes in cardiac chamber volume, etc., thereby expressing the physiological characteristics of the heart. In the diagnosis of cardiovascular diseases, this capture of changes in cardiac physiological characteristics over time helps doctors determine whether the heart function is normal and whether there are problems such as myocardial lesions. And three-dimensional medical images from multiple perspectives show the physiological structure of human organs and tissues from different angles.

[0039] In some medical imaging examinations, contrast agents and other means are used to observe physiological processes. For example, in angiography, the flow and distribution of contrast agents in blood vessels reflect the physiological characteristics of blood circulation. When processing image data, the dynamic change information of contrast agents in different stages of the image (such as the time when the contrast agent arrives at different vascular sites, concentration distribution, etc.) will be captured by local features, and the physiological characteristics of human blood circulation will be expressed through the updating and aggregation of memory tensors. This capture and expression of dynamic information of physiological processes is of great significance for the diagnosis of vascular diseases (such as vascular stenosis, blockage, etc.).

[0040] By integrating the local features of multi-view and multi-stage three-dimensional medical images into the memory tensor, the resulting image features can comprehensively reflect the physiological characteristics of the human body. The memory tensor integrates information from different viewpoints and stages, so that the generated image features not only contain the spatial structure information of organs and tissues, but also cover the information of changes in physiological processes over time. This comprehensive expression ability can present the physiological state of the human body more comprehensively and accurately, and can subsequently provide doctors with richer diagnostic evidence, which helps to improve the understanding of physiological characteristics and the accuracy of disease diagnosis.

[0041] Specifically, for example, the current perspective is the first perspective V1, and the first perspective includes three stages, the first stage Sv1, the second stage Sv2, and the third stage Sv3. The initial memory tensor M0 is set to empty, and the local features of Sv1 of V1 are updated to the memory tensor. Then, the memory tensor M1 stored in Sv1 is updated according to the local features of Sv2 of V1 to obtain the memory tensor M2 stored in Sv2. The memory tensor M2 stored in Sv2 is updated according to the local features of Sv3 to obtain the memory tensor M3 stored in Sv3. And so on, the local features of each stage of the second perspective V2 are updated to the memory tensor M3 finally stored in V1, and so on, to obtain the updated memory tensors of all stages of all perspectives.

[0042] In an embodiment of the present invention, the memory tensor stored in the previous stage of the current perspective is updated using the local features of the current stage of the current perspective, including: using a gating mechanism and the local features of the current stage to update the memory tensor stored in the previous stage, and using the updated memory tensor of the previous stage as the memory tensor stored in the current stage, wherein the gating mechanism includes an update gate and a reset gate, the update gate is used to control the fusion ratio between the local features of the current stage and the memory tensor stored in the previous stage, and the reset gate is used to control the reset level of the memory tensor stored in the previous stage in the current stage.

[0043] It is understandable that the embodiment of the present invention can utilize the gating mechanism and the local features of the current stage ( ) updates the memory tensor stored in the previous stage (also called previous memory ), and uses the memory tensor updated in the previous stage as the memory tensor stored in the current stage, where the gating mechanism includes the update gate ( ) and the reset gate ( ), the update gate is used to control the fusion ratio of the local features of the current stage and the memory tensor stored in the previous stage, and the reset gate is used to control the reset level of the memory tensor stored in the previous stage in the current stage, that is, it is used to control the role of the previous memory in the update of the current stage. The reset gate and the update gate jointly determine the degree of fusion of the local features of the current stage and the previous memory.

[0044] In an embodiment of the present invention, before using the gating mechanism and the local features of the current stage to update the memory tensor stored in the previous stage, it includes: calculating the value of the update gate and the value of the reset gate according to the local features of the current stage and the memory tensor stored in the previous stage; updating the memory tensor stored in the previous stage according to the value of the update gate, the value of the reset gate and the local features of the current stage, wherein the value of the update gate is the fusion ratio, and the value of the reset gate is the reset level.

[0045] It can be understood that the embodiments of the present invention can calculate the values ​​of the update gate and the reset gate based on the local features of the current stage and the memory tensor stored in the previous stage, and update the memory tensor stored in the previous stage based on the value of the update gate, the value of the reset gate and the local features of the current stage, wherein the value of the update gate is the fusion ratio and the value of the reset gate is the reset level to determine the degree of fusion of the local features of the current stage and the memory tensor stored in the previous stage.

[0046] In an embodiment of the present invention, the value of the update gate and the value of the reset gate are calculated according to the local features of the current stage and the memory tensor stored in the previous stage, including: performing a three-dimensional convolution operation on the local features of the current stage to obtain a first feature; performing a three-dimensional convolution operation on the memory tensor stored in the previous stage to obtain a second feature; combining the first feature and the second feature to obtain a first combined feature, inputting the first combined feature into a first target activation function, and the first target activation function outputs the value of the update gate and the value of the reset gate.

[0047] The first target activation function may be a Sigmoid activation function, which is not specifically limited.

[0048] It can be understood that the values ​​of the update gate and the reset gate of the embodiment of the present invention can be calculated through a 3D convolution operation. The three-dimensional convolution operation can be performed on the local features of the current stage and the memory tensor stored in the previous stage respectively to obtain the first feature and the second feature respectively. The first feature and the second feature are combined to obtain the first combined feature, and the first combined feature and the second combined feature are input into the first target activation function. The first target activation function outputs the value of the update gate and the value of the reset gate.

[0049] It should be noted that the values ​​of the update gate and the reset gate in the embodiment of the present invention are between 0 and 1.

[0050] Among them, the update gate ( ) and the reset gate ( ) is calculated using the following formula: ; ; In the above formula, represents the element-level Sigmoid activation function, represents element-wise multiplication (the operation is broadcast over the batch dimension), is the local feature of the current stage (also called the input feature of the current stage), is the memory tensor stored in the previous stage (also called previous memory), and is a three-dimensional (3D) convolution operation, For each batch of data, the validity mask, , , , is the batch size, are the spatial dimensions of local features, namely depth, height, and width, The update gate is used to control the fusion ratio between the input features of the current stage and the previous memory. The update gate value is calculated through a 3D convolution operation, which takes into account the input features of the current stage and the features in the previous memory, and combines the validity mask in each batch sample to ensure that only data of valid perspectives are processed.

[0051] The three-dimensional convolution (Conv3d) of the embodiment of the present invention is a convolution operation for processing 3D data, which is usually applied to images, videos and other data with three-dimensional structures. In convolutional neural networks (CNNs), 2D convolution is more widely used to process planar images, while 3D convolution can be extended to three dimensions of depth, width and height, so as to better process space, time and other three-dimensional information. Therefore, the medical data processing of the present invention adopts 3D convolution operation.

[0052] Among them, the specific execution steps of the 3D convolution operation include: Assume that there are the following input data (i.e., the local features of the current stage of the present invention or the memory tensor stored in the previous stage) and convolution kernel, the input tensor ,in, is the batch size (batchsize), is the number of input channels (e.g. the number of grayscale channels of medical images), are the depth, height and width of the input data respectively; Convolution Kernel ,in, Respectively represent the size of the convolution kernel in depth, height, and width, is the number of input channels, is the number of output channels.

[0053] In 3D convolution, the convolution kernel In the input tensor Slide up (perform sliding window operation) to generate output tensor (i.e., the first and second features output by the embodiment of the present invention), output tensor ,in, It is the output depth, height and width after the convolution operation. The output size is determined by setting the stride and padding.

[0054] For each position of the output tensor , the convolution operation can be expressed as: ; The above formula describes how the convolution kernel (filter) and the input tensor (input tensor) interact during the convolution process to generate the output tensor (output tensor).

[0055] : This is the output tensor An element in the batch In the output channel The corresponding depth ,high , the value of the width w' position. In other words, this is the result of the convolution operation; : This is the input tensor The convolution window (i.e., the convolution kernel) will slide on the input tensor, so , ,and These expressions represent the current position of the convolution kernel and the corresponding position of the input data; :This is the convolution kernel An element in represents the specific position of the convolution kernel in the depth, width, and height directions, and distinguishes the input channels and output channels .

[0056] After the above three-dimensional convolution operation, the three-dimensional features (including depth, width and height) of the three-dimensional medical image are specially processed. Specifically, each feature in the three-dimensional data often involves information in multiple dimensions. The aggregation and update of this information requires convolution operations to capture the correlation between different spatial positions.

[0057] In an embodiment of the present invention, a gating mechanism and a local feature of a current stage are used to update a memory tensor stored in a previous stage, including: resetting the memory tensor stored in the previous stage using a reset gate; performing a three-dimensional convolution operation on the reset memory tensor stored in the previous stage to obtain a third feature; combining the first feature and the third feature to obtain a second combined feature, inputting the second combined feature into a second target activation function, and the second target activation function outputs a candidate memory tensor; and fusing the candidate memory tensor and the memory tensor stored in the previous stage using an update gate.

[0058] The second target activation function is a nonlinear activation function, such as a hyperbolic tangent function.

[0059] It can be understood that the embodiments of the present invention can use the reset gate to reset the memory tensor stored in the previous stage, and perform a three-dimensional convolution operation on the reset memory tensor stored in the previous stage to obtain a third feature, combine the first feature and the third feature to obtain a second combined feature, input the second combined feature to the second target activation function, and the second target activation function outputs a candidate memory tensor. The candidate memory tensor and the memory tensor stored in the previous stage are fused using the update gate to obtain the updated memory tensor stored in the previous stage, that is, the memory tensor stored in the current stage.

[0060] Among them, the reset gate determines how to weight the current input and the previous memory, thereby adjusting the degree of memory retention and reset. When the value of the reset gate is close to 1, the previous memory will have a stronger influence on the current candidate memory; when it is close to 0, the previous memory has a weaker influence on the current update.

[0061] Specifically, the candidate memory tensor (also referred to as candidate memory) of the present invention is calculated by combining the local features (also referred to as input features) of the current stage with the memory tensor (also referred to as previous memory) stored in the previous stage after reset. When generating candidate memory, the previous memory needs to be reset first so that only part of the information required for the current input is retained. The reset operation is completed by multiplying with the reset gate, thereby adjusting how much previous memory information is retained.

[0062] Then, the candidate memory Generated through a 3D convolution operation, it not only considers the current input features, but also generates new memory representations by interacting with previous memories. Candidate memories are usually activated nonlinearly (such as the hyperbolic tangent function tanh) to enhance their expressive power in order to capture complex spatial and temporal features.

[0063] Candidate Memory Based on the current characteristics and the previous memory after reset The calculation formula is: ; in, For candidate memory, is a 3D convolution operation, is the current input feature, To reset the gate value, For previous memory, is the previous memory after reset, For each batch of data, the validity mask, Represents element-wise multiplication (the operation is broadcast over the batch dimension).

[0064] After generating the candidate memory, the previous memory is fused with the candidate memory, and the update gate controls the influence ratio of the input features of the current stage and the previous memory. Specifically, when When it approaches 1, the new candidate memory will have a greater impact on the final memory. When it approaches 0, the previous memory will be retained more. Through such a gating mechanism, the model can flexibly and dynamically adjust the memory between different time steps, so as to effectively capture the key information in multi-view images or multi-stage data processing.

[0065] The final memory update is calculated using the following formula: This formula uses the update gate To control the contribution ratio of the current feature and candidate memory to the final memory. When it approaches 1, more information comes from candidate memory ;when When it approaches 0, memory updating tends to keep the previous memory .

[0066] In addition, it should be noted that after the memory update, layer normalization (LayerNorm) is performed, which is applied to the hidden dimension , that is, the channel dimension, the formula is expressed as: ; Layer normalization helps stabilize the learning process and ensures the numerical stability of memory and the efficiency of subsequent model training.

[0067] Finally, the updated memory tensor will have the same shape as the input, i.e. , so as to be used for subsequent feature aggregation and alignment tasks.

[0068] In the implementation of the present invention, after extracting the local features of the three-dimensional images of multiple viewing angles, the method further includes: filtering out invalid viewing angles using a viewing angle mask; and resetting the local features corresponding to the invalid viewing angles to zero.

[0069] It can be understood that the embodiments of the present invention can use the perspective mask to filter out invalid perspectives and reset the local features corresponding to the invalid perspectives to zero. Compared with the simple superposition of perspective or spatial information processing in traditional methods, the present invention uses a dynamic mask to eliminate invalid perspectives, avoiding the interference of irrelevant data on the memory update process, thereby further improving the accuracy of image feature representation.

[0070] In summary, the embodiment of the present invention updates the memory tensor through a dynamic memory update mechanism, which not only takes into account the spatial relationship, but also dynamically adjusts the memory information of each perspective through three-dimensional local features, combines the current perspective with the three-dimensional features of the previous memory, and enhances the learning ability and stability of the subsequent model in multi-dimensional data. This mechanism can dynamically update the memory between different time steps and different perspectives to ensure the effective fusion of cross-perspective information, and introduces a three-dimensional gating mechanism in the memory update process. By dynamically adjusting the three-dimensional update gate and reset gate, in the process of integrating multi-perspective and multi-stage image features, it accurately screens important information and suppresses noise and invalid features.

[0071] The dynamic memory update mechanism is applicable to three-dimensional spatial data. It can be dynamically adjusted in three dimensions: depth, width, and height to enhance the fusion effect of image features. The specific update process is as follows: 1. At each stage of each perspective, the current input features (i.e., local features at the current stage) and the previous memory (memory tensor stored at the previous stage) are interacted to generate new candidate memories. This process is achieved through convolution operations, and the role of the convolution kernel is to extract the spatial and temporal features between the current input and the historical memory. Under the action of the convolution kernel, these features can span the three dimensions of depth, width, and height, thereby effectively capturing the associated information at each spatial position.

[0072] 2. The 3D convolution operation processes the input features and extracts spatial and temporal correlation features. The convolution kernel is designed to slide in three-dimensional space, thereby capturing the spatial relationship of the input data in three directions: depth, width, and height. Through this convolution operation, the model can identify local features in the image and aggregate these features for further memory updates. This operation not only models the spatial structure of the image, but also takes into account the relationship between different time steps, thus providing rich information for the dynamic memory network.

[0073] 3. When processing multi-view data, a validity mask is introduced to ensure that only valid view information will affect the memory update. This is especially important because in the process of multi-view image processing, some views may not be updated due to missing data or quality issues. Through the validity mask, valid view data is dynamically screened out to ensure that only meaningful input information is used for memory update. This is crucial to ensure the effective fusion of 3D medical imaging data and avoid invalid data interfering with the memory update process.

[0074] 4. The whole process combines 3D convolution, dynamic memory update and validity mask, fully considering the information of each dimension in the three-dimensional space, ensuring the effective integration of multi-view and multi-stage medical imaging data in memory update. This method can not only accurately capture the characteristics in space and time, but also ensure the screening and strengthening of important information in a complex data environment.

[0075] In step S103, the updated memory tensors are aggregated to obtain image features of the three-dimensional image.

[0076] It can be understood that the embodiments of the present invention can aggregate the updated memory tensors to obtain the image features of the three-dimensional image. Through memory updating and three-dimensional convolution operations, meaningful features in the three-dimensional image can be screened out from different perspectives and dimensions, thereby enhancing the image feature expression capability of the three-dimensional image.

[0077] In the implementation of the present invention, the updated memory tensors are aggregated to obtain image features of the three-dimensional image, including: performing global average pooling on the updated memory tensors; aggregating the memory tensors after global average pooling to obtain global features of the three-dimensional image; and projecting and normalizing the global features to obtain image features of the three-dimensional image.

[0078] It can be understood that the embodiments of the present invention can perform global average pooling on the updated memory tensor, aggregate the memory tensors after global average pooling to obtain the global features of the three-dimensional image, project and normalize the global features to obtain the image features of the three-dimensional image, so as to ensure the consistency and stability of the image features between different perspectives and eliminate unnecessary differences between different perspectives.

[0079] It should be noted that the three-dimensional image data processing method of the embodiment of the present invention can also directly process multi-stage three-dimensional images of the same perspective in the absence of multi-perspective data. The processing flow is the same, and the local features of each stage are used to update the memory tensor until all stages are updated to obtain the corresponding image feature representation.

[0080] The following describes the three-dimensional image data processing method of the embodiment of the present invention through a specific embodiment, which specifically includes a dynamic memory update mechanism (also known as storage update) and a 3D convolution operation, taking the application of this method in an image encoder as an example.

[0081] The storage update method of the image encoder flexibly extracts local features through the encoding and memory update mechanism of multi-view image data. The core of this method is to use image information from different perspectives, gradually update the memory through a gating mechanism, and finally obtain an image feature representation containing different perspectives. First, flatten the input data and process each perspective through a visual encoder to extract local feature tokens. Then, use the perspective mask to filter invalid perspectives, reorganize local features into a 3D grid, and update the memory tensor. Finally, through global 3D average pooling and projection normalization, an aggregated image feature representation of each batch of samples is generated. This method effectively captures key information in multi-view data and provides powerful feature representation capabilities for image analysis tasks.

[0082] Definition: Batch size: ; Number of viewing angles (maximum number of viewing angles): ; Number of input channels: ; Spatial dimensions of each 3D view: ; The spatial dimensions of the 3D patch grid: (denotes the number of patches in each dimension; the total number of tokens returned per view: ; Hidden layer dimensions (token embedding dimensions): .

[0083] enter: -Multi-view 3D data; - View mask (1 for valid views, 0 for fill / invalid views).

[0084] Output: Output of the visual encoder: , each view is encoded as a length The embedding dimension of each token is .

[0085] 1. The view is flattened and processed through the visual encoder.

[0086] 1.1 Flatten the input data.

[0087] To process each view independently, we first transform the input data Flattened so that all views are processed on the batch dimension. The reshaped tensor The shape is: ; in, Corresponding to the batch Through the flattening operation, the image data from all perspectives are combined into a form that is easier to process.

[0088] 1.2 Processing by visual encoder Next, the flattened data is fed into the visual encoder. The function of the visual encoder is to encode the data of each view into a series of tokens, each of which corresponds to a local or global feature of the image. After encoding, the output tensor is The shape is: ; here, is the number of tokens per view, is the embedding dimension of each token. Through the visual encoder, the spatial information of each view is converted into a set of embedded tokens of fixed dimension.

[0089] 1.3 Reshape into perspective format After encoding, the tensor Rescaled to The shape of to recover the perspective information: ; At this time, the tensor by Each batch contains token sequences from multiple perspectives.

[0090] 2. Extract the patch token.

[0091] The encoding result of each view contains a special [CLS] token for representing global information and several patch tokens for representing local features of the image.

[0092] Remove one [CLS] token, the remaining A token represents a local feature of the view, called a patch token. Extract these patch tokens and get: ; These patch tokens capture the characteristics of each local region in the image so as to understand the image at multiple scales.

[0093] 3. Applying View Mask To avoid processing image information from invalid perspectives, a perspective mask is introduced , which indicates which views are valid (value 1) and which are invalid (value 0). For each invalid view, its corresponding patch token is set to zero.

[0094] for The masking operation of the token is as follows: ,when When the invalid perspective Tokens are set to zero to prevent invalid data from interfering with subsequent feature extraction.

[0095] 4. Reorganize patch tokens into a 3D grid.

[0096] To capture the local spatial features of the image, the patch tokens are reorganized into a 3D grid. First, verify that the number of patch tokens matches the expected match: Then, the patch token is reshaped into a shape of 5D tensor and rearrange the dimensions so that the data can adapt to subsequent 3D operations: ; The shape of the rearranged patch token is , which can effectively capture the local spatial features of the image.

[0097] 5. Update memory.

[0098] In the storage update method, the memory tensor is updated step by step through the patch tokens of each view , which stores information from different views. During processing, we first extract the patch token from the current view and combine it with the view mask , ensuring that only data from valid perspectives is used for memory updates.

[0099] The memory update operation is as follows: ; in, is an operation that updates the memory through a gating mechanism, which progressively enhances the memory content based on the current patch token and the previous memory tensor.

[0100] 6. Global average pooling and final projection.

[0101] After all views are processed, the final memory tensor Perform global 3D average pooling to obtain an aggregate representation of each batch of samples : ; After global average pooling, Perform projection and normalization to obtain the final feature vector : .

[0102] The storage update method of the image encoder encodes 3D image data from different perspectives into a unified image feature representation through a series of sophisticated steps. This process utilizes the spatial information of multi-view images and combines it with a dynamic memory update mechanism to effectively extract global and local features. Specifically, the steps of the storage update method are as follows: First, the input multi-view 3D image data is flattened. Through the flattening operation, the image data of each view is processed independently, so that the characteristics of each view can be better captured. The flattened data is passed to the visual encoder, which converts the data of each view into a series of tokens, each of which represents the local or global features of the image in a high-dimensional embedding space.

[0103] Next, the global information token ([CLS] token) and the corresponding local information token (patch token) are extracted from the token sequence output by the visual encoder. In this way, the global and local features are processed separately, providing a basis for feature aggregation in subsequent steps. At the same time, the view mask is introduced to filter out invalid view angles, ensuring that only valid view angle information participates in the subsequent memory update process.

[0104] Then, the extracted patch tokens are reorganized into a 3D grid form. The main purpose of this step is to enable the network to effectively capture the spatial relationship between local regions in the image and enhance the modeling ability of spatial features by mapping local features into a spatial grid.

[0105] After that, the memory tensor is updated with the patch token of each view. The memory update process gradually enhances the feature expression of the image by combining the local features of the current view with the previous memory information. The memory update operation allows the network to share information between multiple views, thereby obtaining more comprehensive image features.

[0106] Finally, after processing all viewpoints, the final memory tensor is subjected to global 3D average pooling. This step aggregates the features of all viewpoints to obtain the global feature representation of each batch of samples. The aggregated features are then projected and normalized to obtain the final image representation. This process ensures the consistency and stability of image features across viewpoints while eliminating unnecessary differences between different viewpoints.

[0107] Through the above steps, the image encoder can flexibly update and aggregate data from different perspectives, capture the key information, and finally generate an accurate image feature representation. This method can not only process multi-view data, but also dynamically adjust the information in memory to adapt to different data and task requirements. Therefore, this method has broad application prospects in multi-view image processing, medical image analysis, and other tasks that require cross-view information integration.

[0108] The three-dimensional image data processing method proposed in the embodiment of the present invention can identify three-dimensional images of multiple perspectives in three-dimensional image data and extract local features of three-dimensional images of multiple perspectives, capture detail changes in three-dimensional images, and use the local features of multiple perspectives to update memory tensors, aggregate the updated memory tensors, and obtain image features of three-dimensional images. By dynamically updating the memory tensors using local features of different perspectives, information between different perspectives can be transmitted, and spatial information of three-dimensional image data of different perspectives can be effectively captured and integrated, thereby fully enhancing the image feature expression capability of three-dimensional images and improving the comprehensiveness and accuracy of image feature expression.

[0109] The 3D image data processing method of the above embodiment focuses on the processing of 3D image data, while the data processing method of the following embodiment focuses on the comprehensive processing of 3D image data and text data. The incomplete parts between the embodiments can be referenced to each other.

[0110] It should be noted that the data processing method described in the following embodiments takes the processing of three-dimensional medical image data and medical text data in the medical field as an example.

[0111] like Figure 2 As shown, the data processing method includes the following steps: In step S201, the data to be processed and the task requirements input by the user are obtained, wherein the data to be processed includes at least one of three-dimensional image data and text data.

[0112] Among them, the task requirements include at least one of a retrieval task, a generation task and a matching task. The retrieval task, for example, retrieves the corresponding medical text based on the three-dimensional medical image, or retrieves the corresponding three-dimensional medical image based on the medical text. The generation task, for example, generates a medical text describing the three-dimensional medical image data based on the three-dimensional medical image data. The matching task, for example, inputs multiple three-dimensional medical images and multiple medical texts and matches the correspondence between the two.

[0113] In step S202, the data to be processed and the task requirements are input into the data processing model, and the medical data processing model outputs the analysis results corresponding to the task requirements, wherein the data processing model includes an image encoding module, a text encoding module and a feature alignment module. The image encoding module processes the three-dimensional image data based on the above-mentioned three-dimensional image data processing method to obtain the image features of the three-dimensional image. The text encoding module processes the text data based on the text encoder to obtain text features. The feature alignment module is used to align the image features and the text features, and determine the analysis results corresponding to the task requirements according to the results of the feature alignment.

[0114] It can be understood that the embodiment of the present invention can input the data to be processed and the task requirements into the data processing model, and the data processing model outputs the analysis results corresponding to the task requirements. The data processing model includes three modules, namely, an image encoding module, a text encoding module and a feature alignment module, wherein: The image encoding module processes the 3D image data based on the above 3D image data processing method to obtain image features of the 3D image. The specific application entity of the image encoding module may be the image encoder in the above embodiment, that is, the image encoder is used to process the 3D image. The text encoding module processes the text data based on the text encoder to obtain text features. The text encoder can be BERT (Bidirectional Encoder Representations from Transformers). The feature alignment module is used to align image features and text features, and determine the analysis results corresponding to the task requirements based on the results of the feature alignment.

[0115] Through the above-mentioned data processing method, according to the user's task requirements and input data, through the collaborative work of the image encoding module, text encoding module and feature alignment module in the medical data processing model, effective fusion of cross-modal data can be achieved. In addition, since the image encoding module fuses three-dimensional image features from different perspectives, the integration and update mechanism of image features is optimized, and the alignment efficiency and accuracy between cross-modal data are further improved, that is, the accuracy and consistency of the fusion between multimodal data, thereby improving the performance of the data processing model under various tasks.

[0116] For example, if the user's task requirement is a retrieval task and the input data is three-dimensional medical image data, the image encoding module of the medical data processing model can be used to process the input three-dimensional medical image data to obtain corresponding image features. The text encoding module is used to process the medical text data stored in a preset medical database (which stores three-dimensional medical image data and medical text data) to obtain multiple text features. The feature alignment module is used to align the image features output by the image encoding module and the multiple text features output by the text encoding module to obtain a text feature with the highest feature alignment and use it as the analysis result of this task requirement, that is, to retrieve the most matching medical text data based on the input three-dimensional medical image data.

[0117] In an embodiment of the present invention, before inputting the data to be processed and the task requirements into the data processing model, it also includes: obtaining a training data set for the data processing model, wherein the training data set includes medical text data, three-dimensional image data, and a matching relationship between the text data and the three-dimensional image data; and training the data processing model using the training data set and the target loss function.

[0118] It is understandable that the present invention can utilize training data sets and target loss functions to train data processing models to improve the processing capabilities of the models under various tasks.

[0119] In an embodiment of the present invention, a data processing model is trained using a training data set and a target loss function, including: dividing the training data set into multiple batches of training data; iteratively training the data processing model using the multiple batches of training data; during the iterative training process, calculating the loss value of the data processing model using the target loss function, and updating the model parameters of the data processing model according to the loss value.

[0120] Among them, the objective loss function is: ; in, , , is a hyperparameter used to control the relative importance of each loss function, which can also be understood as the weight corresponding to each loss function. is the text encoding loss function, is the retrieval loss function, Generate loss function for text, is the target loss function.

[0121] It can be understood that the implementation of the present invention can divide the training data set into multiple batches of training data, and use multiple batches of training data to iteratively train the data processing model. During the iterative training process, the target loss function is used to calculate the loss value of the data processing model, and the model parameters of the data processing model are updated according to the loss value, thereby improving the processing capability and effect of subsequent models.

[0122] In the implementation of the present invention, the model parameters of the data processing model are updated according to the loss value, including: calculating the gradient of the model parameters in the data processing model according to the loss value; and updating the model parameters according to the gradient of the model parameters and the learning rate of the model parameters.

[0123] It can be understood that the embodiment of the present invention can calculate the gradient of the model parameters in the data processing model according to the loss value, and update the model parameters according to the gradient of the model parameters and the learning rate of the model parameters.

[0124] It should be noted that the size of the training batch plays a role in balancing memory usage, computing speed, and model performance during the training process. The specific model parameter gradients and model parameter learning rates are calculated as follows.

[0125] Training batch: The batch size directly affects the accuracy and memory consumption of gradient calculation. During training, the gradient calculation formula for each batch is: ; in, is the batch size, is the loss function for each sample, are the parameters of the model, and Increasing the batch size can reduce gradient noise and make the model more stable, but it will also increase video memory consumption.

[0126] Gradients of model parameters: When the video memory limit is strict, a gradient accumulation strategy is used to simulate large-batch training. The gradients of multiple small batches are accumulated before a parameter update. Assume that the cumulative number of steps for each update is , then the parameter gradient of each update is: ; Gradient accumulation allows us to simulate the effect of large batch training with smaller batch sizes, but at the expense of increased computation time.

[0127] Learning rate: The choice of learning rate has an important impact on the convergence speed and stability of training. Too high a learning rate may lead to unstable training, while too low a learning rate may cause the model to converge too slowly. Through a suitable learning rate scheduling strategy, the training efficiency can be effectively improved.

[0128] Learning rate decay: Using the cosine decay strategy to adjust the learning rate can gradually reduce the learning rate during the training process to avoid fluctuations caused by excessive learning rate in the later stage. The cosine decay formula is as follows: ; in, is the initial learning rate, is the current training step, is the total number of training steps. Through cosine decay, the learning rate is larger in the early stage and gradually decreases in the later stage, thus effectively avoiding the instability of training caused by excessively high learning rate.

[0129] Learning rate warm-up: To prevent instability caused by excessive gradients at the beginning of training, learning rate warm-up gradually increases the learning rate to stabilize the early training process. Set the warm-up ratio , which means that in the initial training stage, the learning rate gradually increases from 0 to the value of the initial learning rate. The warm-up process can be expressed as follows: ; in, is the number of warm-up steps, is the initial learning rate, is the current step.

[0130] In addition, it should be noted that the embodiment of the present invention also balances training time, memory usage and model performance through optimizer configuration during the training process.

[0131] The optimizer is responsible for adjusting the parameters of the model during training to reduce the loss function. In joint training, the AdamW (Adaptive Moment Estimation with Weight Decay Optimizer) optimizer is widely used because it can effectively handle sparse gradient problems and combines weight decay to prevent overfitting. The update rules of the AdamW optimizer are as follows: ; in, are the current model parameters, It was the last moment The model parameters, is the learning rate, is the first-order moment estimate of the gradient, is the second moment estimate of the gradient, is a minimum value used to avoid division by zero errors. is the weight decay coefficient. Weight decay reduces the risk of overfitting by applying L2 regularization to model parameters. The specific form is: ; in, is the L2 norm of the parameter. Reasonable optimizer configuration and weight decay coefficient selection can accelerate model convergence and improve its generalization ability on test data. In the case of less training data or more noise, stronger weight decay can effectively prevent overfitting.

[0132] Optimizer selection and learning rate: The AdamW optimizer combines adaptive learning rate and L2 regularization. By automatically adjusting the learning rate and introducing weight decay during training, it can accelerate convergence and improve the final performance of the model.

[0133] In an embodiment of the present invention, before using the target loss function to calculate the loss value of the data processing model, it also includes: obtaining the true value of the text data in the training data set and the output value of the data processing model, and constructing a text encoding loss function based on the difference between the output value and the true value; obtaining the true value of the matching relationship between the text data and the three-dimensional image data in the training data set and the output matching relationship of the data processing model, and constructing a retrieval loss function based on the difference between the output matching relationship and the true value of the matching relationship; obtaining the text true value of the text data corresponding to the three-dimensional image data in the training data set and the output text of the data processing model, and constructing a text generation loss function based on the difference between the output text and the text true value; constructing a target loss function based on the text encoding loss function, the retrieval loss function, the text generation loss function and their corresponding hyperparameters.

[0134] Since related technologies usually rely on a single loss function for cross-modal alignment in the joint training process of multimodal data such as images and texts, however, this method often cannot fully balance the contributions of each modality, especially when processing multi-view and cross-stage information in medical images, there is a lack of fine-tuning of different modal features, resulting in the failure to achieve optimal learning effect of the model. Therefore, an embodiment of the present invention can jointly train medical text data and three-dimensional medical image data, use different loss functions to promote the alignment of image features and text features, improve the model's processing capabilities for multimodal data, and improve the accuracy and consistency of multimodal data fusion.

[0135] Text encoding loss, retrieval loss, and text generation loss are key loss functions in multimodal learning. Text encoding loss helps the training model understand and encode text information, retrieval loss promotes mutual retrieval between images and text, and text generation loss optimizes the ability of images to generate text. Through the joint optimization of these loss functions, effective joint training of medical text and multimodal medical imaging data can be achieved, thereby improving the performance of the model in visual understanding and language generation tasks. The following are the three main loss functions: text encoding loss, retrieval loss, and text generation loss, as well as their mathematical expressions.

[0136] 1. Text Encoding Loss.

[0137] The text encoding loss aims to optimize the learning of text encoders (such as BERT) so that they can accurately understand text information and effectively align it with image features. In joint training, the Masked Language Modeling (MLM) loss function is used to optimize the text encoder, especially to predict masked words. The specific loss function is: ; in, Indicates the word that is covered. is an unmasked word, is the image feature, is the probability model for predicting masked words, is the training dataset, It is the training data set The samples obtained from Seek expectations.

[0138] 2. Retrieval Loss

[0139] Retrieval loss is used to train the model to achieve mutual retrieval between images and text in multimodal data. In this case, the goal is to optimize the model so that it can retrieve the correct text based on the image content, and vice versa. To this end, contrastive loss can be used to optimize the image-text similarity calculation. For a pair of image and text ( ), the retrieval loss is calculated as: ; in, Representing images and text The similarity between (usually using cosine similarity), is the temperature parameter, is the training dataset, is a negative sample, It is the training data set The samples obtained from Seek expectations.

[0140] 3. Text Generation Loss.

[0141] The text generation loss is mainly used to optimize the model to generate accurate text descriptions given an image. In this case, the goal is to minimize the difference between the generated text and the real text. You can use the autoregressive language model loss (Language Modeling Loss) for training. The specific loss function is: ; in, The generated text words, is the word generated previously, is the image feature, It is a probability model that generates the current text word based on the image and the previous text. is the training dataset, It is the training data set The samples obtained from Seek expectations.

[0142] Joint training objective: During the joint training process, the above three loss functions can be weighted and combined into the final target loss function: ; in, is a hyperparameter that controls the relative importance of each loss function. By optimizing this total loss function, the model is able to simultaneously optimize the encoding of images and text, making it perform well on both retrieval and generation tasks.

[0143] According to the data processing method proposed in the embodiment of the present invention, the user's input data and task requirements can be obtained, and the effective fusion of cross-modal data is achieved through the collaborative work of the image encoding module, the text encoding module and the feature alignment module in the data processing model. In addition, since the image encoding module fuses three-dimensional image features from different perspectives, the integration and update mechanism of image features is optimized, and the alignment efficiency and accuracy between cross-modal data are further improved, thereby improving the performance and effect of the data processing model under various tasks.

[0144] In summary, the data processing method of the embodiment of the present invention mainly includes two aspects: one is the processing of multi-view 3D image data, and the other is to optimize the alignment of image and text features through joint training to improve the performance and effect of the data processing model in various tasks, specifically including: 1. Extract image features through 3D convolution operations, especially to perform detailed modeling of spatial information in three dimensions: depth, width, and height. Compared with traditional methods that mainly rely on two-dimensional convolution or single-view information, the method of the present invention enhances the model's ability to capture spatial information by fusing three-dimensional information from multiple perspectives, so that fine-grained features obtained from different spatial positions can be effectively integrated. This method can more accurately represent the spatial relationship in the image, especially for three-dimensional medical imaging data, this method can deeply mine three-dimensional structural information.

[0145] 2. In processing 3D image data, a dynamic memory update mechanism is used to integrate the feature information of each perspective, fully capture the detail features from different perspectives, different spatial dimensions and different stages, further enhance the model's expressiveness in multiple perspectives and dimensions, and ensure the effective fusion of cross-perspective information. Specifically: A 3D gating mechanism is introduced to accurately screen important information and suppress noise and invalid features in the process of integrating multi-view and multi-stage medical image features by dynamically adjusting the 3D update gate and reset gate. This mechanism is particularly suitable for 3D spatial data and can dynamically adjust the depth, width and height dimensions to enhance the fusion effect of image features.

[0146] 3. Apply visual masks to ensure that only valid view information is involved in memory updates during 3D image processing and model training. Compared with the simple superposition of view or spatial information processing in traditional methods, the present invention uses dynamic masks to eliminate invalid view, avoids the interference of irrelevant data on the memory update process, and realizes efficient fusion of multi-view image features. This mask not only operates in the spatial dimension, but also considers the impact of temporal dynamic information (i.e., multi-stage information) on memory updates, and can subsequently improve the training efficiency and effect of the model.

[0147] 4. An efficient multimodal feature aggregation framework is proposed to effectively integrate 3D image features and text information. The alignment of image features and text features in the shared space is optimized through a joint loss function, which especially strengthens the fusion of information from different perspectives and stages in 3D space.

[0148] 5. By optimizing the joint loss function, combining text encoding loss, retrieval loss and text generation loss, and balancing the optimization goals of different tasks in three-dimensional spatial data, the performance of the model in multi-view medical imaging and text tasks is enhanced.

[0149] 6. Through the three-dimensional dynamic memory mechanism and feature fusion strategy, the present invention solves the problem of information fusion between different perspectives and stages, ensures that image and text data can be accurately aligned in three-dimensional space, improves the efficiency and accuracy of multimodal training, and shows stronger performance especially when processing high-dimensional medical imaging data.

[0150] The following describes a method for jointly training multimodal medical images and medical texts in an embodiment of the present invention through a specific embodiment, that is, a method for training a data processing model in the medical field. Figure 3 As shown, for example, multiple three-dimensional medical images are input, including the axial plain scan stage, axial arterial stage and axial perfusion stage of the skull, and medical text is input. The three-dimensional features of the three-dimensional medical images at different stages and the text features of the medical text are aligned and fused, and the medical data processing model is trained and optimized through a joint loss function.

[0151] The data processing method for the medical field of this embodiment enables researchers to more comprehensively and accurately integrate complex medical imaging data and clinical text information, thereby revealing deep pathological and physiological mechanisms, promoting the innovative development of medical research and clinical practice, and researchers can more deeply analyze the key features in complex imaging patterns, such as combining multi-stage imaging to analyze the dynamic changes of contrast agents, locate the progression stage of lesions, and evaluate the treatment effect. This capability provides new technical support for the diagnosis and treatment of complex diseases (such as cancer, cardiovascular disease or neurodegenerative diseases).

[0152] In addition, through multimodal fusion, the present invention can significantly improve the efficiency and accuracy of diagnostic report generation, reduce the time cost of manual interpretation of images and writing reports, and enable doctors to focus more on personalized treatment of patients. And through efficient multi-view image encoders and text alignment models, researchers can explore the temporal dynamic characteristics of disease development and construct disease-related imaging biomarkers and text description networks. For example, by analyzing the association between multi-stage images and patient medical history, early diagnostic clues for specific diseases can be revealed, providing a new research direction for precision medicine.

[0153] In the medical system, the clinical value of this invention is particularly prominent. Doctors can use the image-text alignment information generated by the medical data processing model to quickly identify the key features of the patient's disease progression and formulate a comprehensive treatment plan based on multimodal information, thereby improving the accuracy of treatment and reducing misdiagnosis and delays. In an environment with limited medical resources, it can also help optimize the diagnostic process and make medical services more efficient and inclusive.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0155] The embodiment of the present invention further provides a three-dimensional image data processing device.

[0156] like Figure 4 As shown, the three-dimensional image data processing device 10 includes: a first acquisition module 101 , an extraction module 102 and an aggregation module 103 .

[0157] Among them, the first acquisition module 101 is used to obtain three-dimensional images of multiple perspectives in three-dimensional image data; the extraction module 102 is used to extract local features of the three-dimensional images of multiple perspectives, and use the local features of multiple perspectives to update the memory tensor, wherein the memory tensor is used to store and fuse the local features of multiple perspectives; the aggregation module 103 is used to aggregate the updated memory tensor to obtain the image features of the three-dimensional image.

[0158] In an embodiment of the present invention, the three-dimensional images of multiple perspectives include three-dimensional images of multiple stages, and the extraction module 102 is further used to: obtain local features of the three-dimensional images at the current stage at the current perspective; use the local features at the current stage at the current perspective to update the memory tensor stored at the previous stage at the current perspective until the memory tensors at the multiple stages at the current perspective are updated; after updating the memory tensor using the local features of the multiple stages at the current perspective, use the local features of the multiple stages at the next perspective to update the memory tensor stored at the current perspective until the memory tensors of the multiple perspectives are updated to obtain the final updated memory tensor.

[0159] In an embodiment of the present invention, the extraction module 102 is further used to: use the gating mechanism and the local features of the current stage to update the memory tensor stored in the previous stage, and use the updated memory tensor in the previous stage as the memory tensor stored in the current stage, wherein the gating mechanism includes an update gate and a reset gate, the update gate is used to control the fusion ratio between the local features of the current stage and the memory tensor stored in the previous stage, and the reset gate is used to control the reset level of the memory tensor stored in the previous stage in the current stage.

[0160] In an embodiment of the present invention, the three-dimensional image data processing device 10 of the embodiment of the present invention also includes: a calculation module, which is used to calculate the value of the update gate and the value of the reset gate according to the local characteristics of the current stage and the memory tensor stored in the previous stage before using the gating mechanism and the local characteristics of the current stage to update the memory tensor stored in the previous stage; according to the value of the update gate, the value of the reset gate and the local characteristics of the current stage, the memory tensor stored in the previous stage is updated, wherein the value of the update gate is the fusion ratio, and the value of the reset gate is the reset level.

[0161] In an embodiment of the present invention, the computing module is further used to: perform a three-dimensional convolution operation on the memory tensor stored in the previous stage to obtain a second feature; combine the first feature and the second feature to obtain a first combined feature, input the first combined feature into a first target activation function, and the first target activation function outputs an update gate value and a reset gate value.

[0162] In an embodiment of the present invention, the extraction module 102 is further used to: reset the memory tensor stored in the previous stage using a reset gate; perform a three-dimensional convolution operation on the reset memory tensor stored in the previous stage to obtain a third feature; combine the first feature and the third feature to obtain a second combined feature, input the second combined feature into a second target activation function, and the second target activation function outputs a candidate memory tensor; and fuse the candidate memory tensor and the memory tensor stored in the previous stage using an update gate.

[0163] In the embodiment of the present invention, the 3D image data processing device 10 of the embodiment of the present invention further includes: a screening module.

[0164] The screening module is used to extract local features of three-dimensional images of multiple perspectives, and then use the perspective mask to screen out invalid perspectives; and reset the local features corresponding to the invalid perspectives to zero.

[0165] In an embodiment of the present invention, the aggregation module 103 is further used to: perform global average pooling on the updated memory tensor; aggregate the memory tensors after global average pooling to obtain global features of the three-dimensional image; project and normalize the global features to obtain image features of the three-dimensional image.

[0166] It should be noted that, for the description of the features in the embodiment corresponding to the three-dimensional image data processing device, reference may be made to the relevant description of the embodiment corresponding to the three-dimensional image data processing method, which will not be described in detail here.

[0167] An embodiment of the present invention further provides a data processing device.

[0168] like Figure 5 As shown, the data processing device 20 includes: a second acquisition module 201 and an input module 202 .

[0169] Among them, the second acquisition module 201 is used to obtain the data to be processed and the task requirements input by the user, wherein the data to be processed includes at least one of three-dimensional image data and text data; the input module 202 is used to input the data to be processed and the task requirements into the data processing model, and the data processing model outputs the analysis results corresponding to the task requirements, wherein the data processing model includes an image encoding module, a text encoding module and a feature alignment module, the image encoding module processes the three-dimensional image data based on the above-mentioned three-dimensional image data processing device 10 to obtain the image features of the three-dimensional image, the text encoding module processes the text data based on the text encoder to obtain text features, and the feature alignment module is used to align the image features and the text features, and determine the analysis results corresponding to the task requirements according to the results of the feature alignment.

[0170] In the embodiment of the present invention, the data processing device 20 of the embodiment of the present invention further includes: a training module.

[0171] Among them, the training module is used to obtain a training data set of the data processing model before inputting the data to be processed and the task requirements into the data processing model, wherein the training data set includes text data, three-dimensional image data, and the matching relationship between the text data and the three-dimensional image data; and the data processing model is trained using the training data set and the target loss function.

[0172] In an embodiment of the present invention, the training module is further used to: divide the training data set into multiple batches of training data; iteratively train the data processing model using multiple batches of training data; during the iterative training process, calculate the loss value of the data processing model using the target loss function, and update the model parameters of the data processing model according to the loss value.

[0173] In an embodiment of the present invention, the training module is further used to: calculate the gradient of the model parameters in the data processing model according to the loss value; and update the model parameters according to the gradient of the model parameters and the learning rate of the model parameters.

[0174] In this embodiment of the present invention, the target loss function is: ; in, , , is a hyperparameter used to control the relative importance of each loss function. is the text encoding loss function, is the retrieval loss function, Generate loss function for text, is the target loss function.

[0175] It should be noted that, for the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, which will not be described in detail here.

[0176] An embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a memory 601 and a processor 602, the memory 601 stores a computer program, and the processor 602 is configured to run the computer program to execute the steps in any of the above-mentioned three-dimensional image data processing method embodiments, or the steps in any of the above-mentioned data processing method embodiments.

[0177] An embodiment of the present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps in any one of the above-mentioned three-dimensional image data processing method embodiments, or the steps in any one of the above-mentioned data processing method embodiments when run.

[0178] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0179] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any one of the above three-dimensional image data processing method embodiments, or the steps in any one of the above data processing method embodiments.

[0180] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0181] The above is a detailed introduction to a medical data processing method provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A three-dimensional image data processing method, characterized in that: The following steps are involved: Acquire three-dimensional images of multiple viewing angles in the three-dimensional image data; Extracting local features of the three-dimensional images from the multiple perspectives, and updating a memory tensor using the local features from the multiple perspectives, wherein the memory tensor is used to store and fuse the local features from the multiple perspectives; The updated memory tensors are aggregated to obtain image features of the three-dimensional image.

2. The three-dimensional image data processing method according to claim 1, characterized in that: The three-dimensional images of the multiple perspectives include three-dimensional images of multiple stages, and the updating of the memory tensor using the local features of the multiple perspectives includes: Obtaining local features of the three-dimensional image at the current stage under the current viewing angle; Using the local features at the current stage under the current perspective, the memory tensor stored at the previous stage under the current perspective is updated until the memory tensors at multiple stages under the current perspective are updated; After the memory tensor is updated using the local features of multiple stages of the current perspective, the memory tensor stored in the current perspective is updated using the local features of multiple stages of the next perspective until the memory tensors of the multiple perspectives are updated, thereby obtaining a final updated memory tensor.

3. The three-dimensional image data processing method according to claim 2, characterized in that: The updating of the memory tensor stored in the previous stage of the current perspective by using the local features of the current stage of the current perspective includes: The memory tensor stored in the previous stage is updated using the gating mechanism and the local features of the current stage, and the updated memory tensor in the previous stage is used as the memory tensor stored in the current stage, wherein: The gating mechanism includes an update gate and a reset gate. The update gate is used to control the fusion ratio between the local features of the current stage and the memory tensor stored in the previous stage. The reset gate is used to control the reset level of the memory tensor stored in the previous stage in the current stage.

4. The three-dimensional image data processing method according to claim 3, characterized in that: Before updating the memory tensor stored in the previous stage using the gating mechanism and the local features of the current stage, it includes: Calculating the value of the update gate and the value of the reset gate according to the local features of the current stage and the memory tensor stored in the previous stage; The memory tensor stored in the previous stage is updated according to the value of the update gate, the value of the reset gate and the local features of the current stage, wherein the value of the update gate is the fusion ratio, and the value of the reset gate is the reset level.

5. The three-dimensional image data processing method according to claim 4, characterized in that: The calculating the value of the update gate and the value of the reset gate according to the local features of the current stage and the memory tensor stored in the previous stage includes: Performing a three-dimensional convolution operation on the local features of the current stage to obtain a first feature; Performing a three-dimensional convolution operation on the memory tensor stored in the previous stage to obtain a second feature; The first feature and the second feature are combined to obtain a first combined feature, and the first combined feature is input into a first target activation function, and the first target activation function outputs a value of the update gate and a value of the reset gate.

6. The three-dimensional image data processing method according to claim 5, characterized in that: The updating of the memory tensor stored in the previous stage by using the gating mechanism and the local features of the current stage includes: Resetting the memory tensor stored in the previous stage using the reset gate; Perform a three-dimensional convolution operation on the reset memory tensor stored in the previous stage to obtain the third feature; Combining the first feature and the third feature to obtain a second combined feature, inputting the second combined feature into a second target activation function, and the second target activation function outputting a candidate memory tensor; The candidate memory tensor and the memory tensor stored in the previous stage are fused using the update gate.

7. The three-dimensional image data processing method according to claim 1, characterized in that: After extracting the local features of the three-dimensional images from multiple perspectives, it also includes: Use the view mask to filter out invalid view angles; The local features corresponding to the invalid viewing angle are reset to zero.

8. The three-dimensional image data processing method according to claim 1, characterized in that: The aggregating the updated memory tensor to obtain the image features of the three-dimensional image includes: Performing global average pooling on the updated memory tensor; Aggregating the memory tensors after global average pooling to obtain the global features of the three-dimensional image; The global features are projected and normalized to obtain image features of the three-dimensional image.

9. A data processing method, characterized in that: The following steps are involved: Acquire the data to be processed and the task requirements input by the user, wherein the data to be processed includes at least one of three-dimensional image data and text data; The data to be processed and the task requirements are input into a data processing model, and the data processing model outputs an analysis result corresponding to the task requirements, wherein the data processing model includes an image encoding module, a text encoding module and a feature alignment module, the image encoding module processes the three-dimensional image data based on the three-dimensional image data processing method according to any one of claims 1 to 8 to obtain image features of the three-dimensional image, the text encoding module processes the text data based on a text encoder to obtain text features, and the feature alignment module is used to perform feature alignment on the image features and the text features, and determine the analysis result corresponding to the task requirements according to the result of the feature alignment.

10. The data processing method according to claim 9, characterized in that: Before inputting the data to be processed and the task requirements into the three-dimensional data processing model, the method further includes: Acquire a training data set of the three-dimensional data processing model, wherein the training data set includes text data, three-dimensional image data, and a matching relationship between the text data and the three-dimensional image data; The data processing model is trained using the training data set and the target loss function.

11. The data processing method according to claim 10, characterized in that: The step of training the data processing model using the training data set and the target loss function includes: Dividing the training data set into multiple batches of training data; Iteratively training the data processing model using the multiple batches of training data; During the iterative training process, the target loss function is used to calculate the loss value of the data processing model, and the model parameters of the data processing model are updated according to the loss value.

12. The data processing method according to claim 11, characterized in that: The updating of the model parameters of the data processing model according to the loss value comprises: Calculating the gradient of the model parameters in the data processing model according to the loss value; The model parameters are updated according to the gradient of the model parameters and the learning rate of the model parameters.

13. The data processing method according to any one of claims 10 to 12, characterized in that: The objective loss function is: ； in, , , is a hyperparameter, is the text encoding loss function, is the retrieval loss function, Generate loss function for text, is the target loss function.

14. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the three-dimensional image data processing method according to any one of claims 1 to 8, or the steps of the data processing method according to any one of claims 9 to 13 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the three-dimensional image data processing method according to any one of claims 1 to 8, or the steps of the data processing method according to any one of claims 9 to 13.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the three-dimensional image data processing method according to any one of claims 1 to 8 or the steps of the data processing method according to any one of claims 9 to 13 are implemented.

Images