Image processing model training method and device, electronic equipment and storage medium

Abstract

The application provides a training method and device of an image processing model, an electronic device and a storage medium. The method comprises: acquiring a plurality of multi-modal images as training samples, the types of the multi-modal images comprising full-modal images and missing-modal images; based on each multi-modal image, calling an initialized image processing model to perform a first training task of reconstructing a full-modal image, in the process of performing the first training task, the image processing model outputs a first full-modal reconstructed image; performing image completion processing on each first full-modal reconstructed image based on the full-modal image to obtain a full-modal template image; determining a consistency loss between the multi-modal image pair and the full-modal template image; based on each multi-modal image, calling the trained image processing model to perform a second training task of segmenting each multi-modal image, and taking the consistency loss as a constraint condition in the second training task. Through the application, the accuracy of segmenting multi-modal images can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a training method, apparatus, electronic device, and storage medium for an image processing model. Background Technology

[0002] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. Computer Vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in identifying, locating, and measuring targets, and then further processes the images to create images more suitable for human observation or transmission to instruments for detection.

[0003] Multimodal images often contain missing parts, such as missing patches in the image corresponding to a particular modality, or missing modalities. Related techniques for segmenting abnormal regions in multimodal images with missing modalities typically involve complex model designs. This complicates the processing flow, requires more parameters and computation during training and deployment, and ultimately reduces the accuracy of multimodal image segmentation.

[0004] Currently, there is no good solution for image processing of multimodal images with missing modalities. Summary of the Invention

[0005] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image processing model, which can improve the accuracy of segmenting multimodal images.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a method for training an image processing model, including:

[0008] Acquire multiple multimodal images for use as training samples, wherein the types of the multimodal images include full-modal images and missing modal images;

[0009] Based on each of the multimodal images, the initialized image processing model is invoked to perform a first training task to reconstruct the full-modal image, wherein, during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multimodal images;

[0010] Based on the full-modal image, image completion processing is performed on each of the first full-modal reconstructed images to obtain a full-modal template image;

[0011] Determine the consistency loss between the multimodal image pair and the full-modal template image, wherein the multimodal image pair includes any two of the multimodal images;

[0012] Based on each of the multimodal images, the trained image processing model is invoked to perform a second training task to segment each of the multimodal images, wherein the consistency loss is used as a constraint condition for updating the parameters of the image processing model in the second training task.

[0013] This application provides an image processing method, the method comprising:

[0014] Receive multimodal images to be processed;

[0015] Based on the multimodal image, an image processing model is invoked to perform image segmentation processing to obtain the segmentation result corresponding to the multimodal image. The image processing model is trained based on the image processing model training method provided in the embodiments of this application.

[0016] This application provides a training apparatus for an image processing model, comprising:

[0017] The sample acquisition module is configured to acquire multiple multimodal images used as training samples, wherein the types of the multimodal images include full-modal images and missing modal images;

[0018] The pre-training module is configured to call the initialized image processing model to perform a first training task to reconstruct the full-modal image based on each of the multimodal images, wherein, during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multimodal images;

[0019] The pre-training module is further configured to perform image completion processing on each of the first full-modal reconstructed images based on the full-modal images to obtain full-modal template images;

[0020] The model adjustment module is configured to determine the consistency loss between the multimodal image pair and the full-modal template image, wherein the multimodal image pair includes any two of the multimodal images;

[0021] The model adjustment module is further configured to call the trained image based on each of the multimodal images.

[0022] The image processing model is subjected to a second training task to segment each of the multimodal images, wherein the consistency loss is used as a constraint to update the parameters of the image processing model in the second training task.

[0023] This application provides an image processing apparatus, the image processing apparatus comprising:

[0024] The image receiving module is configured to receive multimodal images to be processed;

[0025] The image processing module is configured to call an image processing model to perform image segmentation processing based on the multimodal image to obtain the segmentation result corresponding to the multimodal image. The image processing model is trained based on the image processing model training method provided in the embodiments of this application.

[0026] This application provides an electronic device, including:

[0027] Memory is used to store executable instructions for a computer;

[0028] The processor, when executing computer-executable instructions stored in the memory, implements the training method for the image processing model provided in the embodiments of this application.

[0029] This application provides a computer-readable storage medium storing computer-executable instructions for inducing a processor to execute and implement the training method of the image processing model provided in this application.

[0030] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the training method for the image processing model provided in this application.

[0031] The embodiments of this application have the following beneficial effects:

[0032] By training the image processing model in stages, it is enabled to reconstruct missing parts in multimodal images and accurately segment specific regions within them. Using consistency loss as a constraint, the image processing model maintains consistency in segmentation results when processing multimodal images with different missing modalities, thus improving the accuracy of multimodal image segmentation. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the application mode of the training method for the image processing model provided in the embodiments of this application;

[0034] Figure 2A This is a schematic diagram of the server structure provided in an embodiment of this application;

[0035] Figure 2B This is a schematic diagram of the server structure provided in an embodiment of this application;

[0036] Figure 2C This is a schematic diagram of the structure of the image processing model provided in the embodiments of this application;

[0037] Figures 3A to 3K This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application;

[0038] Figure 4A This is a diagram illustrating joint training;

[0039] Figure 4B This is a schematic diagram of the missing modality image provided in the embodiments of this application;

[0040] Figure 4C This is a schematic diagram of the segmented region provided in the embodiments of this application;

[0041] Figure 4D This is a comparison chart of training effects provided in the embodiments of this application;

[0042] Figure 4E This is a schematic diagram of the training samples provided in the embodiments of this application;

[0043] Figure 5A This is a schematic diagram of the image processing flow provided in the embodiments of this application;

[0044] Figure 5B This is a schematic diagram of the segmentation results provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the training process of the image processing model provided in the embodiments of this application;

[0046] Figure 7A This is a schematic diagram of the segmentation results provided in an embodiment of this application;

[0047] Figure 7B This is a consistency loss analysis table provided in the embodiments of this application;

[0048] Figure 7C as well as Figure 7D This is a comparison result table provided in the embodiments of this application;

[0049] Figure 8 This is a flowchart illustrating the training method for the image processing model provided in this application embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0053] It should be noted that in the embodiments of this application, user information, user feedback data and other related data are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0056] 1) Image Segmentation: Image segmentation is a crucial process in computer vision. It involves dividing visual input into segments to simplify image analysis. A segment represents an object or a portion of an object and is composed of a set of pixels or "superpixels." Image segmentation organizes pixels into larger parts, eliminating the need to treat individual pixels as units of observation. Image segmentation is used to identify parts of an image and understand what objects they belong to, forming the basis for object detection and classification. Image segmentation can be applied in fields such as face detection, medical imaging, and autonomous driving.

[0057] 2) Magnetic Resonance Imaging (MRI): Images acquired through magnetic resonance imaging (MRI) technology. MRI is a relatively new medical imaging technique that uses static and radiofrequency magnetic fields to image human tissues. During the imaging process, it obtains high-contrast, clear images without the use of electron ionizing radiation or contrast agents. It can reflect organ dysfunction and early lesions from the molecular and cellular levels. A single MRI scan typically contains multiple modalities, with different modalities highlighting different lesion areas.

[0058] 3) Missing Modality: In clinical applications, a set of MRI images includes sub-images of multiple modalities. Due to image corruption, artifacts, acquisition protocols, patient allergies to contrast agents, or cost, MRI images often exhibit one or more missing modalities. For example, a full-modality MRI set may include images of four modalities, but during actual acquisition, only three modalities are acquired, resulting in missing modalities in the acquired MRI images.

[0059] 4) Masked Autoencoder (MAE): As a self-supervised image framework, the masked autoencoder has achieved great success in the field of self-supervision. The proxy task of the masked autoencoder is to guide the model to restore the original pixel values ​​of an image based on the visible small patches (tiles) in the image.

[0060] 5) Model Inversion (MI): Model inversion has long been used in the field of interpretability in deep learning. The goal of this technique is to synthesize images that best represent certain network predictions, such as saliency maps for classification.

[0061] 6) Supervised learning: By training on training data that contains both features and identification labels, machine learning establishes a relationship between features and labels. After training, it can predict labels based solely on feature data.

[0062] 7) Knowledge Distillation: Knowledge distillation involves constructing a lightweight, smaller model and training it using the supervision information from a larger, more powerful model. This results in better performance and accuracy for the smaller model. The larger model is called the teacher model, and the smaller model is called the student model. The supervision information output by the teacher model is called knowledge, and the process by which the student model learns and transfers this supervision information is called distillation.

[0063] 8) Self-Distillation (SD): Self-distillation uses supervised learning to perform knowledge distillation. Compared to the original knowledge distillation method, in the process of self-distillation, the teacher model and the student model are a single model, that is, the model guides itself to learn and complete the knowledge distillation.

[0064] 9) Co-training: Co-training is a type of semi-supervised learning method based on "divergence," originally designed for "multi-view" data. In the multimodal scenario applied in the embodiments of this application, co-training refers to training the full-modal data model and the missing modality data model together, and using the content consistency between different modality combinations to perform knowledge transfer between corresponding models.

[0065] This application provides a training method for an image processing model, a training device for an image processing model, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of segmenting multimodal images.

[0066] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of user terminals, such as laptops, tablets, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and in-vehicle terminals, or as servers. The following will describe exemplary applications when the device is implemented as a server.

[0067] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the training method for the image processing model provided in the embodiments of this application; for example, Figure 1 The system involves a training server 200-1, an image processing server 200-2, a network 300, and a terminal device 400. The training server 200-1 and the image processing server 200-2 communicate through the network 300 or through other means. The terminal device 400 connects to the image processing server 200-2 through the network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0068] For example, the user is a researcher or medical professional, and the multimodal image to be processed can be an MRI image of the human body. A set of MRI images includes sub-images of multiple modalities. The segmentation result is the region with abnormalities in the multimodal image. The image processing server 200 is a server used to segment the region with abnormalities (e.g., tumors) in the MRI image. The user can determine the lesions and other problems in the human body based on the segmentation result. The following is an explanation based on the above example.

[0069] Training server 200-1 acquires full-modality images and multiple missing modality images as training samples, and trains the initialized image processing model based on the training samples using the image processing model training method provided in this application embodiment, obtaining a trained image processing model, and then synchronizes the trained image processing model to image processing server 200-2. The trained image processing model is used for segmentation processing of MRI images.

[0070] Upon receiving the multimodal image to be processed from the terminal device 400, the image processing server 200-2 calls an image processing model to perform image segmentation processing based on the multimodal image to obtain a segmentation result. The image processing server 200-2 then sends the segmentation result to the terminal device 400 via the network 300. The terminal device 400 displays the segmentation result to the user, who can use the segmentation result as a diagnostic basis.

[0071] In some embodiments, the training method of the image processing model of this application can also be applied to the training process of different image processing models and different application scenarios, as described in detail below.

[0072] (1) Medical image processing, for example: training samples include: MRI images of human organs with lesions and MRI images of healthy human organs. The trained image processing model is used to segment the MRI images of human organs. The segmentation result is the lesion area of ​​the human organ. Users can use the segmentation result as a diagnostic basis.

[0073] (2) Industrial inspection, for example: training samples include: computed tomography (CT) images of opaque objects with defects (e.g., industrial materials or parts), CT images of objects with quality conforming to the label, and the trained image processing model is used to detect defective areas in opaque objects (e.g., pores, inclusions, pinholes, shrinkage cavities, delamination). Technicians determine the defects in the items through the segmentation results, thereby improving the efficiency of quality inspection work.

[0074] (3) Face detection, for example: training samples include: video sequences containing faces, labeled data are the face regions in each frame of the video sequence, the trained image processing model is used to segment the face regions in the image, and the trained image processing model can be used to provide face recognition services.

[0075] (4) Autonomous driving, for example: training samples include: video sequences including street scenes, labeled data is the area where obstacles (e.g. vehicles, roadblocks, guardrails, etc.) are located in each frame of the video sequence, and the trained image processing model is used to segment the images collected in real time by the camera of the autonomous vehicle to obtain the obstacle area in the image so that the autonomous vehicle can determine the safe driving area based on the obstacle area.

[0076] This application embodiment can be implemented using blockchain technology. The image processing model trained in this application embodiment can be uploaded to the blockchain for storage, and the reliability of the image processing model can be guaranteed through a consensus algorithm. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0077] This application embodiment can be implemented using database technology. A database, simply put, can be viewed as an electronic filing cabinet storing electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, having minimal redundancy, and being independent of application programs.

[0078] A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile devices; or according to the query language used, such as Structured Query Language (SQL) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0079] This application embodiment can also be implemented using cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business models. It can form a resource pool, available on demand, offering flexibility and convenience. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, and driven by demands for search services, social networks, mobile commerce, and open collaboration, every item may eventually possess its own hash-coded identification mark, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0080] In some embodiments, training server 200-1 and image processing server 200-2 can be integrated into a single physical server.

[0081] In some embodiments, the training server 200-1 or the image processing server 200-2 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Electronic devices can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, etc., but are not limited to these. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0082] See Figure 2A , Figure 2A This is a schematic diagram of the server structure provided in an embodiment of this application. Figure 2A The training server 200-1 shown includes at least one processor 410, memory 450, at least one network interface 420, and a user interface 430. The various components in the training server 200-1 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general labeled all buses as Bus System 440.

[0083] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0084] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0085] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0086] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0087] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0088] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0089] In some embodiments, the training apparatus for the image processing model provided in this application can be implemented in software. Figure 2A A training device 455 for an image processing model stored in memory 450 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a sample acquisition module 4551, a pre-training module 4552, and a model tuning module 4553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0090] See Figure 2B , Figure 2B This is a schematic diagram of the server structure provided in an embodiment of this application. Figure 2B The image processing server 200-2 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the image processing server 200-2 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2B The general labeled all buses as Bus System 440.

[0091] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0092] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0093] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0094] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0095] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0096] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0097] In some embodiments, the training apparatus for the image processing model provided in this application can be implemented in software. Figure 2B An image processing apparatus 456 stored in memory 450 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: an image receiving module 4554 and an image processing module 4555. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0098] The training method for the image processing model provided in this application will be described by referring to the exemplary application and implementation of the server provided in the embodiments of this application. See also Figure 3A , Figure 3A This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Figure 1 The server in the middle is the execution entity, and will combine Figure 3A The steps shown are explained.

[0099] In step 301, multiple multimodal images are acquired to serve as training samples.

[0100] For example, the types of multimodal images include full-modal images and missing modal images, and multiple multimodal images are used as training samples.

[0101] In this embodiment, the multimodal image is an MRI image of a human organ, which is used as an example for illustration. A set of MRI images includes sub-images of multiple modalities. During the actual acquisition process, some sub-images of some modalities of the MRI image, or patches in some sub-images, may be lost, forming missing modal images. The image processing model is used to segment abnormal (e.g., lesion) regions present in the MRI image.

[0102] For example, obtaining a multimodal image can be achieved by randomly masking the tiles in the full-modal image. Masking the tiles can be done using image processing software (Photoshop, PS).

[0103] In some embodiments, reference Figure 3J , Figure 3J This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 301 is implemented through the following steps 3011 to 3012, which are described in detail below.

[0104] In step 3011, the full-modal image is acquired.

[0105] For example, a full-modal image includes sub-images of multiple modalities. Taking multimodal images as an example, an MRI image is used to illustrate the process of acquiring a set of full-modal MRI images containing abnormal (e.g., lesion) regions.

[0106] In step 3012, the patches in the sub-images of the full modality image are subjected to multiple different masking processes to obtain multiple different missing modality images. The multiple missing modality images and the full modality image are used as training samples.

[0107] For example, masking the entire sub-image is a special case of processing patches within a sub-image. (See reference...) Figure 4E , Figure 4E This is a schematic diagram of the training samples provided in the embodiments of this application; Figure 4E Training samples for 15 different multimodal images obtained by masking the full modality image, when the full modality image includes four modalities, are given.

[0108] In some embodiments, reference Figure 2C , Figure 2C This is a schematic diagram of the structure of the image processing model provided in the embodiments of this application; the initialized image processing model 201C includes: a multimodal mask autoencoder 210C; the multimodal mask autoencoder 210C is used to perform mask processing for full-modal images.

[0109] For example, the initial image processing model does not yet have the ability to accurately reconstruct the missing parts in a multimodal image, but it can perform masking on the full-modal image to obtain images of different missing modalities.

[0110] In this embodiment, training samples are obtained by using an initialized image processing model. This allows for the simultaneous acquisition of labels corresponding to the training samples during the acquisition process, saving the cost of acquiring training samples, alleviating the complexity of the training task, and conserving the computing resources required for the server to train the model.

[0111] Continue to refer to Figure 3A In step 302, based on each multimodal image, the initialized image processing model is invoked to perform the first training task of reconstructing the full-modal image.

[0112] For example, during the execution of the first training task, the image processing model outputs the first full-modal reconstructed image corresponding to each multimodal image. The goal of the first training task is to enable the initialized image processing model to reconstruct multimodal images with missing parts.

[0113] For ease of explanation, the multimodal images in the training samples are represented as Where W, H, and D are the width W, height H, and number of slices D in the image, respectively, and N is the number of modalities, representing a multimodal image. Each modality comprises multiple smaller blocks. The multimodal image includes: missing modality images. ... and full-modal images , where n is a positive integer greater than 1.

[0114] In some embodiments, reference Figure 3B , Figure 3B This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 302 is implemented through the following steps 3021 to 3023, which are described in detail below.

[0115] In step 3021, the initialized image processing model is invoked to perform reconstruction processing based on each multimodal image, thereby obtaining the first full-modal reconstructed image corresponding to each multimodal image.

[0116] For example, the reconstruction process is achieved by predicting the missing parts based on the non-missing parts in the multimodal image, and then combining the predicted missing parts with the multimodal image to obtain the completed reconstructed image.

[0117] In some embodiments, reference Figure 3C , Figure 3C This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 3021 is implemented through the following steps 30211 to 30213, which are described in detail below.

[0118] In step 30211, the initialized image processing model is invoked based on each multimodal image to perform the following processing: the multimodal image is encoded to obtain the first encoded vector of the multimodal image.

[0119] For example, the first encoding vector is the encoding vector of the non-missing portion of the multimodal image. (See reference) Figure 4B , Figure 4B This is a schematic diagram of a missing modality image provided in an embodiment of this application; the non-missing portion of the missing modality image consists of three modes, including FLAIR, T1c, and T2. The missing portion is mode T1. Based on Figure 4B Taking the missing modality image as an example, we will encode the three modalities FLAIR, T1c, and T2 in the missing modality image to obtain the first encoding vector.

[0120] In step 30212, missing part prediction processing is performed based on the first encoding vector to obtain the first prediction vector of the missing part in the multimodal image.

[0121] For example, continuing with the example above, based on the first encoding vector for the missing part ( Figure 4B The sub-image corresponding to the T1 mode is predicted to obtain the encoding vector of the missing part, which is also the first prediction vector.

[0122] In step 30213, the first prediction vector and the first encoding vector are integrated to obtain the first full-modal reconstructed image.

[0123] For example, the first encoding vector corresponding to the unmissing part and the first prediction vector of the missing part are padded to form the encoding vector corresponding to the full-modal image. The encoding vector is then restored to the image to obtain the first full-modal reconstructed image, which can be represented as the full-modal image. .

[0124] In some embodiments, continue to refer to Figure 2C The initialized image processing model 201C includes: a multimodal mask autoencoder 210C and a regression network 220C. The multimodal mask autoencoder includes: an encoder layer 211C and a decoder layer 212C. The encoder layer 211C is used to perform encoding processing. The decoder layer 212C is used to perform missing part prediction processing. The regression network 220C is used to perform integration processing.

[0125] Continue to refer to Figure 3B In step 3022, a first mean squared error loss is determined based on each first full-modal reconstructed image and the full-modal image.

[0126] For example, the first mean squared error loss can be characterized by the formula .in, Characterize the full-modal images in the training samples. Representational Alternative Multimodal Images The missing portion of the image is the first full-modal reconstructed image. For operations on the content at the corresponding position, F is a reconstruction function that cascades a multimodal mask autoencoder and a regression head.

[0127] In step 3023, the initialized image processing model is backpropagated based on the first mean square error loss to obtain the trained image processing model.

[0128] In this application, the initialized image processing model is backpropagated iteratively. The constraints in the backpropagation process are described below.

[0129] In some embodiments, reference Figure 3D , Figure 3D This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 3023 is implemented through the following steps 30231 to 30232, which are described in detail below.

[0130] In step 30231, the first full-modal reconstructed image is substituted into the regularization function to obtain the first regularization term, and the minimum sum of the first mean squared error loss and the first regularization term is taken as the first constraint condition.

[0131] For example, the regular expression function is R(). yes The regularization term and the first constraint can be summarized as the following formula (3):

[0132]

[0133] in, These are weight values, which can be set according to the actual training needs.

[0134] In step 30232, the parameters of the initialized image processing model are updated based on the first constraint and the first mean squared error loss to obtain the trained image processing model.

[0135] For example, the parameters of the initialized image processing model are iteratively updated until the first constraint condition is met. The image processing model that meets the first constraint condition is then used as the trained model. (Continue to refer to...) Figure 2C After the first training task, the trained image processing model 202C is obtained, in which the regression network 220C is replaced by the segmentation network 230C in order to facilitate the second training task.

[0136] In this embodiment of the application, the first training task enables the image processing model to learn the relationship between different modalities in a multimodal image, thereby enabling the image processing model to reconstruct the image and improve the accuracy of filling in the missing parts of the missing modal image.

[0137] Continue to refer to Figure 3A In step 303, image completion processing is performed on each first full-modal reconstructed image based on the full-modal image to obtain a full-modal template image.

[0138] For example, the backpropagation processing in steps 303 and 302 is performed synchronously. When the first full-modal reconstructed image is obtained, a full-modal template image is obtained based on the first full-modal reconstructed image and the full-modal image. During the backpropagation processing iteration, the full-modal template image is continuously optimized using the first full-modal reconstructed image obtained from the forward propagation output before each backpropagation processing. When the first training task is completed, the corresponding optimized full-modal template image is also obtained.

[0139] In some embodiments, reference Figure 3E , Figure 3E This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 303 is implemented through the following steps 3031 to 3034, which are described in detail below.

[0140] In step 3031, the following processing is performed for each multimodal image: the missing parts in the multimodal image are determined, and the missing parts are filled in based on the first full-modal reconstructed image to obtain the first filled image.

[0141] For example, step 3031 can be represented by the following formula That is, using the first full modality to reconstruct the image Fill in the corresponding content in the multimodal image. The missing parts are used to obtain the first completed image.

[0142] In step 3032, linear regression processing is performed on the first completed image to obtain the linear regression result, and the first mean squared error loss between the linear regression result and the full modality image is obtained.

[0143] For example, linear regression is implemented through a regression network, and linear regression can be represented by the formula... The first mean squared error loss has already been explained above and will not be repeated here.

[0144] In step 3033, a target full-modal reconstruction image that minimizes the first mean squared error loss is obtained from each first full-modal reconstruction image, and the target full-modal reconstruction image is substituted into the regularization function to obtain the first regularization term.

[0145] As an example, the first regular expression term has already been explained above and will not be repeated here.

[0146] In step 3034, the sum of the first regularization term and the target full-modal reconstructed image is used as the full-modal template image.

[0147] Example, full-modal template image It can be expressed as the following formula (1):

[0148]

[0149] This application embodiment obtains a full-modal template image, enabling the image processing model to learn the relationship between each modality in the multimodal image, thereby improving the accuracy of reconstructing the multimodal image and saving computational resources.

[0150] Continue to refer to Figure 3A In step 304, the consistency loss between the multimodal image pair and the full-modal template image is determined.

[0151] For example, a multimodal image pair includes any two multimodal images; assuming that the two multimodal images are each represented by a first image. Second image Consistency loss can be characterized as That is, to obtain the first image. Second image Separately by full-modal template images The mean squared error loss between the images after completion.

[0152] In some embodiments, reference Figure 3F , Figure 3F This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 304 is implemented through the following steps 3041 to 3042, which are described in detail below.

[0153] In step 3041, the following processing is performed on each multimodal image in the multimodal image pair: the missing part in the multimodal image is determined, and the missing part is filled in based on the full-modal template image to obtain a second filled image.

[0154] For example: the first image The missing modality T1 in the image will be replaced with the full modality template image. Modality T1 in the first image is supplemented. This yields a second completed image. The missing modality T1c in the image will be replaced with a full modality template image. Modal T1c in the second image is supplemented. This yields another second completed image.

[0155] In step 3042, the second mean square error loss between the two corresponding second completed images of the multimodal image pair is determined, and the second mean square error loss is used as the consistency loss.

[0156] For example, the two second complete images corresponding to each multimodal image in a multimodal image pair include: the second complete image of the first multimodal image corresponding to each multimodal image in the multimodal image pair, and the second complete image of the second multimodal image corresponding to each multimodal image in the multimodal image pair.

[0157] In this embodiment, by obtaining the consistency loss, it is convenient to introduce a self-distillation method to train the image processing model, thereby promoting the consistency of multimodal images with different missing conditions in the latent space of the image processing model and improving the accuracy of image segmentation by the image processing model.

[0158] Continue to refer to Figure 3A In step 305, based on each multimodal image, the trained image processing model is invoked to perform a second training task to segment each multimodal image.

[0159] For example, the image processing model invoked in step 305 is the image processing model trained after the first training task. Figure 2C The trained image processing model 202C is used as a constraint to update the parameters of the image processing model in the second training task with consistency loss.

[0160] In some embodiments, reference Figure 3G , Figure 3G This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 305 is implemented through steps 3051 to 3053, which are described in detail below.

[0161] In step 3051, the trained image processing model is invoked to perform image segmentation processing based on each multimodal image, and the predicted segmentation result corresponding to each multimodal image is obtained.

[0162] For example, the segmentation process includes two parts: image reconstruction and segmentation of the reconstructed image. In the trained image processing model, the regression network is replaced with a segmentation network, which reduces the redundancy of the model.

[0163] In some embodiments, reference Figure 3H , Figure 3H This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 3051 is implemented through the following steps 30511 to 30514, which are described in detail below.

[0164] In step 30511, the trained image processing model is invoked based on each multimodal image to perform the following processing: the multimodal image is encoded to obtain the second encoding vector of the multimodal image.

[0165] For example, the second encoding vector is the encoding vector of the non-missing part in the multimodal image; the principle of encoding processing can be referred to step 30211 above, and will not be repeated here.

[0166] In step 30512, the missing part in the multimodal image is obtained, and the third encoding vector corresponding to the missing part is extracted from the full-modal template image.

[0167] For example, the missing parts in the multimodal image are obtained, and the patches corresponding to the positions of the missing parts are extracted from the full-modal template image. The extracted patches are then encoded to obtain the third encoding vector.

[0168] In step 30513, missing part prediction processing is performed based on the third encoding vector and the second encoding vector to obtain the second reconstructed full-modal image.

[0169] For example, predicting the actual missing parts in a multimodal image based on the third and second encoding vectors can improve the accuracy of the reconstructed image, thus obtaining a second reconstructed full-modal image that is more consistent with the actual image.

[0170] In step 30514, the second reconstructed full-modal image is segmented, and the predicted segmentation results are obtained for each of the multimodal images.

[0171] In some embodiments, reference Figure 2C The image processing model 202C trained by the first training task includes: a multimodal mask autoencoder 210C and a segmentation network 230C. The multimodal mask autoencoder 210C includes: an encoder layer 211C and a decoder layer 212C. The encoder layer 211C is used to perform encoding processing and obtain a third encoding vector. The decoder layer 212C is used to perform missing part prediction processing. The segmentation network 230C is used to perform segmentation processing.

[0172] Continue to refer to Figure 3G In step 3052, the segmentation loss of the image processing model is determined based on the predicted segmentation result and the actual segmentation result.

[0173] For example, for multimodal images The segmentation loss is obtained by performing segmentation. It is represented by the following formula (5):

[0174]

[0175] in, It is the sum of the widely used Dice loss and cross-entropy loss. It corresponds to the decoder layer 212C. The result obtained by segmenting the feature map output by the neural network layer with a sampling ratio is also known as the predicted segmentation result. Characterizes the actual segmentation result.

[0176] In step 3053, the image processing model is backpropagated based on the consistency loss and segmentation loss to obtain the retrained image processing model.

[0177] Example: Image processing model after retraining ( Figure 2C The image processing model 203C, trained in [the dataset], is used for segmentation of multimodal images with missing modalities. The consistency loss is used as a constraint during backpropagation, referencing [the relevant data / process]. Figure 3I , Figure 3I This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Step 3053 is implemented through the following steps 30531 to 30534, which are described in detail below.

[0178] In step 30531, feature maps of the second completed images are extracted from the second completed images corresponding to the two multimodal images in the multimodal image pair.

[0179] In some embodiments, continue to refer to Figure 2C The trained image processing model 202C includes a multimodal mask autoencoder 210C, which includes an encoder layer 211C and a decoder layer 212C. The decoder layer 212C includes multiple layers of feature extraction layers (neural network layers). The feature map is obtained by calling the feature extraction layer.

[0180] In step 30532, the third mean square error loss between the feature maps of the second completed images corresponding to the two multimodal images is determined, and the third mean square error loss is equal to the consistency loss as the second constraint condition.

[0181] For example, the second constraint can be represented by the following formula (2):

[0182]

[0183] in, , These are multimodal images Two different missing cases; yes , The corresponding feature map in the latent space, These are the number of channels, depth, height, and width of the feature map, respectively. Formula (2) means that... and Mean square error between the feature maps in the corresponding latent space , obtain and Consistency loss between From the distillation process, there is a loss of consistency. With mean square error With the goal of achieving equality, the parameters of the multimodal mask autoencoder are adjusted.

[0184] In step 30533, the sum of the consistency loss and the partitioning loss is minimized as the third constraint.

[0185] For example, the third constraint can be represented by the following formula (4):

[0186]

[0187] in It is a segmentation loss. λ represents the segmentation label (the actual segmented region), and λ is the loss weight, which is set to 0.1 in this embodiment. This embodiment employs a deep-supervised strategy to train a multimodal segmentation network (image processing model).

[0188] In step 30534, the parameters of the image processing model are updated based on the consistency loss and segmentation loss until the second and third constraints are satisfied.

[0189] For example, the second constraint, representing self-distillation, promotes consistency in the latent space of multimodal images with different missing modalities, thereby improving the accuracy of image segmentation. The third constraint, representing an improvement in segmentation accuracy, iteratively trains until the constraints are met, which can improve the accuracy of image processing model segmentation of images with missing modalities.

[0190] This application also proposes an image processing method, see [link to relevant documentation] Figure 3K , Figure 3K This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. Figure 1 The image processing server 200-2 in the image processing module is the main execution unit, and will combine... Figure 3K The steps shown are explained.

[0191] In step 306, the multimodal image to be processed is received.

[0192] For example, multimodal images can be MRI images of human organs, and missing data may exist in multimodal images.

[0193] In step 307, an image processing model is invoked based on the multimodal image to perform image segmentation processing, thereby obtaining the segmentation result corresponding to the multimodal image.

[0194] For example, in response to the presence of missing parts in a multimodal image, image processing server 200-2 invokes an image processing model to perform segmentation processing on the multimodal image. The image processing model is trained based on the training method of the image processing model provided in the embodiments of this application.

[0195] In some embodiments, step 302 is implemented by calling an image processing model based on the multimodal image to perform the following processing: encoding the multimodal image to obtain a fourth encoding vector of the multimodal image, wherein the fourth encoding vector is the encoding vector of the non-missing part in the multimodal image; obtaining the missing part in the multimodal image and extracting the fifth encoding vector corresponding to the missing part from the full-modal template image; performing missing part prediction processing based on the fourth encoding vector and the fifth encoding vector to obtain a third reconstructed full-modal image; and performing segmentation processing on the third reconstructed full-modal image to obtain the predicted segmentation result corresponding to the multimodal image.

[0196] In some embodiments, the image processing model includes: a multimodal mask autoencoder and a segmentation network, wherein the multimodal mask autoencoder includes: an encoder layer and a decoder layer; the encoder layer is used to perform encoding processing and obtain a fifth encoded vector; the decoder layer is used to perform missing part prediction processing; and the segmentation network is used to perform segmentation processing.

[0197] This application embodiment trains the image processing model in stages, enabling it to reconstruct missing parts in multimodal images and accurately segment specific regions within them. By using consistency loss as a deterministic constraint, the image processing model maintains consistency in segmentation results when processing multimodal images with different missing modalities, thus improving the accuracy of multimodal image segmentation.

[0198] The following will describe an exemplary application of the image processing model training method provided in this application embodiment in a real-world application scenario.

[0199] In clinical applications, MRI images comprise sub-images of multiple modalities. Due to image corruption, artifacts, acquisition protocols, patient allergies to contrast agents, or cost considerations, MRI images often exhibit one or more missing modalities. Processing multimodal images with missing modalities involves two types of methods: specialized and general-purpose. General-purpose methods train only one model to handle all missing modalities, while specialized methods require training a separate model for each missing modality (for a task with N modalities, a specialized method requires training...). (One model).

[0200] In related technologies, general-purpose methods, whether by explicitly generating missing modalities or generating general feature representations in the latent space, involve relatively complex model designs, such as multiple encoders and decoders and complex interactions within the model. This makes the processing flow more complex and requires more parameters and computation during training and deployment. In addition, existing general-purpose methods ignore the relationships between different modality combinations, so the resulting model performance may be suboptimal.

[0201] Specialized methods, through a joint training strategy, enable the model to achieve better results even with missing modalities, especially when there are many missing modalities. (Reference) Figure 4A , Figure 4A This is a diagram illustrating joint training; Figure 4A The process of joint training in related technologies is demonstrated. Image processing model 401A is trained based on full-modal images (including four modalities: FLAIR, T1, T1c, and T2), and image processing model 402A is trained based on images with missing modalities (which are missing two modalities, T1 and T1c, compared to full-modal images). Consistency constraints are applied between the features and outputs of the models corresponding to the full-modal and missing modalities (one of them). For each case of missing modalities, separate training is required. , These represent the full modality images ( ) and missing modality images ( Consistency constraints are applied between the network features (latent space) and the output.

[0202] However, specialized methods require separate model training for each missing modality, incurring greater time and computational costs during training and requiring more storage space during deployment. Furthermore, existing specialized methods can only perform mutual distillation between different modalities (e.g., full modality and any single modality), and cannot model the relationships between multiple missing modalities.

[0203] The image processing model training method provided in this application belongs to the general method for handling missing modalities, training one image processing model to cope with all missing modalities. The multimodal mask autoencoder of this application adopts a classic single encoder-decoder structure. By designing pre-training and incorporating model inversion for missing modality completion, the image processing model learns better full-modal and missing modal feature representations in a self-supervised manner without task-related annotations. Furthermore, the method of this application incorporates a self-distillation training strategy during fine-tuning, allowing the model to perform better on segmentation tasks in both missing and full modalities. The model trained in this application performs knowledge distillation between feature maps corresponding to different modalities (including full and missing modalities). Compared to joint training, only one model needs to be trained to cope with all missing modalities, and better results are achieved in both missing and full modalities. Reference Figure 4D , Figure 4D This is a comparison chart of training effects provided in the embodiments of this application; Figure 4D This demonstrates the number of parameters in models trained using different methods at deployment time, as well as the average Dice coefficient based on all missing modal combinations on the public benchmark dataset BraTS2018 test set. Figure 4D The Dice coefficient (DSC%) is a set similarity metric, commonly used to evaluate medical image segmentation. It uses values ​​between 0 and 1 to measure the overlap between the segmented region and the actual tumor region (Ground Truth). A higher Dice score indicates better segmentation performance. The radius of the model circle represents computational complexity, which can be obtained by calculating the model's giga-floating-point operations per second (GFLOPS). Compared to four existing state-of-the-art solutions: Heteromodal Variational Encoder-Decoder (U-HVED) for simultaneous modality completion and segmentation, Adversarial Joint Training Network (ACN) for brain tumor segmentation with missing modalities, Style Matching (U-Net) in missing modality brain tumor segmentation (SMU-Net), and Region-Aware Fusion Network (RFNet) for incomplete multimodal brain tumor segmentation. References Figure 4D As can be seen, the embodiments of this application are based on a multimodal mask self-encoder ( The image processing model trained with relatively low parameter count and computational complexity achieves better segmentation results than existing technologies.

[0204] refer to Figure 8 , Figure 8This is a flowchart illustrating the training method of the image processing model provided in this application embodiment. The server will be used as the execution entity in the following description, combined with... Figure 8 The training method of the image processing model provided in the embodiments of this application will be explained.

[0205] In step 801, training samples are obtained.

[0206] For example, training samples are generated using an untrained multimodal mask autoencoder. The full-modal image is input into the untrained multimodal mask autoencoder, which randomly discards some modalities and small patches of the remaining modalities to construct training samples.

[0207] Example, reference Figure 6 , Figure 6 This is a schematic diagram of the training process of the image processing model provided in this application embodiment; the untrained multimodal mask autoencoder includes a multimodal mask autoencoder 601 and a regression network 602. The multimodal mask autoencoder 601 includes an encoder 603 and a decoder 604. The encoder 603 and the decoder 604 include multiple feature extraction layers.

[0208] Multimodal mask autoencoder pre-training framework ( This is a mask autoencoder pre-training method for medical multimodal images. Given a multimodal image... Where W, H, and D are the width W, height H, and number of slices D in the image, respectively, and N is the number of modalities, representing a multimodal image. Each modality consists of multiple small patches. The multimodal image x contains no modalities or missing patches within a modality. The multimodal image x is a sample template. Random sampling based on the multimodal image x yields multiple different training samples. Multiple missing modal images and the full modal image are used as training samples.

[0209] In real-world scenarios, one or more modalities may be missing from an image. Therefore, training samples can be obtained using the following methods:

[0210] Multimodal images Input to an untrained multimodal mask autoencoder An untrained multimodal mask autoencoder lacks the ability to reconstruct missing parts of a multimodal image, but it can still perform random masking. Therefore, the untrained multimodal mask autoencoder randomly masks the multimodal image. Some modalities were used to simulate the situation of missing modalities. Additionally, some 3D blocks of the remaining available modalities were randomly masked, as shown in the image below. Based on... Obtaining multiple (n) training sample images with different modalities can be characterized as multimodal images with missing features. ... and full-modal images , where n is a positive integer greater than 1.

[0211] For example, this explanation uses the entire mode of a random mask as an example. (See reference...) Figure 4E , Figure 4E This is a schematic diagram of the training samples provided in the embodiments of this application; Figure 4E Training samples for 15 different multimodal images obtained by masking the full modality image, when the full modality image includes four modalities, are given.

[0212] Continue to refer to Figure 8 In step 802, the image processing model is pre-trained based on model inversion to obtain a full-modal image for modal completion.

[0213] For example, step 802 corresponds to the first training task mentioned above. By using model inversion, embodiments of this application design a method based on a multimodal mask autoencoder that saves both time and space while obtaining synthetic data to fill in missing modalities at extremely low cost. Model inversion has long been used in the field of deep learning interpretability; the goal of this technique is to synthesize images that best represent certain network predictions, such as saliency maps for classification.

[0214] Model inversion can be achieved as follows: A multimodal mask autoencoder is invoked based on a sample image. The encoder in the multimodal mask autoencoder encodes the sample image to obtain an encoded vector. The decoder of the multimodal mask autoencoder predicts the pixel value vectors of the missing parts based on the encoded vectors. The pixel value vectors of the missing parts are then integrated with the pixel value vectors of the intact parts to obtain the completed full-modal image. .

[0215] Based on each training sample and training samples Corresponding full-modal image A full-modal template image is obtained through optimization. Optimized full-modal images This enables the model to better reconstruct partially masked images and optimize the target. (Full-modal template image) can be represented by the following formula (1):

[0216] , It is a sample image with missing modes, randomly generated from a multimodal image x. Indicates substitution The content that was covered up is For operations on the content at the corresponding position, F is a reconstruction function that cascades a multimodal mask autoencoder f and a regression head. It is the mean squared error (MSE) loss. yes Regular terms, yes The corresponding weight is set to 0.005. The function is used to obtain the loss for the mean squared error. smallest .

[0217] Formula (1) means that, based on the predicted full-modal image, the missing modalities will be... Complete the image, obtaining the completed image and the original full-modal image. The mean squared error between the two is used to find the one that minimizes the mean squared error. This will minimize the mean square error. With full modal images of The regularization results are summed to obtain the full-modality template image. .

[0218] For example, during the pre-training process, the first pre-training uses 0 masking. The content is as follows. Iterative pre-training is performed multiple times, with each pre-training iteration using the full-modality template image obtained from the previous training optimization. To complete the content, use the corresponding information. The content is masked, rather than being directly masked with 0 (a blank mask).

[0219] In this embodiment, the above processing enables better reconstruction of multimodal images with missing content (modalities or partial blocks). The supplemented content captures information representing specific modalities, which also helps improve the multimodal segmentation performance when missing modalities are present. In the actual pre-training process, the multimodal mask autoencoder is iteratively optimized through backpropagation, while simultaneously optimizing the full-modal image. Optimization results In this way, no new modules need to be introduced during the training of the multimodal mask autoencoder, and the cost of optimizing to obtain the full-modal template image is extremely low.

[0220] This application employs a two-stage training method, including pre-training (the first stage) and fine-tuning (the second stage). In the pre-training stage, the loss function is... The optimization objective of the pre-training phase (the first constraint mentioned above) can be summarized as the following formula (3):

[0221]

[0222] Corresponding to formula (1), the pre-training stage enables the multimodal mask autoencoder to learn the relationships and anatomical integrity between modes in the data without any annotations, so as to perform modal completion and obtain... Optimization result: full-modal template image .

[0223] Continue to refer to Figure 8 In step 803, the pre-trained image processing model is subjected to self-distillation based on training samples of different modalities.

[0224] For example, based on the pre-training framework of a multimodal mask autoencoder, this application embodiment designs a computationally efficient self-distillation method that can distill task-related knowledge within the same model in combinations of two training sample images with different missing conditions.

[0225] For example, in each training batch, this application embodiment randomly samples multiple samples with different missing conditions based on the same full-modality sample sampling. The full-modality sample and the multiple samples with different missing conditions are combined into a sample set. Two different modalities (including full-modality and multiple missing modalities) are randomly selected from the sample set, and the multimodal mask autoencoder is called to perform reconstruction processing respectively. During the reconstruction processing, the feature map of the completed modality corresponding to each sample can be obtained (which can be represented as a matrix composed of pixel value vectors). Consistency loss is used in the self-distillation process to promote the semantic consistency of the combination of sample images of the two missing modalities in the latent space (second constraint condition), which can be represented as the following formula (2):

[0226]

[0227] in, , These are multimodal images Two different missing cases; yes , The corresponding feature map in the latent space, These are the number of channels, depth, height, and width of the feature map, respectively. Formula (2) means that... and Mean square error between the feature maps in the corresponding latent space , obtain and Consistency loss between From the distillation process, there is a loss of consistency. With mean square error With the goal of achieving equality, the parameters of the multimodal mask autoencoder are adjusted.

[0228] In this embodiment, distillation from more modal combinations to fewer modal combinations can facilitate the recovery of missing modal information by a multimodal mask autoencoder. At the same time, distillation from fewer missing modal combinations to more missing modal combinations can facilitate the model learning modality-specific information.

[0229] Continue to refer to Figure 8 In step 804, the trained image processing model is fine-tuned.

[0230] For example, during the fine-tuning phase of training, to simulate real-world modality loss scenarios, 0 to 3 modalities are randomly removed and replaced with the full modality template image. The corresponding mode is replaced. See also... Figure 6 The regression network 602 used in the pre-training phase was replaced with a randomly initialized segmentation network. (Segmentation head), the weights of the other parts of the model are initialized using the weights obtained after the first stage of pre-training, and the optimization objective (third constraint) of the second stage is shown in the following formula (4):

[0231]

[0232] in It is a segmentation loss. λ represents the segmentation label (the actual segmented region), and λ is the loss weight, which is set to 0.1 in this embodiment. This embodiment employs a deep-supervised strategy to train a multimodal segmentation network (image processing model), referencing... Figure 6 A multimodal mask autoencoder includes an encoder and a decoder. Both the encoder and decoder consist of multiple neural network blocks. In the decoder, the first two neural network blocks (corresponding to sampling ratios of 1 / 2 and 1 / 4) are used... The corresponding loss (indicated by) is also added to the segmentation loss. Specifically, embodiments of this application use a A convolutional layer followed by a trilinear interpolation upsampling layer yields the segmented output for the corresponding network block. The total segmentation loss can then be expressed as:

[0233]

[0234] It is the sum of the widely used Dice loss and cross-entropy loss. It corresponds The first stage outputs the segmentation results from the neural network blocks at the sampling ratio (including the network's final output, i.e., the segmented regions obtained by filling in missing images and segmenting the filled-in images). The second stage fine-tunes the network (composed of a multimodal mask autoencoder and a segmentation network) into a multimodal segmentation network that can simultaneously handle missing modalities.

[0235] This embodiment is implemented on the PyTorch (1.7.1) neural network framework. The network structure of the image processing model in this embodiment is a three-dimensional "U"-shaped network, and its encoder and decoder are both composed of network blocks with residual structures. This embodiment uses the Adam algorithm as the optimizer during network training, with 600 and 300 training rounds in the first and second stages, respectively. The initial learning rate is 3e-4, and a cosine annealing learning rate scheduling mechanism is used during training (updating the learning rate according to the decay period of the cosine waveform, decreasing from the maximum to the minimum value in the first half of the cycle, and increasing from the minimum to the maximum value in the second half of the cycle).

[0236] The hardware environment for training the model in this embodiment is described below. The image processing model can be trained on two NVIDIA 2080Ti graphics cards with a batch size of 2. To standardize all data, the pixel values ​​of these images are cropped to 1% to 99% of the intensity values, then min-max scaled to the range [0, 1], and finally randomly cropped to a fixed size of 128×128×128 voxels for training. The side length of the random 3D blocks is set to 16 pixels. Initialized with Gaussian noise, λ is set to 0.1. Embodiments of this application use common data augmentation techniques to enhance the diversity of the training data, including random signal value scaling and adjustment, and random flipping along three dimensions.

[0237] Continue to refer to Figure 8 In step 805, based on the nuclear magnetic resonance image to be processed, the trained image processing model is invoked to perform image segmentation processing.

[0238] For example, an image processing model is invoked based on missing modal data. This model includes a multimodal mask autoencoder and a segmentation network. The multimodal mask autoencoder obtains the sequence number of the missing modality and the location of the missing patch in the missing modal data, and then uses the full-modal template image obtained during the training phase. The corresponding modalities and small patches are filled into the missing modal data to obtain a padded multimodal image. The segmentation network in the image processing model performs image segmentation on each modality in the padded multimodal image to obtain the abnormal region (tumor region). (Reference) Figure 7A , Figure 7AThis is a schematic diagram of the segmentation results provided in the embodiments of this application. The images in the upper row are the original images and full-modal images corresponding to each modality (including: FLAIR, T1, T1c, T2). The images in the lower row are the segmentation results corresponding to each modality, the segmentation results corresponding to the full-modal image (Full), and the actual segmentation results (Ground truth).

[0239] refer to Figure 5A , Figure 5A This is a schematic diagram of the image processing flowchart provided in an embodiment of this application. The image processing model trained in this embodiment can be stored in a cloud server. Multimodal image data is input into the cloud server, where any zero or more modalities of the multimodal image data may be missing. The cloud server performs segmentation processing on the multimodal image data based on the image processing model and outputs the brain tumor region segmentation result. (Reference) Figure 4C , Figure 4C This is a schematic diagram of the segmented region provided in the embodiments of this application. Figure 4C The results of brain tumor region segmentation are shown. Image GT is a modality of brain MRI image obtained by completing the modality. Segmentation region 401C is the abnormal region obtained by segmenting image GT. Different lesions (e.g., edema, necrosis, enhanced tumor, non-enhanced tumor core, etc.) are represented by different display methods (e.g., different colors or different gray levels) in the abnormal region.

[0240] The application scenarios of this application embodiment can be other types of multimodal medical image data combinations and other body parts (such as lung tumors), see reference. Figure 5B , Figure 5B This is a schematic diagram of the segmentation results provided in an embodiment of this application; Figure 5B Figure (a) shows the segmentation result obtained by segmenting a lung image acquired through positron emission tomography (PET) according to an embodiment of this application. Figure (b) shows the segmentation result obtained by segmenting a lung image acquired through computed tomography (CT) according to an embodiment of this application.

[0241] The effects of the embodiments in this application are as follows:

[0242] (1) This embodiment of the application can perform knowledge distillation among multiple missing modal combinations without using joint training. Only one model needs to be trained to handle all missing modal cases, simplifying the training process, reducing the overall computational load and GPU memory consumption during training, as well as the storage consumption during deployment. At the same time, this embodiment of the application can implicitly model the relationship between multiple missing modal combinations. Compared with the joint training framework, this embodiment of the application can achieve better results than the existing best methods in missing modal data.

[0243] (2) The self-distillation strategy proposed in this application combined with the multimodal mask autoencoder can also achieve better results in full-modal data. Experimental results on the BraTS 2018 official online validation dataset show that its segmentation results in full-modal data are better than the existing best brain MRI tumor segmentation method in the absence of modality.

[0244] The effectiveness of this application's embodiments was experimentally verified in the BraTS 2018 brain tumor segmentation competition. The BraTS series dataset consists of multi-contrast MRI images of four modalities: T1, T1c, T2, and FLAIR. These data were processed and organized by the competition organizers, including skull removal, resampling to a uniform resolution (1 m³), ​​and co-registration on the same template. In this competition, four intratumoral structures (edema, enhancing tumor, necrotic and non-enhancing tumor core) were divided into three tumor regions and used as the segmentation targets: 1. Whole Tumor (WT), including all tumor regions; 2. Tumor Core (TC), composed of enhancing tumor, necrotic areas, and the non-enhancing tumor core; 3. Enhancing Tumor (ET).

[0245] The BraTS2018 dataset includes 285 cases and corresponding tumor region annotations. In this embodiment, the training set is divided into training (199 cases), validation (29 cases), and test (57 cases), and the Dice coefficient (DSC %) and 95% Hausdorff distance (HD95) are used as evaluation metrics. Furthermore, this embodiment also uses an online evaluation system to verify the performance of this embodiment across all modalities on the official validation set. (Reference) Figure 7C , Figure 7C This is a comparison result table provided in the embodiments of this application, showing the comparison results (DSC%, mean ± std) between the scheme of this application embodiment and the existing best method on the BraTS2018 dataset. Existing and missing modalities are represented by... and The asterisk (*) indicates that the p-value obtained by the Wilcoxon signed rank test is less than 0.05 compared with the result of the method in the embodiment of this application.

[0246] Figure 7C The comparison table presents the results of comparing the method of this application embodiment with four existing best-in-class brain MRI tumor segmentation methods in the absence of modality on the BraTS 2018 dataset. Figure 7C The comparison results table shows that the method proposed in this application has the best overall performance on the test set, achieving the best average score in all three tumor regions. Furthermore, the technique proposed in this application achieves the best results in most cases. Notably, the overall performance of the method proposed in this application surpasses that of two specialized methods (ACN, SMU-Net), which use a separate model for each missing modality, with approximately fifteen times the number of parameters compared to the method in this application. This application attributes this to two reasons: 1. Each model in the specialized methods can only model a one-to-one relationship between two missing modalities, while the mutual distillation method in this application implicitly models the relationship between all missing modalities; 2. The modalities and patch occlusion used during model training can be considered a form of data augmentation, allowing the network to be trained more thoroughly.

[0247] Furthermore, the method proposed in this application also outperforms the current best solution, RFNet, achieving higher average metrics across all three tumor regions. The method in this application employs a conventional encoder-decoder architecture, resulting in lower parameter count and computational complexity compared to RFNet. In summary, the method proposed in this application achieves optimal performance on multimodal brain MRI tumor segmentation tasks with missing modalities, and utilizes a more efficient and economical architecture.

[0248] refer to Figure 7D , Figure 7D This is a comparison results table provided by the embodiments of this application. It compares the results (mean ± std) of the scheme in the BraTS2018 data of this application under full modal conditions with the existing best method. "Challenge" indicates the winning scheme in the corresponding competition. NA: Unavailable. * indicates that the p-value obtained using the Wilcoxon signed rank test is less than 0.05 compared to the method in the embodiments of this application. †: Reproduced using the original author's code; ‡: Provided by the original author. Figure 7DIn the comparison results table, in addition to the four comparison schemes already exemplified above, two self-supervised methods are also included in the comparison: a general self-supervised method for medical image analysis (ModGen); and a self-supervised method for multimodal medical image data (CMJP). The results show that the technique proposed in this application's embodiments achieves the best results in a total of six cases across two metrics. Furthermore, the results of the winning scheme in the corresponding competition are also included in the table as a reference (Challenge). The results of this application's embodiments are comparable to the winning scheme in most cases, and even surpass it in some cases, while the competition scheme underwent extensive engineering adjustments for full-modal segmentation. These results demonstrate that the multimodal representation learned by the framework of this application's embodiments is not only robust to missing modalities but also achieves excellent results in the full-modal case.

[0249] To verify the effectiveness of the self-distillation method proposed in this application, the embodiments of this application compared the results of adding consistency loss to different locations in the network (including each layer of the encoder and the output) with and without adding consistency loss. Experimental results are referenced below. Figure 7B , Figure 7B This is a consistency loss analysis table provided in the embodiments of this application; the following conclusions can be drawn from it:

[0250] (1) The output of the first three network blocks (feature-1, feature-2, feature-3) with the consistency loss added is lower than that without the consistency loss. This is because shallow features are more susceptible to the differences between data of different modal combinations. Therefore, forcibly adding the consistency loss on top of them will affect the model’s feature extraction and reduce the performance.

[0251] (2) Adding consistency loss (feature-4) to the deepest layer of the network encoder improves the network performance because the deepest layer emphasizes the semantic structure of the image and is less affected by the differences between different modal combinations.

[0252] (3) The results of directly adding consistency loss to the output corresponding to different modal combinations are significantly reduced. This is because in the self-distillation scenario, directly adding consistency loss to the output can easily cause the results of modal combinations with more modalities to be affected by the results of modal combinations with fewer modalities, which are less effective, thus resulting in poor overall performance.

[0253] The following description continues to illustrate the exemplary structure of the image processing model training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2AAs shown, the software modules in the training device 455 of the image processing model stored in the memory 450 may include: a sample acquisition module 4551, configured to acquire multiple multimodal images for use as training samples, wherein the types of multimodal images include full-modal images and missing modality images; a pre-training module 4552, configured to, based on each multimodal image, call an initialized image processing model to perform a first training task of reconstructing a full-modal image, wherein, during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each multimodal image; pre-training Module 4552 is further configured to perform image completion processing on each first full-modal reconstructed image based on the full-modal image to obtain a full-modal template image; model adjustment module 4553 is configured to determine the consistency loss between the multimodal image pair and the full-modal template image, wherein the multimodal image pair includes any two multimodal images; model adjustment module 4553 is further configured to, based on each multimodal image, call the trained image processing model to perform a second training task of segmenting each multimodal image, wherein the consistency loss is used as a constraint condition for updating the parameters of the image processing model in the second training task.

[0254] In some embodiments, the pre-training module 4552 is configured to call the initialized image processing model to perform reconstruction processing based on each multimodal image to obtain a first full-modal reconstructed image corresponding to each multimodal image; determine a first mean squared error loss based on each first full-modal reconstructed image and the full-modal image; and perform backpropagation processing on the initialized image processing model based on the first mean squared error loss to obtain a trained image processing model.

[0255] In some embodiments, the pre-training module 4552 is configured to call an initialized image processing model based on each multimodal image to perform the following processing: encoding the multimodal image to obtain a first encoding vector of the multimodal image, wherein the first encoding vector is the encoding vector of the non-missing part in the multimodal image; performing missing part prediction processing based on the first encoding vector to obtain a first prediction vector of the missing part in the multimodal image; and integrating the first prediction vector and the first encoding vector to obtain a first full-modal reconstructed image.

[0256] In some embodiments, the initialized image processing model includes: a multimodal mask autoencoder and a regression network, wherein the multimodal mask autoencoder includes: an encoder layer and a decoder layer; the encoder layer is used to perform encoding processing; the decoder layer is used to perform missing part prediction processing; and the regression network is used to perform integration processing.

[0257] In some embodiments, the pre-training module 4552 is configured to substitute the first full-modal reconstructed image into a regularization function to obtain a first regularization term, and to use the minimum sum of the first mean squared error loss and the first regularization term as a first constraint condition; based on the first constraint condition and the first mean squared error loss, to update the parameters of the initialized image processing model to obtain the trained image processing model.

[0258] In some embodiments, the pre-training module 4552 is configured to perform the following processing for each multimodal image: determine the missing parts in the multimodal image, perform completion processing on the missing parts based on the first full-modal reconstructed image to obtain a first completed image; perform linear regression processing on the first completed image to obtain a linear regression result, and obtain a first mean squared error loss between the linear regression result and the full-modal image; obtain a target full-modal reconstructed image that minimizes the first mean squared error loss from each first full-modal reconstructed image, substitute the target full-modal reconstructed image into a regularization function to obtain a first regularization term; and use the sum of the first regularization term and the target full-modal reconstructed image as a full-modal template image.

[0259] In some embodiments, the model adjustment module 4553 is configured to perform the following processing for each multimodal image in a multimodal image pair: determine the missing part in the multimodal image, perform completion processing on the missing part based on the full-modal template image to obtain a second completed image; determine the second mean squared error loss between the two second completed images corresponding to the multimodal image pair, and use the second mean squared error loss as the consistency loss, wherein the two second completed images corresponding to the multimodal image pair include: the second completed image of the first multimodal image in the multimodal image pair, and the second completed image of the second multimodal image in the multimodal image pair.

[0260] In some embodiments, the model adjustment module 4553 is configured to call the trained image processing model to perform image segmentation processing based on each multimodal image, and obtain the predicted segmentation result corresponding to each multimodal image; determine the segmentation loss of the image processing model based on the predicted segmentation result and the actual segmentation result; and perform backpropagation processing on the image processing model based on the consistency loss and the segmentation loss to obtain a retrained image processing model, wherein the retrained image processing model is used to segment multimodal images with missing modalities.

[0261] In some embodiments, the model adjustment module 4553 is configured to call the trained image processing model based on each multimodal image to perform the following processing: encoding the multimodal image to obtain a second encoding vector of the multimodal image, wherein the second encoding vector is the encoding vector of the non-missing part in the multimodal image; obtaining the missing part in the multimodal image and extracting the third encoding vector corresponding to the missing part from the full-modal template image; performing missing part prediction processing based on the third encoding vector and the second encoding vector to obtain a second reconstructed full-modal image; and performing segmentation processing on the second reconstructed full-modal image to obtain the prediction segmentation results corresponding to the multimodal images respectively.

[0262] In some embodiments, the trained image processing model includes: a multimodal mask autoencoder and a segmentation network, wherein the multimodal mask autoencoder includes: an encoder layer and a decoder layer; the encoder layer is used to perform encoding processing and obtain a third encoded vector; the decoder layer is used to perform missing part prediction processing; and the segmentation network is used to perform segmentation processing.

[0263] In some embodiments, the model adjustment module 4553 is configured to extract feature maps of the second completed images from the second completed images corresponding to the two multimodal images in the multimodal image pair; determine the third mean squared error loss between the feature maps of the second completed images corresponding to the two multimodal images, and set the third mean squared error loss equal to the consistency loss as a second constraint; set the minimum sum of the consistency loss and the segmentation loss as a third constraint; and update the parameters of the image processing model based on the consistency loss and the segmentation loss until the second constraint and the third constraint are satisfied.

[0264] In some embodiments, the trained image processing model includes a multimodal mask autoencoder, which includes an encoder layer and a decoder layer, wherein the decoder layer includes multiple layers of feature extraction layers; the feature map is obtained by calling the feature extraction layers.

[0265] In some embodiments, the sample acquisition module 4551 is configured to acquire a full-modality image, wherein the full-modality image includes sub-images of multiple modalities; perform multiple different masking processes on the patches in the sub-images of the full-modality image to obtain multiple different missing modalities images, and use the multiple missing modalities images and the full-modality image as training samples.

[0266] In some embodiments, the initialized image processing model includes: a multimodal mask autoencoder; the multimodal mask autoencoder is used to perform mask processing for a full-modal image.

[0267] This application also provides an image processing apparatus. The following description continues to illustrate the exemplary structure of the image processing apparatus 456 provided in this application as a software module. In some embodiments, such as... Figure 2B As shown, the software modules stored in the image processing device 456 in the memory 450 may include: an image receiving module 4554, configured to receive a multimodal image to be processed; and an image processing module 4555, configured to call an image processing model based on the multimodal image to perform image segmentation processing and obtain the segmentation result corresponding to the multimodal image, wherein the image processing model is trained based on the image processing model training method provided in the embodiments of this application.

[0268] In some embodiments, the image processing module 4555 is configured to call an image processing model based on the multimodal image to perform the following processing: encode the multimodal image to obtain a fourth encoding vector of the multimodal image, wherein the fourth encoding vector is the encoding vector of the non-missing part in the multimodal image; obtain the missing part in the multimodal image and extract the fifth encoding vector corresponding to the missing part from the full-modal template image; perform missing part prediction processing based on the fourth encoding vector and the fifth encoding vector to obtain a third reconstructed full-modal image; and perform segmentation processing on the third reconstructed full-modal image to obtain the predicted segmentation result corresponding to the multimodal image.

[0269] In some embodiments, the image processing model includes: a multimodal mask autoencoder and a segmentation network, wherein the multimodal mask autoencoder includes: an encoder layer and a decoder layer; the encoder layer is used to perform encoding processing and obtain a fifth encoded vector; the decoder layer is used to perform missing part prediction processing; and the segmentation network is used to perform segmentation processing.

[0270] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the image processing model training method described in this application embodiment, or the image processing method described in this application embodiment.

[0271] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the training method for the image processing model provided in this application. For example, ... Figure 3A The training method for the image processing model is shown. Alternatively, it will cause the processor to execute the image processing method provided in the embodiments of this application, for example, such as... Figure 3A The image processing method shown.

[0272] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0273] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0274] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0275] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0276] In summary, through phased training of the image processing model in this embodiment, the image processing model is equipped with the ability to reconstruct missing parts in multimodal images and accurately segment specific regions within multimodal images. By using consistency loss as a deterministic constraint, the image processing model can maintain consistency between segmentation results when processing multimodal images with different missing modalities, thereby improving the accuracy of multimodal image segmentation.

[0277] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training an image processing model, characterized in that, The method comprises: acquiring a plurality of multi-modal images as training samples, wherein the types of the multi-modal images include full-modal images and missing-modal images; based on each of the multi-modal images, calling the initialized image processing model to perform a first training task of reconstructing the full-modal images, wherein during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multi-modal images respectively; determining a missing part of each of the multi-modal images, performing completion processing on each of the missing parts based on each of the first full-modal reconstructed images to obtain a first completed image; performing linear regression processing on the first completed image to obtain a linear regression result, and acquiring a first mean square error loss between the linear regression result and the full-modal image; from each of the first full-modal reconstructed images, acquiring a target full-modal reconstructed image that minimizes the first mean square error loss, substituting the target full-modal reconstructed image into a regularization function to obtain a first regularization term; and taking the sum of the first regularization term and the target full-modal reconstructed image as a full-modal template image; based on the full-modal template image, performing completion processing on each of the missing parts to obtain a second completed image of a first multi-modal image in a pair of multi-modal images and a second completed image of a second multi-modal image in the pair of multi-modal images, and taking a second mean square error loss between the second completed image of the first multi-modal image and the second completed image of the second multi-modal image as a consistency loss; based on each of the multi-modal images, calling the trained image processing model to perform a second training task of segmenting each of the multi-modal images, wherein the consistency loss is used as a constraint condition for updating parameters of the image processing model in the second training task.

2. The method of claim 1, wherein, The method comprises: acquiring a plurality of multi-modal images as training samples, wherein the types of the multi-modal images include full-modal images and missing-modal images; based on each of the multi-modal images, calling the initialized image processing model to perform a first training task of reconstructing the full-modal images, wherein during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multi-modal images respectively; determining a missing part of each of the multi-modal images, performing completion processing on each of the missing parts based on each of the first full-modal reconstructed images to obtain a first completed image; performing linear regression processing on the first completed image to obtain a linear regression result, and acquiring a first mean square error loss between the linear regression result and the full-modal image; from each of the first full-modal reconstructed images, acquiring a target full-modal reconstructed image that minimizes the first mean square error loss, substituting the target full-modal reconstructed image into a regularization function to obtain a first regularization term; and taking the sum of the first regularization term and the target full-modal reconstructed image as a full-modal template image; based on the full-modal template image, performing completion processing on each of the missing parts to obtain a second completed image of a first multi-modal image in a pair of multi-modal images and a second completed image of a second multi-modal image in the pair of multi-modal images, and taking a second mean square error loss between the second completed image of the first multi-modal image and the second completed image of the second multi-modal image as a consistency loss; based on each of the multi-modal images, calling the trained image processing model to perform a second training task of segmenting each of the multi-modal images, wherein the consistency loss is used as a constraint condition for updating parameters of the image processing model in the second training task. The method comprises: acquiring a plurality of multi-modal images as training samples, wherein the types of the multi-modal images include full-modal images and missing-modal images; based on each of the multi-modal images, calling the initialized image processing model to perform a first training task of reconstructing the full-modal images, wherein during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multi-modal images respectively; determining a missing part of each of the multi-modal images, performing completion processing on each of the missing parts based on each of the first full-modal reconstructed images to obtain a first completed image; performing linear regression processing on the first completed image to obtain a linear regression result, and acquiring a first mean square error loss between the linear regression result and the full-modal image; from each of the first full-modal reconstructed images, acquiring a target full-modal reconstructed image that minimizes the first mean square error loss, substituting the target full-modal reconstructed image into a regularization function to obtain a first regularization term; and taking the sum of the first regularization term and the target full-modal reconstructed image as a full-modal template image; based on the full-modal template image, performing completion processing on each of the missing parts to obtain a second completed image of a first multi-modal image in a pair of multi-modal images and a second completed image of a second multi-modal image in the pair of multi-modal images, and taking a second mean square error loss between the second completed image of the first multi-modal image and the second completed image of the second multi-modal image as a consistency loss; based on each of the multi-modal images, calling the trained image processing model to perform a second training task of segmenting each of the multi-modal images, wherein the consistency loss is used as a constraint condition for updating parameters of the image processing model in the second training task.

3. The method of claim 2, wherein: the method comprises: based on each of the multi-modal images, calling the initialized image processing model to perform a first training task of reconstructing the full-modal images, wherein during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multi-modal images respectively; based on each of the multi-modal images, calling the initialized image processing model to perform the following processing: performing encoding processing on the multi-modal image to obtain a first encoding vector of the multi-modal image, wherein the first encoding vector is an encoding vector of a non-missing part in the multi-modal image; performing missing part prediction processing based on the first encoding vector to obtain a first prediction vector of the missing part in the multi-modal image; performing integration processing on the first prediction vector and the first encoding vector to obtain a first full-modal reconstructed image.

4. The method of claim 3, wherein the initialized image processing model comprises a multi-modal mask autoencoder and a regression network, and the multi-modal mask autoencoder comprises an encoder layer and a decoder layer; the encoder layer is configured to perform the encoding processing; the decoder layer is configured to perform the missing part prediction processing; the regression network is configured to perform the integration processing.

5. The method of claim 2, wherein, the back propagation processing of the initialized image processing model based on the first mean square error loss comprises: substituting the first full-modal reconstructed image into a regularization function to obtain a first regularization term, and taking the sum of the first mean square error loss and the first regularization term as a first constraint condition; updating parameters of the initialized image processing model based on the first constraint condition and the first mean square error loss to obtain the trained image processing model.

6. The method of claim 1, wherein, the second training task of calling the trained image processing model to segment each multi-modal image based on each multi-modal image comprises: calling the trained image processing model to perform image segmentation processing based on each multi-modal image to obtain a prediction segmentation result corresponding to each multi-modal image respectively; determining a segmentation loss of the image processing model based on the prediction segmentation result and an actual segmentation result; performing back propagation processing on the image processing model based on the consistency loss and the segmentation loss to obtain a retrained image processing model, wherein the retrained image processing model is used to segment multi-modal images of a missing modality.

7. The method of claim 6, wherein, the image segmentation processing of calling the trained image processing model based on each multi-modal image comprises: calling the trained image processing model based on each multi-modal image to perform the following processing: performing encoding processing on the multi-modal image to obtain a second encoding vector of the multi-modal image, wherein the second encoding vector is an encoding vector of a non-missing part in the multi-modal image; obtaining a missing part in the multi-modal image, and extracting a third encoding vector corresponding to the missing part from the full-modal template image; performing missing part prediction processing based on the third encoding vector and the second encoding vector to obtain a second reconstructed full-modal image; performing segmentation processing on the second reconstructed full-modal image to obtain the prediction segmentation result corresponding to the multi-modal image respectively.

8. The method of claim 7, wherein the trained image processing model comprises a multi-modal mask autoencoder and a segmentation network, and the multi-modal mask autoencoder comprises an encoder layer and a decoder layer. The encoder layer is configured to perform the encoding process and obtain the third encoding vector; The decoder layer is configured to perform the missing part prediction process; The segmentation network is configured to perform the segmentation process.

9. The method of claim 6, wherein, The consistency loss and the segmentation loss are used to perform back propagation processing on the image processing model, including: extracting feature maps of the second completed images corresponding to the two multi-modal images in the multi-modal image pair respectively; determining a third mean square error loss between the feature maps of the second completed images corresponding to the two multi-modal images respectively, and setting the third mean square error loss equal to the consistency loss as a second constraint condition; setting the sum of the consistency loss and the segmentation loss to be minimum as a third constraint condition; updating parameters of the image processing model based on the consistency loss and the segmentation loss until the second constraint condition and the third constraint condition are satisfied.

10. The method of claim 1, wherein, The method includes: obtaining a full-modal image, wherein the full-modal image includes sub-images of multiple modalities; performing multiple different mask processing on patches in the sub-images of the full-modal image to obtain multiple different missing modal images; using the multiple missing modal images and the full-modal image as training samples.

11. An image processing method, characterized by, The method includes: receiving a multi-modal image to be processed; calling an image processing model based on the multi-modal image to perform image segmentation processing to obtain a segmentation result corresponding to the multi-modal image, wherein the image processing model is trained based on the training method of the image processing model according to any one of claims 1 to 10.

12. The method of claim 11, wherein, The method includes: calling an image processing model based on the multi-modal image to perform the following processing: performing encoding processing on the multi-modal image to obtain a fourth encoding vector of the multi-modal image, wherein the fourth encoding vector is an encoding vector of a non-missing part in the multi-modal image; obtaining a missing part in the multi-modal image and extracting a fifth encoding vector corresponding to the missing part from the full-modal template image; performing missing part prediction processing based on the fourth encoding vector and the fifth encoding vector to obtain a third reconstructed full-modal image; performing segmentation processing on the third reconstructed full-modal image to obtain a predicted segmentation result corresponding to the multi-modal image.

13. The method of claim 12, wherein the image processing model includes a multi-modal mask autoencoder and a segmentation network, wherein the multi-modal mask autoencoder includes an encoder layer and a decoder layer; the encoder layer is configured to perform the encoding process and obtain the fifth encoding vector; the decoder layer is configured to perform the missing part prediction process; the segmentation network is configured to perform the segmentation process.

14. An apparatus for training an image processing model, comprising: The device includes: The sample acquisition module is configured to acquire multiple multimodal images used as training samples, wherein the types of the multimodal images include full-modal images and missing modal images; The pre-training module is configured to call the initialized image processing model to perform a first training task to reconstruct the full-modal image based on each of the multimodal images, wherein, during the execution of the first training task, the image processing model outputs a first full-modal reconstructed image corresponding to each of the multimodal images; The pre-training module is further configured to: determine the missing parts of each of the multimodal images; perform image completion processing on each of the missing parts based on each of the first full-modal reconstructed images to obtain a first completed image; perform linear regression processing on the first completed image to obtain a linear regression result, and obtain a first mean squared error loss between the linear regression result and the full-modal image; obtain a target full-modal reconstructed image that minimizes the first mean squared error loss from each of the first full-modal reconstructed images; substitute the target full-modal reconstructed image into a regularization function to obtain a first regularization term; and use the sum of the first regularization term and the target full-modal reconstructed image as a full-modal template image. The model adjustment module is configured to perform completion processing on each missing part based on the full-modal template image to obtain the second completed image of the first multimodal image in the multimodal image pair and the second completed image of the second multimodal image in the multimodal image pair, and to use the second mean squared error loss between the second completed image of the first multimodal image and the second completed image of the second multimodal image as the consistency loss; The model adjustment module is further configured to call the trained image processing model to perform a second training task of segmenting each of the multimodal images based on each of the multimodal images, wherein the consistency loss is used as a constraint condition for updating the parameters of the image processing model in the second training task.

15. The apparatus according to claim 14, characterized in that, The pre-training module is further configured to call the initialized image processing model to perform reconstruction processing based on each of the multimodal images, to obtain a first full-modal reconstructed image corresponding to each of the multimodal images; and to determine a first mean squared error loss based on each of the first full-modal reconstructed images and the full-modal images. The image processing model is backpropagated based on the first mean square error loss to obtain the trained image processing model.

16. The apparatus according to claim 15, characterized in that, The pre-training module is further configured to call the initialized image processing model based on each of the multimodal images to perform the following processing: encoding the multimodal images to obtain a first encoding vector of the multimodal images, wherein the first encoding vector is the encoding vector of the non-missing part in the multimodal images; performing missing part prediction processing based on the first encoding vector to obtain a first prediction vector of the missing part in the multimodal images; and integrating the first prediction vector and the first encoding vector to obtain a first full-modal reconstructed image.

17. The apparatus according to claim 16, characterized in that, The initialized image processing model includes a multimodal mask autoencoder and a regression network. The multimodal mask autoencoder includes an encoder layer and a decoder layer. The encoder layer is used to perform the encoding process. The decoder layer is used to perform the missing part prediction process. The regression network is used to perform the integration process.

18. The apparatus according to claim 15, characterized in that, The pre-training module is further configured to substitute the first full-modal reconstructed image into a regularization function to obtain a first regularization term, and to use the minimum sum of the first mean squared error loss and the first regularization term as a first constraint condition; based on the first constraint condition and the first mean squared error loss, to update the parameters of the initialized image processing model to obtain the trained image processing model.

19. The apparatus according to claim 14, characterized in that, The model adjustment module is further configured to call the trained image processing model to perform image segmentation processing based on each of the multimodal images, and obtain the predicted segmentation result corresponding to each of the multimodal images; determine the segmentation loss of the image processing model based on the predicted segmentation result and the actual segmentation result; and perform backpropagation processing on the image processing model based on the consistency loss and the segmentation loss to obtain the retrained image processing model, wherein the retrained image processing model is used to segment multimodal images with missing modalities.

20. The apparatus according to claim 19, characterized in that, The model adjustment module is further configured to call the trained image processing model based on each of the multimodal images to perform the following processing: encoding the multimodal image to obtain a second encoding vector of the multimodal image, wherein the second encoding vector is the encoding vector of the non-missing part in the multimodal image; obtaining the missing part in the multimodal image and extracting the third encoding vector corresponding to the missing part from the full-modal template image; performing missing part prediction processing based on the third encoding vector and the second encoding vector to obtain a second reconstructed full-modal image; and performing segmentation processing on the second reconstructed full-modal image, wherein the multimodal image corresponds to the predicted segmentation result respectively.

21. The apparatus according to claim 20, characterized in that, The trained image processing model includes a multimodal mask autoencoder and a segmentation network. The multimodal mask autoencoder includes an encoder layer and a decoder layer. The encoder layer is used to perform the encoding process and obtain the third encoding vector. The decoder layer is used to perform the missing part prediction process. The segmentation network is used to perform the segmentation process.

22. The apparatus according to claim 19, characterized in that, The model adjustment module is further configured to extract feature maps of the second completed images from the second completed images corresponding to the two multimodal images in the multimodal image pair; determine a third mean squared error loss between the feature maps of the second completed images corresponding to the two multimodal images, and make the third mean squared error loss equal to the consistency loss as a second constraint; minimize the sum of the consistency loss and the segmentation loss as a third constraint; and update the parameters of the image processing model based on the consistency loss and the segmentation loss until the second constraint and the third constraint are satisfied.

23. The apparatus according to claim 14, characterized in that, The sample acquisition module is further configured to acquire a full-modality image, wherein the full-modality image includes sub-images of multiple modalities; perform multiple different masking processes on the patches in the sub-images of the full-modality image to obtain multiple different missing modalities images; and use the multiple missing modalities images and the full-modality image as training samples.

24. An image processing apparatus characterized by comprising: The image processing device includes: The image receiving module is configured to receive multimodal images to be processed. The image processing module is configured to call an image processing model to perform image segmentation processing based on the multimodal image to obtain the segmentation result corresponding to the multimodal image, wherein the image processing model is trained based on the training method of the image processing model according to any one of claims 1 to 10.

25. The apparatus according to claim 24, characterized in that, The image processing module is further configured to call an image processing model based on the multimodal image to perform the following processing: encode the multimodal image to obtain a fourth encoding vector of the multimodal image, wherein the fourth encoding vector is the encoding vector of the non-missing part in the multimodal image; obtain the missing part in the multimodal image and extract the fifth encoding vector corresponding to the missing part from the full-modal template image; perform missing part prediction processing based on the fourth encoding vector and the fifth encoding vector to obtain a third reconstructed full-modal image; and perform segmentation processing on the third reconstructed full-modal image to obtain the prediction segmentation result corresponding to the multimodal image.

26. The apparatus according to claim 25, characterized in that, The image processing model includes a multimodal mask autoencoder and a segmentation network. The multimodal mask autoencoder includes an encoder layer and a decoder layer. The encoder layer is used to perform the encoding process and obtain the fifth encoded vector. The decoder layer is used to perform the missing part prediction process. The segmentation network is used to perform the segmentation process.

27. An electronic device, comprising: The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the training method of the image processing model according to any one of claims 1 to 10, or the image processing method according to any one of claims 11 to 13.

28. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, perform steps comprising: When the computer-executable instructions are executed by the processor, they implement the training method of the image processing model according to any one of claims 1 to 10, or the image processing method according to any one of claims 11 to 13.

29. A computer program product comprising computer programs or computer executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, they implement the training method of the image processing model according to any one of claims 1 to 10, or the image processing method according to any one of claims 11 to 13.

Images