Image synthesis and segmentation methods, devices, terminals and media

Abstract

This invention provides an image synthesis and segmentation method, apparatus, terminal, and medium. It involves deforming target image data to obtain deformed image data; inputting the infant image and deformed image data into a pre-trained image synthesis and segmentation network, and outputting the corresponding synthesized segmentation result. This application utilizes deforming processing to complete the mapping from children's brain images to isointense infant brain images, providing more training data for isointense infant brain image segmentation. Simultaneously, the mapping from isointense infant brain images to children's brain images assists in the segmentation of infant brain images. Furthermore, the designed unified synthesis and segmentation network constrains the synthesis network to downstream segmentation networks, resulting in a more accurate synthesized image for isointense infant brain images (MRI).

Description

Technical Field

[0001] This invention relates to the field of medical imaging, and in particular to an image synthesis and segmentation method, apparatus, terminal, and medium. Background Technology

[0002] Magnetic resonance imaging (MRI) is a very important medical imaging technology, characterized by low radiation, high sensitivity to soft tissue imaging, multi-planar imaging, and multiple presentation modes. In order to measure brain growth patterns and morphological changes in neurodevelopmental disorders or to isolate lesion sites, MRI images need to be segmented. Image segmentation, as a general digital image analysis technique, plays a vital role in the medical field.

[0003] However, at around 6 months of age (the isointense period), the contrast between white and gray matter in an infant's brain is extremely low, making it difficult to automatically segment MRI images. In existing technologies, traditional brain tissue segmentation methods are usually based on statistical analysis and partial differential equations, which often rely on prior knowledge of the adult brain and are not suitable for infants at the isointense period. Deep learning segmentation algorithms require a large number of training samples, but the low contrast between white and gray matter in an infant's brain at around 6 months of age (the isointense period) makes it difficult to manually annotate tissue segments to obtain a large number of training samples for deep learning, resulting in unsatisfactory performance. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an image synthesis and segmentation method, device, terminal and medium to solve the problem in the prior art that the contrast between the white matter and gray matter of an infant's brain is extremely low at around 6 months of age (isothermal period), making it difficult to automatically segment MRI images of the infant.

[0005] To achieve the above and other related objectives, the present invention provides an image synthesis and segmentation method, characterized in that the method includes: acquiring target image data, the target image data including: an infant image and / or a child image; performing deformation processing on the target image data to obtain deformed image data; wherein, the deformed image data includes: a deformed infant image corresponding to the infant image and / or a deformed child image corresponding to the child image; inputting the infant image and the deformed image data into a pre-trained image synthesis and segmentation network, and outputting a corresponding synthesis and segmentation result.

[0006] In one embodiment of the present invention, the image synthesis and segmentation network includes: a synthesis network, configured to output a synthesized child image corresponding to the distorted baby image and / or a synthesized baby image corresponding to the distorted child image based on the input distorted baby image and / or distorted child image; and a segmentation network, connected to the synthesis network, configured to output a baby image synthesis and segmentation result corresponding to the baby image or distorted baby image and / or a child image synthesis and segmentation result corresponding to the distorted child image based on the input baby image or distorted baby image and the corresponding synthesized child image, and / or the distorted child image and the corresponding synthesized baby image.

[0007] In one embodiment of the present invention, the synthesis network includes: a first synthesis sub-network for inputting the deformed child image and outputting the synthesized baby image; a second synthesis sub-network for inputting the baby image or deformed baby image and outputting the synthesized child image; and an output module connected to the first and second synthesis sub-networks for receiving the synthesized baby image and / or the synthesized child image; wherein the first synthesis network is trained using a baby image training set; wherein the baby image training set includes: multiple baby image or deformed baby image samples and deformed child images corresponding to each sample; the second synthesis network is trained using a child image training set; wherein the child image training set includes: multiple deformed child image samples and baby image or deformed baby image corresponding to each sample.

[0008] In one embodiment of the present invention, the first synthesis sub-network includes: a first generator, a first discriminator, and a first registration module; the second synthesis sub-network includes: a second generator, a second discriminator, and a second registration module; wherein, the first generator is connected to the first discriminator, the first discriminator is connected to the first registration module, the second generator is connected to the second discriminator, and the second discriminator is connected to the second registration module.

[0009] In one embodiment of the present invention, the segmentation network includes:

[0010] A first feature extraction module, connected to the output module, is used to input the deformed child image and / or the synthesized child image, extract its features, and output the corresponding synthesized child image features extracted from the synthesized child image and / or the deformed child image features corresponding to the deformed child image; a second feature extraction module, connected to the output module, is used to input the infant image or deformed infant image and / or the synthesized infant image, and output the corresponding deformed infant image features extracted from the infant image or deformed infant image and / or the corresponding synthesized infant image features; a fusion module, connected to the first feature extraction module and the second feature extraction module, is used to input the synthesized child image features and the infant image or deformed infant image features for fusion, obtain the infant image synthesis segmentation result corresponding to the infant image or deformed infant image, and / or input the synthesized infant image features and deformed child image features for fusion, and output the corresponding child image synthesis segmentation result corresponding to the deformed child image.

[0011] In one embodiment of the present invention, the deformation processing of the image data includes: performing deformation registration on the infant image based on the affine child standard image obtained by affine registration of the child standard image to obtain the deformed infant image; and / or performing deformation registration on the child image based on the infant standard image to obtain the deformed child image.

[0012] In one embodiment of the present invention, the synthetic network updates its parameters based on a perceptual consistency loss.

[0013] To achieve the above and other related objectives, the present invention provides a method for achieving the above and other related objectives.

[0014] To achieve the above and other related objectives, the present invention provides an image synthesis and segmentation apparatus, comprising: an acquisition device for acquiring image data to be synthesized, the image data including an infant image and / or a child image; a processing device for performing deformation processing on the image data to obtain deformed image data, the deformed image data including a deformed infant image and / or a deformed child image; and a synthesis and segmentation device for inputting the infant image and the deformed image data into a pre-trained image synthesis and segmentation network, and outputting a synthesized segmented image.

[0015] To achieve the above and other related objectives, the present invention provides a computer-readable storage medium storing computer instructions, which, when executed, perform the method described above.

[0016] To achieve the above and other related objectives, the present invention provides an electronic terminal, the device comprising: a memory and a processor; the memory being used to store computer instructions; and the processor executing the computer instructions to implement the method described above.

[0017] As described above, the image synthesis and segmentation method, apparatus, terminal, and medium of the present invention have the following beneficial effects: Unlike existing traditional brain tissue segmentation methods and deep learning segmentation algorithms, the image synthesis and segmentation method, apparatus, terminal, and medium provided in this application introduce a synthesis framework and a segmentation framework into the field of isointense infant image (MRI) segmentation. Compared with the prior art, this application completes the mapping from children's brain images to isointense infant brain images through deformation processing, providing more training data for isointense infant brain image segmentation. Simultaneously, the mapping from isointense infant brain images to children's brain images assists in the segmentation of infant brain images; and through the designed unified synthesis and segmentation network, the synthesis network is constrained by the downstream segmentation network to form a more accurate synthesized image for isointense infant brain image (MRI) segmentation. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart of an image synthesis and segmentation method according to an embodiment of the present invention.

[0019] Figure 2 shows a schematic diagram of the deformation data acquisition process in one embodiment of the present invention.

[0020] Figure 3 The diagram shown is a schematic diagram of an image synthesis and segmentation network structure according to an embodiment of the present invention.

[0021] Figure 4 The diagram shows a synthetic network structure in one embodiment of the present invention.

[0022] Figure 5 The diagram shown is a structural schematic of an image synthesis and segmentation method according to an embodiment of the present invention.

[0023] Figure 6 The diagram shown is a schematic of the image synthesis and segmentation network structure framework in one embodiment of the present invention.

[0024] Figure 7 The diagram shown is a structural schematic of an image synthesis and segmentation apparatus according to an embodiment of the present invention.

[0025] Figure 8 The diagram shown is a schematic representation of the image synthesis and segmentation terminal structure in one embodiment of the present invention. Detailed Implementation

[0026] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0027] Please see Figures 1 to 8 It should be understood that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes to aid those skilled in the art and to facilitate understanding and reading. They are not intended to limit the scope of the invention and therefore have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effectiveness and purpose of the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity and not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention's implementation.

[0028] like Figure 1 The diagram illustrates a flowchart of an image synthesis and segmentation method according to an embodiment of the present invention. The base image synthesis and segmentation method in this embodiment mainly includes the following steps:

[0029] To clarify each step, the present invention is described as follows: The method provided by the present invention can not only perform the main function of automatically and accurately segmenting the infant image, but also automatically and accurately segment the child image. Furthermore, it can simultaneously perform automatic and accurate segmentation of both the infant image and the child image, that is, simultaneously inputting the infant image and the child image and outputting the corresponding segmentation results of the infant image and the corresponding segmentation results of the child image.

[0030] Step S11: Obtain target image data, which includes: infant images and / or child images;

[0031] Specifically, both the infant image and the child image are magnetic resonance imaging (MRI) images. The target image data refers to the infant image and / or the child image to be segmented. In this embodiment, the target image data can be stored in a setting device, which can be network-connected to the image synthesis and segmentation device to obtain the target image data. The network connection can be wireless or wired. If the setting device and the synthetic lethal gene partner recommendation device are connected, the mobile network standard can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, 5G, etc.

[0032] Step S12: Perform deformation processing on the target image data to obtain deformed image data; wherein, the deformed image data includes: a deformed baby image corresponding to the baby image and / or a deformed child image corresponding to the child image;

[0033] Specifically, this application assists in infant image segmentation by mapping images of infants around 6 months old (isothermal period) to standard children's images. This mapping requires deforming the target image to eliminate differences in size and shape during the mapping process, as shown in Figure 2. For example, when the acquired image is an infant image to be segmented, to complete the mapping from the infant image to the standard children's image, the image needs to be deformed to obtain a deformed infant image with a similar size and shape but a different tissue intensity distribution. The contrast between white and gray matter in the deformed infant image is higher than that in the standard children's image. The infant image to be segmented; alternatively, the segmentation of the child's brain image can be assisted by mapping the child image to a standard infant image. For example, when the acquired image is a child image to be segmented, in order to complete the mapping from the child's brain image to the selected standard infant image, the image to be segmented needs to be deformed to obtain a deformed child image that is similar in size and shape to the selected standard infant image but has different tissue intensity. The deformed child image retains the original contrast between white and gray matter while being similar in size and shape to the standard infant image. When both the image to be segmented and the infant image to be segmented are acquired simultaneously, the principle is the same as the methods described above for their respective corresponding aspects.

[0034] In some implementations of this embodiment, the deformation processing of the image data includes:

[0035] Step S121: Based on the affine child standard image obtained by affine registration of the child standard image, perform deformable registration on the infant image to obtain the deformed infant image; and / or based on the infant standard image, perform deformable registration on the child image to obtain the deformed child image;

[0036] Specifically, when the obtained image is the infant image to be segmented, multiple standard child images are selected as templates. First, each standard child image is aligned with the infant image to be segmented through affine registration, resulting in affine child standard images of the same size and shape as the infant image to be segmented, corresponding to each of the standard child images. Then, the infant image to be segmented is registered to each affine child standard image through deformation registration, thereby obtaining the deformed infant image corresponding to the infant image to be segmented. The deformed infant image has a similar size and shape to each affine child image, but a different tissue intensity distribution. The contrast between white and gray matter in the obtained deformed infant image is higher than that in the infant image to be segmented. When the obtained image is a child image to be segmented, multiple standard infant images are selected as templates. The infant image to be segmented is deformed and registered to each of the standard infant images. The deformation field is applied to the child image to be segmented to obtain a deformed child image corresponding to the infant image to be segmented. The deformed child image is similar in size and shape to each of the standard infant images, but the tissue intensity distribution is different. The obtained deformed child image retains the original contrast between white matter and gray matter while being similar in size and shape to the standard infant image. Alternatively, the infant image to be segmented and the child image to be segmented can be deformed simultaneously. The deformation process is the same as the above-described process for their respective counterparts, and the deformed infant image and the deformed child image are output respectively.

[0037] Step S13: Input the infant image and deformed image data into a pre-trained image synthesis and segmentation network, and output the corresponding synthesis and segmentation results;

[0038] Specifically, if the infant image or deformed infant image is obtained, the corresponding synthesized child image is obtained through the image synthesis and segmentation network, and the corresponding segmentation is output. If the deformed child image is obtained, the corresponding synthesized infant image is obtained through the image synthesis and segmentation network, and the corresponding segmentation is output. In the image synthesis and segmentation network, in order to avoid the problem that the synthesized image and the subsequent segmentation task cannot be well coordinated, this application creatively performs the cross-domain image synthesis and segmentation tasks together. The subsequent segmentation network, in turn, provides task constraints for the synthesis network, so that the synthesized image better retains the features related to segmentation and increases the accuracy of the segmentation result.

[0039] In some implementations of this embodiment, the image synthesis and segmentation network includes:

[0040] like Figure 3 As shown, the synthesis network 31 is used to output a synthesized child image corresponding to the input baby image or deformed baby image and / or deformed child image, based on the input baby image or deformed baby image and / or deformed child image. Specifically, the synthesis network realizes the conversion between the baby image or deformed baby image and the synthesized child image, so that the baby image or deformed baby image is synthesized into a synthesized child image similar to the child image to assist in segmentation, and / or the conversion between the deformed child image and the synthesized baby image, so that the deformed child image is synthesized into a synthesized baby image similar to the baby image to assist in segmentation. The synthesis network overcomes the incomplete alignment between the spatial positions of the deformed images, that is, the alignment between the spatial positions of the synthesized baby image and the deformed baby image, and the alignment between the spatial positions of the synthesized child image and the deformed child image.

[0041] In some implementations of this embodiment, the synthesis network 31 includes:

[0042] like Figure 4 As shown, a first synthetic sub-network 301 is used to input the deformed child image and output the synthetic baby image; a second synthetic sub-network 311 is used to input the baby image or deformed baby image and output the synthetic child image; an output module 321 is connected to the first synthetic sub-network 301 and the second synthetic sub-network 311 and is used to receive the synthetic baby image and / or the synthetic child image; wherein, the first synthetic sub-network 301 is trained from a baby image training set; wherein, the baby image training set includes: multiple baby image or deformed baby image samples and deformed child images corresponding to each sample; the second synthetic sub-network 311 is trained from a child image training set; wherein, the child image training set includes: multiple deformed child image samples and baby image or deformed baby image corresponding to each sample.

[0043] Specifically, the first synthesis sub-network 301 is mainly used to provide the required synthetic child image for the segmentation result of the segmentation network to generate the baby image, the second synthesis sub-network 311 is mainly used to provide the required synthetic baby image for the segmentation result of the segmentation network to generate the child image, and the output module 321 is also used to output the deformed child image and the deformed baby image.

[0044] In some implementations of this embodiment, the first synthetic sub-network includes:

[0045] The first synthetic sub-network includes: a first generator, a first discriminator, and a first registration module; wherein, the first generator is connected to the first discriminator, and the first discriminator is connected to the first registration module; specifically, the first discriminator includes a first discriminator A and a first discriminator B, and the second discriminator includes a second discriminator A and a second discriminator B; the first generation module takes the deformed child image as input and outputs the synthetic infant image to be registered, during which the realism of the generated synthetic infant image to be registered is enhanced by the loss function of the first generation module; the first discriminator module is connected to the first generation module and takes the estimated probability of the synthetic infant image to be registered as input, that is, estimates whether the input synthetic infant image to be registered is the desired synthetic infant image. The probability of a deformed baby image is given, and the synthesized baby image to be registered is output. During this process, the probability of the synthesized baby image to be registered being the deformed baby image is increased by the loss function of the discriminator. The first registration module, connected to the first discrimination module, takes the synthesized baby image to be registered as input and registers it with the deformed baby image, and outputs the synthesized baby image. During this process, the synthesized baby image is corrected by the registration loss and the smoothness of the deformation field is constrained to minimize the gradient of the deformation field. This can help to better eliminate the influence of incomplete spatial alignment. The first registration module of this application realizes the alignment between the spatial positions of the synthesized baby image and the deformed baby image, thereby improving the accuracy of the baby image segmentation result.

[0046] The second synthetic sub-network 311 includes:

[0047] The system comprises a second generator, a second discriminator, and a second registration module; wherein the second generator is connected to the second discriminator, and the second discriminator is connected to the second registration module; specifically, the second generation module takes the infant image or deformed infant image as input and outputs the synthesized child image to be registered; the second discriminator module, connected to the second generation module, takes the synthesized child image to be registered as input, identifies the category, and outputs the infant image to be registered; the second registration module, connected to the second discriminator module, takes the synthesized child image to be registered as input and registers it with the infant image or deformed infant image, and outputs the synthesized child image; the first registration module of this application realizes the alignment of the spatial positions between the synthesized child image and the deformed child image, improving the accuracy of the child image segmentation result; the functions and effects of training the second sub-network and training the first synthesized sub-network are the same, so they will not be described in detail again;

[0048] In some implementations of this embodiment, the image synthesis and segmentation network includes:

[0049] like Figure 3As shown, the segmentation network 32, connected to the synthesis network 31, is used to output the baby image synthesis segmentation result corresponding to the baby image or deformed baby image and / or the child image synthesis segmentation result corresponding to the deformed child image, based on the input baby image or deformed baby image and the corresponding synthesized child image, and / or the deformed child image and the corresponding synthesized baby image; specifically, the segmentation network 32 adopts the V-Net architecture.

[0050] In some implementations of this embodiment, the segmentation network 32 includes:

[0051] A first feature extraction module, connected to the output module, is used to input the deformed child image and / or the synthetic child image, extract its features, and output the synthetic child image features extracted from the synthetic child image and / or the deformed child image features corresponding to the deformed child image; a second feature extraction module, connected to the output module, is used to input the infant image or deformed infant image and / or the synthetic infant image, and output the infant image or deformed infant image features corresponding to the infant image or deformed infant image and / or the synthetic infant image features corresponding to the synthetic infant image; a fusion module, connected to the first feature extraction module and the second feature extraction module, is used to input the synthetic child image features and the infant image or deformed infant image features for fusion, obtain the infant image synthesis segmentation result corresponding to the infant image or deformed infant image, and / or input the synthetic infant image features and deformed child image features for fusion, and output the child image synthesis segmentation result corresponding to the deformed child image.

[0052] Specifically, the first feature extraction module and the second feature extraction module are two parallel feature extraction modules. When it is necessary to obtain the segmentation result of the infant image to be segmented, the first feature extraction module learns the features of the synthesized child image, and the second feature extraction module learns the features of the deformed infant image. The fusion module fuses the learned features of the synthesized child image with the features of the infant image or the deformed infant image to obtain the synthetic segmentation result of the infant image corresponding to the deformed infant image, that is, the segmentation result of the infant image to be segmented; when it is necessary to obtain the segmentation result of the child image to be segmented, the first feature extraction module learns the features of the deformed child image. The first feature extraction module learns the features of the synthetic baby image, and the second feature extraction module is constrained by the perceptual consistency loss function to ensure feature consistency. The fusion module fuses the learned features of the deformed child and the features of the synthetic baby image and calculates the similarity of the two features through the segmentation loss function to improve the accuracy of the global segmentation result, thereby obtaining the synthetic segmentation result of the child image corresponding to the deformed child image, that is, the segmentation result of the child image to be segmented; when the segmentation result of the child image to be segmented and the segmentation result of the baby image to be segmented are obtained simultaneously, the principle is the same as the above-described methods respectively.

[0053] In some implementations of this embodiment, the synthesis network 32 is constrained by the perceptual consistency loss function to update its parameters. Specifically, to avoid the problem that the synthesized image generated by the synthesis network cannot be well coordinated with the subsequent segmentation task, the cross-domain image synthesis and segmentation tasks are performed together. The subsequent segmentation network provides task constraints for the synthesis network, namely the perceptual consistency loss function. First, the adversarial loss function, the cyclic consistency loss function, and the registration loss function are used to train the synthesis network. Then, the segmentation loss function and the perceptual consistency loss function are used to train the segmentation network. The perceptual consistency loss function is used to constrain the feature consistency of the synthesis network and update the parameters of the synthesis network. The feature consistency of the segmentation network is used to ensure that the features related to segmentation are retained in the synthesized image.

[0054] To better illustrate the above image synthesis and segmentation method, the present invention provides the following specific embodiments.

[0055] Example 1: An image synthesis and segmentation method. Figure 5 The diagram shows the structure of an image synthesis and segmentation method. This method mainly consists of three parts: deformable data augmentation (DA), a registration-synthesis network for infant and child images (RGAN), and a brain segmentation network (SegNet).

[0056] The deformable data augmentation (DA) method first uses deformable registration to register infant and child images, resulting in deformed child and infant images similar in size and shape to the infant images. The specific process is as follows: Using a standard child image as a template, deformable registration is performed on the infant image and its precision standard (GT): First, each child image is aligned with the infant image through affine registration, resulting in an affine child standard image with the same size and shape as the infant image. Then, deformable registration is used to register the infant image and its corresponding GT onto the affine child standard image. The resulting deformed infant image has a similar size and shape to the aforementioned affine child image, but with a different tissue intensity distribution. Alternatively, deformable registration is directly performed on each child image and its GT to each infant image, applying a deformation field to the child image and its GT. The resulting deformed child image is similar in size and shape to the infant image, but with a different tissue intensity distribution. The precision standard (GT) is the result of manual segmentation.

[0057] like Figure 6 The unified image synthesis and segmentation network framework shown addresses the issue of poor coordination between synthesized data and subsequent segmentation tasks. It unifies the infant and child image registration-synthesis network (RGAN) and the brain segmentation network (SegNet) into a unified image synthesis and segmentation network (TRSynNet). TRSynNet is trained using deformable data augmentation (DA) images of deformed children and deformed infants, along with their corresponding ground truth (GT). RGAN consists of two bidirectional generators (GA and GB), their corresponding discriminators (DA and DB), and a registration network (RA and RB). SegNet comprises two parallel feature extraction modules (FA and FB) and one fusion module (Fu). The specific training process is as follows: The generator and its corresponding discriminator are trained using adversarial loss La, the registration network is trained using registration loss Lr, and RGAN is trained using cycle consistency loss Lc. The main purpose of cycle consistency loss is to verify that the original content of the generated image is preserved during image transformation, meaning the transformed image can be transformed back to the original image. Then, SegNet is trained using segmentation loss Ld. Finally, to avoid problems with the synthesized image not coordinating well with subsequent segmentation tasks, RGAN is constrained again using perceptual consistency loss Ls, updating the RGAN parameters and utilizing the feature consistency of the segmentation network to ensure that segmentation-related features are preserved in the synthesized image.

[0058] like Figure 7 The diagram shown illustrates the structure of an image synthesis and segmentation apparatus according to an embodiment of the present invention. In this embodiment, as... Figure 7The diagram illustrates the structure of an image synthesis and segmentation apparatus according to an embodiment of the present invention. In this embodiment, the image synthesis and segmentation apparatus 700 includes:

[0059] Data acquisition module 701 is used to acquire target image data, the target image data including: baby images and / or child images;

[0060] The data processing module 702 is used to perform deformation processing on the target image data to obtain deformed image data; wherein, the deformed image data includes: a deformed baby image corresponding to the baby image and / or a deformed child image corresponding to the child image;

[0061] The synthesis segmentation module 703 inputs the infant image and deformed image data into a pre-trained image synthesis segmentation network and outputs the corresponding synthesis segmentation results.

[0062] The image synthesis and segmentation method provided in this embodiment of the invention can be implemented on the terminal side or the server side. For the hardware structure of the image synthesis and segmentation terminal, please refer to [link to relevant documentation]. Figure 8 This is a schematic diagram of an optional hardware structure of an image synthesis and segmentation terminal 800 provided in an embodiment of the present invention. The terminal 800 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The image synthesis and segmentation terminal 800 includes: at least one processor 801, a memory 802, at least one network interface 804, and a user interface 806. The various components in the device are coupled together through a bus system 808. It is understood that the bus system 808 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 808 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general will label all buses as bus systems.

[0063] The user interface 806 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0064] It is understood that memory 802 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0065] In this embodiment of the invention, the memory 802 is used to store various types of data to support the operation of the image synthesis and segmentation terminal 800. Examples of this data include: any executable program for operation on the image synthesis and segmentation terminal 800, such as the operating system 8021 and application programs 8022; the operating system 8021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 8022 may include various applications, such as media players, browsers, etc., for implementing various application services. The image synthesis and segmentation method provided in this embodiment of the invention can be included in the application program 8022.

[0066] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 801. Processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 801 or by instructions in software form. The processor 801 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 801 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 801 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0067] In an exemplary embodiment, the image synthesis and segmentation terminal 800 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0068] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0069] In summary, this invention provides an image synthesis and segmentation method, apparatus, terminal, and medium. It acquires target image data, including infant images and / or child images; performs deformation processing on the target image data to obtain deformed image data; wherein the deformed image data includes deformed infant images corresponding to the infant images and / or deformed child images corresponding to the child images; inputs the infant images and deformed image data into a pre-trained image synthesis and segmentation network, and outputs corresponding synthesized segmentation results. This application utilizes deformation processing to complete the mapping from child brain images to isointense infant brain images, providing more training data for isointense infant brain image segmentation. Simultaneously, the mapping from isointense infant brain images to child brain images assists in the segmentation of infant brain images; and through a designed unified synthesis and segmentation network, the synthesis network is constrained by the downstream segmentation network to form a more accurate synthesized image for isointense infant brain images (MRI).

[0070] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An image synthesis and segmentation method, characterized in that, The method includes: Acquire target image data, which includes images of infants; The infant image is deformed to obtain a deformed infant image corresponding to the infant image. Multiple standard child images are selected as templates, and each standard child image is aligned with the infant image to be segmented using affine registration. This results in affine child standard images that are the same size and shape as the infant image to be segmented and correspond to each of the standard child images. Then, the infant image to be segmented is registered to each of the affine child standard images using deformed registration, thus obtaining the deformed infant image corresponding to the infant image to be segmented. The deformed infant image has a similar size and shape to each of the affine child standard images, but a different tissue intensity distribution. The contrast between the white matter and gray matter in the obtained deformed infant image is higher than that in the infant image to be segmented. The infant image and the deformed infant image are input into a pre-trained image synthesis and segmentation network, which outputs a synthesized segmentation result corresponding to the infant image; wherein, the image synthesis and segmentation network includes: A synthesis network is used to output a synthesized child image corresponding to the input distorted baby image; A segmentation network, connected to the synthesis network, is used to output a baby image synthesis segmentation result corresponding to the baby image based on the input deformed baby image and the synthesized child image output by the synthesis network.

2. An image synthesis and segmentation method, characterized in that, The method includes: Acquire target image data, which includes images of children; The child image is deformed to obtain a deformed child image corresponding to the child image. Multiple standard infant images are selected as templates, and the child image to be segmented is deformed and registered to each of the standard infant images. The deformation field is applied to the child image to be segmented to obtain a deformed child image corresponding to the child image to be segmented. The deformed child image is similar in size and shape to each of the standard infant images, but the tissue intensity distribution is different. The obtained deformed child image retains the original contrast between white and gray matter while being similar in size and shape to the standard infant images. The child image and the deformed child image are input into a pre-trained image synthesis and segmentation network, which outputs a synthesized segmentation result corresponding to the child image; wherein, the image synthesis and segmentation network includes: A synthesis network is used to output a synthesized baby image corresponding to the input deformed child image; A segmentation network, connected to the synthesis network, is used to output a child image synthesis segmentation result corresponding to the child image based on the input deformed child image and the synthesized baby image output by the synthesis network.

3. An image synthesis and segmentation method, characterized in that, The method includes: Acquire target image data, which includes images of infants and children; The infant image is deformed to obtain a deformed infant image corresponding to the infant image. Multiple standard child images are selected as templates, and each standard child image is aligned with the infant image to be segmented using affine registration. This yields affine child standard images that are the same size and shape as the infant image to be segmented and correspond to each of the standard child images. Then, the infant image to be segmented is registered onto each of the affine child standard images using deformed registration, thus obtaining the deformed infant image corresponding to the infant image to be segmented. The deformed infant image has a similar size and shape to each of the affine child standard images, but a different tissue intensity distribution. The contrast between the white matter and gray matter in the obtained deformed infant image is higher than that in the infant image to be segmented. The child image is deformed to obtain a deformed child image corresponding to the child image; multiple standard infant images are selected as templates, and the child image to be segmented is deformed and registered to each of the standard infant images. The deformation field is applied to the child image to be segmented to obtain a deformed child image corresponding to the child image to be segmented; the deformed child image is similar in size and shape to each of the standard infant images, but the tissue intensity distribution is different. The obtained deformed child image retains the original contrast between white matter and gray matter while being similar in size and shape to the standard infant image; The infant image, the deformed infant image, the child image, and the deformed child image are input into a pre-trained image synthesis and segmentation network, which outputs synthesized segmentation results corresponding to the infant image and the child image; wherein, the image synthesis and segmentation network includes: A synthesis network is used to output a synthesized child image corresponding to the distorted baby image and a synthesized baby image corresponding to the distorted child image, based on the input distorted baby image and distorted child image; A segmentation network, connected to the synthesis network, is used to output a baby image synthesis segmentation result corresponding to the baby image and a child image synthesis segmentation result corresponding to the child image, based on the input deformed baby image and its corresponding synthesized child image, and the deformed child image and its corresponding synthesized baby image.

4. The image synthesis and segmentation method according to claim 1, characterized in that, The synthesis network includes: The second synthesis sub-network is used to take the deformed baby image as input and output a synthesized child image corresponding to the deformed baby image; wherein, the second synthesis sub-network is trained from a child image training set; wherein, the child image training set includes: multiple deformed child image samples and deformed baby images corresponding to each sample; The output module, connected to the second synthesis sub-network, is used to receive the synthesized child image.

5. The image synthesis and segmentation method according to claim 4, characterized in that, The second synthetic subnetwork includes: The second generator, the second discriminator, and the second registration module; The second generator is connected to the second discriminator, and the second discriminator is connected to the second registration module.

6. The image synthesis and segmentation method according to claim 4, characterized in that, The segmentation network includes: The first feature extraction module is connected to the output module and is used to input the synthetic child image and extract its corresponding features, and output the synthetic child image features corresponding to the synthetic child image. The second feature extraction module is connected to the output module and is used to input the deformed baby image and extract its corresponding features, and output the deformed baby image features corresponding to the deformed baby image. The fusion module connects the first feature extraction module and the second feature extraction module. It inputs the features of the synthesized child image and the features of the deformed baby image and fuses them to obtain the baby image synthesis and segmentation result corresponding to the deformed baby image.

7. The image synthesis and segmentation method according to claim 1 or 2, characterized in that, The synthetic network updates its parameters based on a perceptual consistency loss constraint.

8. The image synthesis and segmentation method according to claim 2, characterized in that, The synthesis network includes: A first synthetic sub-network is used to output a synthetic baby image corresponding to the distorted child image based on the input distorted child image; wherein, the first synthetic sub-network is trained from a baby image training set; wherein, the baby image training set includes: multiple distorted baby image samples and distorted child images corresponding to each sample; The output module, connected to the first synthesis sub-network, is used to receive the synthesized baby image.

9. The image synthesis and segmentation method according to claim 8, characterized in that, The first synthetic sub-network includes: The first generator, the first discriminator, and the first registration module; The first generator is connected to the first discriminator, and the first discriminator is connected to the first registration module.

10. The image synthesis and segmentation method according to claim 8, characterized in that, The segmentation network includes: The first feature extraction module is connected to the output module and is used to input the deformed child image and extract its corresponding features, and output the deformed child image features corresponding to the deformed child image. The second feature extraction module is connected to the output module and is used to input the synthetic baby image and extract its corresponding features, and output the synthetic baby image features corresponding to the synthetic baby image. The fusion module connects the first feature extraction module and the second feature extraction module. It inputs the features of the synthesized infant image and the features of the deformed child image, fuses them, and outputs the child image synthesis and segmentation result corresponding to the deformed child image.

11. An image synthesis and segmentation apparatus, characterized in that, The device includes: Acquisition device for acquiring target image data, the target image data including an image of an infant; A processing device is used to deform the infant image to obtain a deformed infant image corresponding to the infant image. The device selects multiple standard child images as templates, aligns each standard child image with the infant image to be segmented using affine registration, and obtains affine child standard images that are the same size and shape as the infant image to be segmented and correspond to each of the standard child images. Then, the infant image to be segmented is registered to each of the affine child standard images using deformable registration, thereby obtaining the deformed infant image corresponding to the infant image to be segmented. The deformed infant image has a similar size and shape to each of the affine child standard images, but a different tissue intensity distribution. The contrast between the white matter and gray matter in the obtained deformed infant image is higher than that in the infant image to be segmented. A synthetic segmentation device is used to input the infant image and the deformed infant image into a pre-trained image synthetic segmentation network and output a synthetic segmentation result; wherein, the image synthetic segmentation network includes: A synthesis network is used to output a synthesized child image corresponding to the input distorted baby image; A segmentation network, connected to the synthesis network, is used to output a baby image synthesis segmentation result corresponding to the baby image based on the input deformed baby image and the synthesized child image output by the synthesis network.

12. An image synthesis and segmentation apparatus, characterized in that, The device includes: Acquisition device for acquiring target image data, the target image data including images of children; A processing device is used to deform the child image to obtain a deformed child image corresponding to the child image; wherein, multiple standard infant images are selected as templates, the child image to be segmented is deformed and registered to each of the standard infant images, and a deformation field is applied to the child image to be segmented to obtain a deformed child image corresponding to the child image to be segmented; the deformed child image is similar in size and shape to each of the standard infant images, but the tissue intensity distribution is different, and the obtained deformed child image retains the original contrast between white matter and gray matter while being similar in size and shape to the standard infant images; A synthetic segmentation device is used to input the child image and the deformed child image into a pre-trained image synthetic segmentation network, and output a synthetic segmentation result corresponding to the child image; wherein, the image synthetic segmentation network includes: A synthesis network is used to output a synthesized baby image corresponding to the input deformed child image; A segmentation network, connected to the synthesis network, is used to output a child image synthesis segmentation result corresponding to the child image based on the input deformed child image and the synthesized baby image output by the synthesis network.

13. An image synthesis and segmentation apparatus, characterized in that, The device includes: Acquisition device for acquiring target image data, the target image data including images of infants and children; A processing device is used to deform the infant image to obtain a deformed infant image corresponding to the infant image; multiple standard child images are selected as templates, and each standard child image is aligned with the infant image to be segmented through affine registration to obtain affine child standard images with the same size and shape as the infant image to be segmented, respectively corresponding to each of the standard child images; then, the infant image to be segmented is registered to each affine child standard image through deformable registration, thereby obtaining the deformed infant image corresponding to the infant image to be segmented; the deformed infant image has a similar size and shape to each of the affine child standard images, but a different tissue intensity distribution, and the contrast between the white matter and gray matter in the obtained deformed infant image is higher than that in the infant image to be segmented; The processing device is further configured to perform deformation processing on the child image to obtain a deformed child image corresponding to the child image; select multiple standard infant images as templates, perform deformation registration on the child image to be segmented to each of the standard infant images, apply the deformation field to the child image to be segmented, and obtain a deformed child image corresponding to the child image to be segmented; the deformed child image is similar in size and shape to each of the standard infant images, but the tissue intensity distribution is different, and the obtained deformed child image retains the original contrast between white matter and gray matter while being similar in size and shape to the standard infant image; A synthetic segmentation device is used to input the infant image, the deformed infant image, the child image, and the deformed child image into a pre-trained image synthetic segmentation network, and output synthetic segmentation results corresponding to the infant image and the child image; wherein, the image synthetic segmentation network includes: A synthesis network is used to output a synthesized child image corresponding to the distorted baby image and a synthesized baby image corresponding to the distorted child image, based on the input distorted baby image and distorted child image; A segmentation network, connected to the synthesis network, is used to output a baby image synthesis segmentation result corresponding to the baby image and a child image synthesis segmentation result corresponding to the child image, based on the input deformed baby image and its corresponding synthesized child image, and the deformed child image and its corresponding synthesized baby image.

14. A computer-readable storage medium, characterized in that, The device stores computer instructions that, when executed, perform the method as described in any one of claims 1 to 10.

15. An electronic terminal, characterized in that, The terminal includes: a memory and a processor; the memory is used to store computer instructions; the processor executes the computer instructions to implement the method as described in any one of claims 1 to 10.

Images