A method for determining an image synthesis model and a related device
By constructing the distribution loss function in a multimodal scenario and adjusting the layer parameters of the feature extraction sub-modal model, the image synthesis model is trained, and the problem of inaccurate synthesis of missing modal images in the prior art is solved, and the image acquisition efficiency is improved.
Patent Information
- Application Number
- CN202210429144.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-22
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-04-22
AI Technical Summary
In multimodal scenarios, the synthesis of missing modal images in the prior art is inaccurate and difficult to meet the needs, and the high variability between devices and expensive inspection costs increase the difficulty of acquiring and utilizing multimodal images.
By obtaining the training sample pair, inputting the initial synthesis model, the model includes a feature extraction sub-model for extracting the features of the first and second modal image, constructing a distribution loss function to reduce the difference in the feature distribution, and adjusting the layer parameters to train the image Synthesis model to enhance the effective feature transferability of cross-modal image synthesis.
A missing image synthesis is achieved that is closer to reality, which improves image acquisition efficiency in multimodal scenes and reduces manpower and material consumption.
Smart Images

Figure CN115131194B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a method for determining an image synthesis model and related devices. Background Art
[0002] The target object can have images of multiple modalities. Images of different modalities can reflect the characteristics of different aspects of the target object. Therefore, images of multiple modalities can describe the target object more comprehensively.
[0003] However, images of multiple modalities often need to be acquired through multiple channels, which consumes a lot of manpower and material resources. For example, Magnetic Resonance Imaging (MRI), which has modality diversity, can obtain multimodal images. This imaging method has significantly improved the productivity of routine diagnosis and advanced research. However, the high variability between devices and the expensive inspection costs make the acquisition and use of multimodal images challenging.
[0004] In multimodal scenarios, synthesizing images of missing modalities by using images of existing modalities is an effective means to reduce the consumption of manpower and material resources. However, the images of missing modalities synthesized in related technologies are not accurate and cannot meet the needs. Summary of the invention
[0005] In order to solve the above technical problems, the present application provides a method for determining an image synthesis model and a related device. The obtained image synthesis model can synthesize missing images that are closer to reality, thereby improving the image acquisition efficiency in multimodal scenarios.
[0006] The embodiments of the present application disclose the following technical solutions:
[0007] On the one hand, an embodiment of the present application provides a method for determining an image synthesis model, the method comprising:
[0008] Acquire a training sample pair, wherein the training sample pair includes a first sample image of a first modality and a second sample image of a second modality;
[0009] Input the training sample pair into an initial synthesis model, wherein the initial synthesis model includes a first feature extraction submodel for inputting the first sample image and a second feature extraction submodel for inputting the second sample image, wherein the first feature extraction submodel includes N first convolutional layers, and the second feature extraction submodel includes N second convolutional layers, where N≥1;
[0010] Constructing an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of the i-th first convolutional layer and a second output feature of the i-th second convolutional layer, wherein the first output feature is determined according to the first sample image, the second output feature is determined according to the second sample image, and i is a positive integer less than or equal to N;
[0011] According to the i-th distribution loss function, the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer are adjusted by minimizing the optimization goal of the distribution difference, and the initial synthesis model is trained to obtain an image synthesis model, which is used to perform cross-modal image synthesis between the first modality and the second modality.
[0012] On the other hand, an embodiment of the present application provides a device for determining an image synthesis model, the device comprising:
[0013] A sample pair acquisition unit, configured to acquire a training sample pair, wherein the training sample pair includes a first sample image of a first modality and a second sample image of a second modality;
[0014] A sample pair input unit, used for inputting the training sample pair into an initial synthesis model, wherein the initial synthesis model includes a first feature extraction sub-model for inputting the first sample image and a second feature extraction sub-model for inputting the second sample image, wherein the first feature extraction sub-model includes N first convolutional layers, and the second feature extraction sub-model includes N second convolutional layers, where N≥1;
[0015] a distribution loss function determining unit, configured to construct an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of an i-th first convolutional layer and a second output feature of an i-th second convolutional layer, wherein the first output feature is determined according to the first sample image, the second output feature is determined according to the second sample image, and i is a positive integer less than or equal to N;
[0016] A layer parameter adjustment unit is used to adjust the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer according to the i-th distribution loss function by minimizing the optimization goal of the distribution difference, and train the initial synthesis model to obtain an image synthesis model, wherein the image synthesis model is used to perform cross-modal image synthesis between the first modality and the second modality.
[0017] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:
[0018] The memory is used to store program code and transmit the program code to the processor;
[0019] The processor is used to execute the method for determining the image synthesis model according to the instructions in the program code.
[0020] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method for determining the image synthesis model.
[0021] On the other hand, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method for determining the image synthesis model.
[0022] It can be seen from the above technical solution that after obtaining the training sample pair, the training sample pair can be input into the initial synthesis model, wherein the training sample pair includes a first sample image of a first modality and a second sample image of a second modality, and the initial synthesis model includes a first feature extraction sub-model for inputting the first sample image and a second feature extraction sub-model for inputting the second sample image, the first feature extraction sub-model includes N layers of first convolutional layers, and the second feature extraction sub-model includes N layers of second convolutional layers, wherein the i-th first convolutional layer can output a first output feature determined according to the first sample image, and the i-th second convolutional layer can output a second output feature determined according to the second sample image, and determine the distribution difference in feature distribution between the first output feature and the second output feature. Since the distribution difference can identify the modal difference between the first modality and the second modality, the larger the difference is, the more unfavorable it is for the subsequent use of the image synthesis model. Cross-modal image synthesis is performed between the first modality and the second modality, so the i-th distribution loss function is constructed through the distribution difference. According to the i-th distribution loss function, the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer are adjusted by minimizing the optimization goal of the distribution difference, so that the initial synthesis model is trained to obtain the image synthesis model. Since the influence of the modality difference between the first modality and the second modality on the image synthesis model is reduced through the above training, the transferability of effective features in cross-modal synthesis is enhanced, so that when the image synthesis model performs cross-modal image synthesis, for example, synthesizing the missing image of the second modality based on the target image of the first modality, since the difference in feature distribution under the two modalities is very small, the image synthesis model can accurately convert the image features of the missing image according to the output features of the target image, thereby synthesizing a missing image that is closer to the real one, thereby improving the image acquisition efficiency in multi-modal scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1 A schematic diagram of determining a scene for an image synthesis model provided in an embodiment of the present application;
[0025] Figure 2 A method flow chart of a method for determining an image synthesis model provided in an embodiment of the present application;
[0026] Figure 3 A schematic diagram of a model training provided in an embodiment of the present application;
[0027] Figure 4 A schematic diagram of another model training provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of another model training provided in an embodiment of the present application;
[0029] Figure 6 A schematic diagram of another model training provided in an embodiment of the present application;
[0030] Figure 7 A schematic diagram of another model training provided in an embodiment of the present application;
[0031] Figure 8 A schematic diagram of image synthesis provided in an embodiment of the present application;
[0032] Fig. 9 A schematic diagram of a model verification provided in an embodiment of the present application;
[0033] Fig.10 Another model verification schematic diagram provided for an embodiment of the present application;
[0034] Fig.11 A device structure diagram of a device for determining an image synthesis model provided in an embodiment of the present application;
[0035] Fig.12 A structural diagram of a terminal device provided in an embodiment of the present application;
[0036] Fig.13 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] The embodiments of the present application are described below in conjunction with the accompanying drawings.
[0038] In multimodal scenarios, synthesizing images of missing modalities by using images of existing modalities is an effective means to reduce the consumption of manpower and material resources. However, the images of missing modalities synthesized in related technologies are not accurate and cannot meet the needs.
[0039] To this end, the present application provides a method for determining an image synthesis model and related devices. The obtained image synthesis model can synthesize missing images that are closer to reality, thereby improving the image acquisition efficiency in multimodal scenarios.
[0040] The method for determining the image synthesis model provided in the embodiment of the present application can be implemented by a computer device, which can be a terminal device or a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.
[0041] This application can be applied to the field of artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0042] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0043] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.
[0044] The embodiments of the present application mainly relate to computer vision technology and machine learning.
[0045] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0046] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0047] For example, in an embodiment of the present application, various types of information included in an image can be identified based on computer vision technology, and an initial synthesis model can be trained based on machine learning to obtain an image synthesis model, so that when the image synthesis model performs cross-modal image synthesis, for example, synthesizing a missing image of a second modality based on a target image of a first modality, since the feature distribution difference under these two modalities is very small, the image synthesis model can accurately convert the image features of the missing image according to the output features of the target image, thereby synthesizing a missing image that is closer to the reality, thereby improving the image acquisition efficiency in multimodal scenarios.
[0048] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0049] It is understandable that in the specific implementation of the present application, the image samples and target images used may involve user information, facial features and other related data. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0050] In order to facilitate understanding of the technical solution provided by the present application, a method for determining an image synthesis model provided by an embodiment of the present application will be introduced below in combination with an actual application scenario.
[0051] Figure 1 A schematic diagram of an image synthesis model determination scene provided in an embodiment of the present application is shown, wherein the server 100 is used as the aforementioned computer device for illustration.
[0052] First, obtain a training sample pair 20 and an initial synthesis model 10 for training the model, the training sample pair 20 includes a first sample image of a first modality and a second sample image of a second modality, the initial synthesis model 10 includes a first feature extraction sub-model 11 and a second feature extraction sub-model 12, the first feature extraction sub-model 11 includes N first convolutional layers, the second feature extraction sub-model 12 includes N second convolutional layers, the first convolutional layer has an arrangement order according to the flow of data, the first convolutional layer has layer numbers 1 to N according to the arrangement order, the second convolutional layer has an arrangement order according to the flow of data, and the second convolutional layer has layer numbers 1 to N according to the arrangement order.
[0053] Afterwards, the training sample pair 20 is input into the initial synthesis model 10. Specifically, the first sample image is input into the first feature extraction sub-model 11, and the second sample image is input into the second feature extraction sub-model 12. The i-th first convolution layer can output a first output feature determined according to the first sample image, and the i-th second convolution layer can output a second output feature determined according to the second sample image. N is a positive integer greater than or equal to 1, i is a positive integer less than or equal to N, the i-th first convolution layer is one of the N first convolution layers, and the i-th second convolution layer is one of the N second convolution layers. Figure 1 The example where N is greater than 3 is used for explanation.
[0054] Afterwards, the distribution difference in feature distribution between the first output feature and the second output feature can be determined. Since the distribution difference can identify the modal difference between the first modality and the second modality, the larger the difference is, the more unfavorable it is for the subsequent cross-modal image synthesis between the first modality and the second modality through the image synthesis model. Therefore, the i-th distribution loss function is constructed through the distribution difference. According to the i-th distribution loss function, the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer are adjusted by the optimization goal of minimizing the distribution difference, so that the initial synthesis model 10 is trained to obtain the image synthesis model 30.
[0055] Due to the above training, the influence of the modal difference between the first modality and the second modality on the image synthesis model is reduced, and the transferability of effective features in cross-modal synthesis is enhanced. When the image synthesis model performs cross-modal image synthesis, for example, synthesizing the missing image of the second modality based on the target image of the first modality, since the difference in feature distribution under the two modalities is very small, the image synthesis model can accurately convert the image features of the missing image according to the output features of the target image, thereby synthesizing a missing image that is closer to the reality, thereby improving the image acquisition efficiency in multimodal scenarios.
[0056] Figure 2 A method flow chart of a method for determining an image synthesis model provided in an embodiment of the present application. In this embodiment, a server is used as the aforementioned computer device for illustration. The method includes:
[0057] S101, obtaining training sample pairs.
[0058] In an embodiment of the present application, a training sample pair is used to train an initial synthesis model, and the initial synthesis model includes a first feature extraction submodel and a second feature extraction submodel, the first feature extraction submodel is used to extract features of an image of a first modality, and the second feature extraction submodel is used to extract features of an image of a second modality. For the initial synthesis model, the training sample pair includes a first sample image of the first modality and a second sample image of the second modality, so that the first sample image is input into the first feature extraction submodel, and the second sample image is input into the second feature extraction submodel. Specifically, the first sample image can be recorded as X, and the second sample image can be recorded as Y.
[0059] The training sample pair may be derived from a training sample set, and the training sample set may include multiple training sample pairs. The multiple training sample pairs may be sequentially input into the initial synthesis model to train the initial synthesis model. Each training sample pair includes two sample images of different modalities. In the embodiment of the present application, a training sample pair including a first sample image and a second sample image is used as an example for explanation. Among them, images of different modalities of the target object are used to reflect the characteristics of different aspects of the target object. Images of multiple modalities can more comprehensively describe the target object. The target object may be a person, a scene, an object to be detected, etc. Images of different modalities of the target object may be, for example, images of different image styles of the target object, or images of different parameters for obtaining the target object. A modality can be used as a domain, and image conversion between different modalities can also be called image conversion between different domains.
[0060] For example, magnetic resonance imaging is a medical imaging technology used for medical diagnosis. Magnetic resonance imaging can be used to obtain multi-modal images of the target object. Based on the multi-modal images, various characteristic data of the target object can be obtained respectively. Images of different modalities can be obtained by different image generation devices. The image generation device is a detection device. The target object can be the brain, spinal cord, heart and blood vessels, joint bones, soft tissues and pelvic cavity, etc. Taking the target object as the brain as an example, the image of mode 1 is used to obtain the spin relaxation time T2 of the brain, the image of mode 2 is used to obtain the proton density (PD) of the brain, the image of mode 3 is used to obtain the magnetic resonance imaging fluid attenuated inversion recovery sequence (FLAIR) of the brain, and the image of mode 4 is used to obtain the lattice relaxation time T1 of the brain. In addition, there are other modes for obtaining physical properties such as diffusion coefficient, magnetic susceptibility, and chemical shift.
[0061] However, the acquisition of multimodal images often requires the use of multiple channels, which requires more manpower and material resources. In multimodal scenarios, synthesizing images of missing modalities through images of existing modalities is an effective means of reducing the consumption of manpower and material resources. However, the images of missing modalities synthesized in related technologies are not accurate and difficult to meet the needs. For example, the synthesized images of missing modalities cannot accurately reflect the target object, resulting in the unusable synthesized images, and the purpose of reducing manpower and material resources cannot be achieved. The initial synthesis model provided in the embodiment of the present application can be trained and converted into an image synthesis model. The image synthesis model can obtain missing images close to reality, thereby improving the image acquisition efficiency in multimodal scenarios.
[0062] Specifically, the training sample set including the training sample pairs may include sample images generated by different image generation devices, so as to prevent overfitting of the initial synthesis model trained using the training sample set. The initial synthesis model extracts fewer features generated by differences between devices, which is conducive to improving the generalization of the initial synthesis model. For example, the training sample set includes multiple images of modality 1, and the multiple images of modality 1 come from device A and device B respectively.
[0063] Specifically, the training sample set including the training sample pairs may include sample images from different institutions, so as to prevent overfitting of the initial synthesis model trained using the training sample set. The initial synthesis model extracts fewer features generated by operational differences between institutions, which is conducive to improving the generalization of the initial synthesis model. For example, the training sample set includes multiple images of modality 1, and the multiple images of modality 1 come from hospital C, hospital D, and research institute E, respectively.
[0064] Specifically, the sample images in the training sample set including the training sample pair may include images of target objects of the same category in different states. The category of the target object may be determined according to the function of the image synthesis model. The target objects of the same category may share the same image synthesis model. The target objects of the same category may be, for example, target objects belonging to people, target objects belonging to the same kind of animals, target objects belonging to the same kind of scenery, and the same detection parts. The state of the target object may be determined according to the function of the image synthesis model. The images of the target object in different states may share the same image synthesis model. The target object may be in different states, for example, people in different standing positions, and may have the same detection parts in different states. The same detection parts in different states may be, for example, a cranial brain in a healthy state and a cranial brain in a brain tumor state. As an example, the training sample set includes a first subset and a second subset. The first subset includes cranial brain images in a healthy state, and the second subset includes cranial brain images in a brain tumor state. The cranial brain images in a healthy state may come from the public IXI data set, and the cranial brain images in a brain tumor state may come from the public BraTS data set.
[0065] When the first sample image and the second sample image are images based on the brain, the brain data can be split into multiple two-dimensional slice data, the first sample image can be an image corresponding to the two-dimensional switching data of the last two bottleneck layers, and the second sample image can also be an image corresponding to the two-dimensional switching data of the last two bottleneck layers.
[0066] During specific implementation, the first sample image and the second sample image belonging to the same training sample pair are generated based on the same type of target object, and may be generated based on different target objects or the same target object. Images of different modalities generated based on the same target object are more likely to extract deep information that is independent of the modality, which is beneficial to improving the accuracy of image synthesis of the image synthesis model. For example, the first sample image and the second sample image are images of different modalities obtained by multiple detections of the same cranial brain. Of course, since it is difficult to generate multimodal images based on the same target object, in actual operations, multimodal images based on the same type of target object are often used as the first sample image and the second sample image of the same training sample pair, which brings challenges to improving the accuracy of the model. The embodiment of the present application eliminates inter-domain differences by improving the transferability of inter-domain features, thereby solving the generalization dilemma of the multivariate nature of multimodal images.
[0067] S102, input the training sample pairs into the initial synthesis model.
[0068] In the embodiment of the present application, the initial synthesis model can be used to implement a convolutional sparse coding algorithm, wherein the sparse coding (SC) algorithm is an image reconstruction technology based on an unsupervised manner, which represents sample data more efficiently by finding a set of basis vectors. The purpose of the sparse coding algorithm is to find a set of "over-complete" basis vectors (over-complete dictionary, also known as dictionary) D so that the input vector (Input signal) X i It can be expressed as a linear combination of these basis vectors, through the sparse coefficient d i The combination of basis vectors corresponding to each input vector can be represented. The sparse coefficient can be determined through training and reflected by the coefficients in the sparse coding algorithm. The input vector can be expressed as: X i =D*d i , where X i ∈R d×1 , that is, a matrix with d rows and 1 column, D∈R d×m , which is a matrix with d rows and m columns, d i ∈R m×1, which is a matrix with m rows and one column. In the sparse coding algorithm, by training two basis vectors corresponding to the source domain and the target domain images respectively, the sparse representation of the source domain and the target domain are made equal. After the source domain image is sparsely represented, the basis vector corresponding to the source domain is replaced with the basis vector corresponding to the target domain to synthesize the target domain image.
[0069] The convolutional sparse coding algorithm is a sparse coding algorithm implemented by a convolutional layer. Compared with the sparse coding algorithm, the convolutional sparse coding algorithm can realize the combination of image features and process global data. It is widely used in the learning of translation-invariant dictionaries in image processing. The convolutional sparse coding algorithm uses a sample-adaptive dictionary, in which each filter is a linear combination of a set of basic filters learned from the data. This increases the flexibility of data calculation and allows the capture of a large number of sample-dependent patterns. It is more prominent when processing large or high-dimensional data sets. The model based on convolutional sparse coding can make the convolutional sparse coding algorithm have less computational time and space complexity through learning. In the convolutional sparse coding algorithm, an image with N pixels is used as the input vector X. The convolutional sparse coding algorithm is implemented by the filter D in the model. The filter includes m basic filters. The parameters of the m basic filters are used to reflect the sparse coefficients. The i-th basic filter d among the m basic filters is the i-th basic filter d. i For the small-size image n that matches the convolution kernel, feature extraction is performed, and the basis vector Γ i is the i-th feature map, which is an image with the same size as the input vector X and corresponding to the i-th basic filter. The input vector can be expressed as:
[0070]
[0071] Similarly, in the convolutional sparse coding algorithm, by training two basis vectors corresponding to the source domain and target domain images respectively, the sparse representations of the source domain and the target domain are made equal. Then, after the source domain image is sparsely represented, the basis vectors corresponding to the source domain are replaced with the basis vectors corresponding to the target domain to synthesize the target domain image.
[0072] The filter D in the convolutional sparse coding algorithm can be implemented by a convolutional layer, and the convolutional layer serves as a convolutional filter. The initial synthesis model based on the convolutional sparse coding algorithm may include a first feature extraction sub-model and a second feature extraction sub-model. The first feature extraction sub-model is used to extract features of an image of a first modality, and the second feature extraction sub-model is used to extract features of an image of a second modality. The first feature extraction sub-model includes N first convolutional layers, and the second feature extraction sub-model includes N second convolutional layers, N≥1, that is, the first feature extraction sub-model and the second feature extraction sub-model have the same number of convolutional layers.
[0073] Among them, when N is 1, the first feature extraction submodel includes 1 first convolution layer, and the second feature extraction submodel includes 1 second convolution layer. The features of the image of the first modality can be extracted through 1 first convolution layer, and the features of the image of the second modality can be extracted through 1 second convolution layer; when N is greater than 1, the first feature extraction submodel includes multiple first convolution layers, and the second feature extraction submodel includes multiple second convolution layers. The first convolution layer has an arrangement order according to the data flow direction, and the first convolution layer has a layer number according to the arrangement order. The second convolution layer has an arrangement order according to the data flow direction, and the second convolution layer has a layer number according to the arrangement order. The layer number of the first convolution layer and the layer number of the second convolution layer are both positive integers less than or equal to N. Then the output feature of the i-th first convolution layer is used as the i-th +1 layer of the first convolutional layer, the output features of the i-th second convolutional layer are used as the input features of the i+1-th second convolutional layer, i is a positive integer less than or equal to N, the feature size of the output features of the i+1-th first convolutional layer is smaller than the feature size of the output features of the i-th first convolutional layer, the feature size of the output features of the i+1-th second convolutional layer is smaller than the feature size of the output features of the i-th second convolutional layer, the hierarchical structure of the first convolutional layer enables the multi-layer first convolutional layer to extract deeper features of the image of the first modality, the hierarchical structure of the second convolutional layer enables the multi-layer second convolutional layer to extract deeper features of the image of the second modality, the multi-layer first convolutional layer and the multi-layer second convolutional layer can be used as convolutional filters, taking advantage of the depth and conveying information in an increasingly compact manner.
[0074] For example, N can be 9, a batch normalization layer can be added after the first convolution layer to promote convergence, a batch normalization layer can also be added after the second convolution layer to promote convergence, a global spatial average pooling layer can be set after the last first convolution layer, and a global spatial average pooling layer can be set after the last second convolution layer. The sampling step size of the first convolution layer and the second convolution layer can be 2 to achieve spatial subsampling of the first sample image and the second sample image. 200 epochs can be set in the initial synthesis model, the learning rate can be set to 0.0002, the batch size can be 32, the balance parameters are set to λ=0.2, β=1, a Gaussian kernel is used in the equation, and has a bandwidth configured as the median pairwise square distance.
[0075] In the embodiment of the present application, the training sample pair can be input into the initial synthesis model. Specifically, the first sample image can be input into the first feature extraction sub-model, and the second sample image can be input into the second feature extraction sub-model. The i-th first convolution layer can output the first output feature determined according to the first sample image, and the i-th second convolution layer can output the second output feature determined according to the second sample image. The first output feature can be recorded as Z x,i , the second output feature can be recorded as Z y,i, when i is 1, the first output feature can be recorded as Z x,1 , the second output feature can be recorded as Z y,1 The first convolutional layer of the i-th layer is used as a feature extractor, its operation function is represented by f, and the output result of the filter is represented by F. Then the first output feature can be expressed as: Z x,i =f(X,F x,i-1 ,λ), the second output feature can be expressed as: Z y,i =f(Y,F y,i-1 ,λ),λ is a constant, A feature representing a tensor attribute of height h and width w, Features representing attributes of a tensor of height h and width w.
[0076] In the embodiment of the present application, the first output feature and the second output feature can be processed by intra-domain standardization (IDS), see Figure 3 , is a schematic diagram of a model training provided in an embodiment of the present application. The difference of intra-domain features can be reduced through intra-domain standardization. The difference of intra-domain features is usually harmful to the effective learning of features. Excessive learning of features with differences in the domain can easily lead to model overfitting. Reducing the difference of intra-domain features can prevent model overfitting. The difference of intra-domain features may be caused by different image generation devices or different image sources. For example, different hospitals use different scanning parameters to obtain different images. Different images have differences in features. Such differences in features are interference features of effective features.
[0077] Specifically, the first initial output feature of the first convolutional layer of the i-th layer and the second initial output feature of the second convolutional layer of the i-th layer can be obtained, and the first initial output feature is mapped to the first image size through global normalization to obtain the first output feature corresponding to the first initial output feature, and the second initial output feature is mapped to the second image size through global normalization to obtain the second output feature corresponding to the second initial output feature. The first image size can be the size of a unit sphere, and the second image size is the size of a unit sphere, so that the maximum norms of the first output feature and the second output feature can be unified, eliminating the scaling ambiguity of the first output feature and the second output feature.
[0078] S103: construct an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of the i-th first convolutional layer and a second output feature of the i-th second convolutional layer.
[0079] In the embodiment of the present application, after obtaining that the i-th first convolutional layer can output the first output feature determined according to the first sample image, and the i-th second convolutional layer can output the second output feature determined according to the second sample image, the distribution difference between the first output feature and the second output feature in feature distribution can be determined. Since the distribution difference can identify the modal difference between the first modality and the second modality, the larger the difference is, the less conducive it is to the subsequent cross-modal image synthesis between the first modality and the second modality through the image synthesis model. Therefore, the i-th distribution loss function is constructed through the distribution difference. The distribution loss function characterizes the size of the distribution difference. The larger the value of the loss function, the larger the distribution difference, and the smaller the value of the loss function, the smaller the distribution difference. Among them, the first output feature and the second output feature are features processed by IDS, see Figure 3 .
[0080] Specifically, the first output feature and the second output feature can be mapped into the same feature space to determine the distribution difference in feature distribution, and then the i-th distribution loss function can be constructed according to the distribution difference. The feature space can be a Hilbert space, and the distribution difference in feature distribution can be determined using a double sample test method such as a maximum mean discrepancy (MMD) or a mean embedding test.
[0081] Narrowing the distribution difference between the first output feature and the second output feature in feature distribution is an inter-domain adaptation (IDA) process that essentially represents the invariant structure across modalities and effectively learns the relevant sparse features of different domains. Through the IDA process, the inter-domain difference can be narrowed in an unsupervised manner to correct the misalignment and adjust the model to better generalize different data sets to achieve the purpose of adaptation. According to the maximum mean difference, the i-th distribution loss function can be determined as:
[0082]
[0083] Among them, f i (X) is the first output feature, φ(f i (X)) is the feature distribution of the first output feature, f i (Y) is the second output feature, φ(f i (Y) is the feature distribution of the second output feature, is the expectation of the feature distribution of the first output feature, is the expectation of the feature distribution of the second output feature, is the distribution difference between the first output feature and the second output feature in feature distribution.
[0084] S104, according to the i-th distribution loss function, adjust the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer by minimizing the optimization goal of the distribution difference, and train the initial synthesis model to obtain the image synthesis model.
[0085] Both sparse coding and convolutional sparse coding in related technologies can realize image synthesis. However, since sparse coding and convolutional sparse coding only focus on the image features of their respective domains and do not consider the adaptive extraction of inter-domain features, the image features have poor transferability between different domains. Based on this, after constructing the i-th distribution loss function according to the distribution difference between the first output feature and the second output feature in feature distribution, the layer parameters of the i-th first convolution layer and the i-th convolution layer can be adjusted according to the i-th distribution loss function by minimizing the optimization goal of the distribution difference, so as to train the initial synthesis model to obtain the image synthesis model, and the image synthesis model can be used for cross-modal image synthesis between the first modality and the second modality. That is, the transferability of convolutional sparse features can be enhanced and quantified through the distribution difference in feature distribution, such as MMD, and the transferability of convolutional sparse features can be improved by minimizing the distribution difference, generalizing the heterogeneous representation of features of cross-modal images.
[0086] Since the distribution difference can identify the modal difference between the first modality and the second modality, the larger the difference is, the more unfavorable it is for the subsequent cross-modal image synthesis between the first modality and the second modality through the image synthesis model. Therefore, the layer parameters are adjusted by minimizing the optimization goal of the distribution difference, thereby reducing the influence of the modal difference between the first modality and the second modality on the image synthesis model, and enhancing the transferability of effective features in cross-modal synthesis. When the image synthesis model performs cross-modal image synthesis, for example, synthesizing the missing image of the second modality based on the target image of the first modality, since the feature distribution difference under the two modalities is very small, the image synthesis model can accurately convert the image features of the missing image according to the output features of the target image, thereby synthesizing a missing image that is closer to the reality, thereby improving the image acquisition efficiency in multi-modal scenarios.
[0087] In the above description, the i-th first convolution layer, the i-th second convolution layer and the i-th distribution loss function are used as examples. In actual operation, the layer numbers i of the first convolution layer and the second convolution layer that need to be adjusted can be set in advance, and the layer parameters of the i-th first convolution layer and the i-th convolution layer can be adjusted according to the i-th distribution loss function by minimizing the optimization goal of the distribution difference.
[0088] In actual operation, multiple distribution loss functions can also be set. Multiple distribution loss functions correspond to multiple first convolutional layers and multiple second convolutional layers. The number of distribution loss functions can be equal to the number of first convolutional layers N, where N is greater than 1, that is, each first convolutional layer can correspond to a distribution loss function, and the layer parameters of each first convolutional layer and each second convolutional layer can be adjusted to make the model have higher accuracy. See Figure 4 , is a schematic diagram of another model training provided in an embodiment of the present application, wherein the 1st first convolutional layer and the 1st second convolutional layer correspond to the 1st distribution loss function, the i-th first convolutional layer and the i-th second convolutional layer correspond to the i-th distribution loss function, and the N-th first convolutional layer and the N-th second convolutional layer correspond to the N-th distribution loss function, see Figure 5 , which is a schematic diagram of another model training provided in an embodiment of the present application, wherein the first output feature and the second output feature are features that have passed through IDS; the number of distribution loss functions may also be less than the number N of first convolutional layers, that is, some first convolutional layers may correspond to distribution loss functions, and the layer parameters of multiple first convolutional layers and multiple second convolutional layers may be adjusted to make the model have higher accuracy.
[0089] When the number of distribution loss functions is equal to the number N of first convolutional layers and N is greater than 1, the N first convolutional layers and the N second convolutional layers all have a one-to-one correspondence with the N distribution loss functions, wherein the i-th distribution loss function among the N distribution loss functions corresponds to the i-th first convolutional layer and the i-th second convolutional layer. Then, according to the i-th distribution loss function, the layer parameters of the i-th first convolutional layer and the i-th convolutional layer are adjusted by minimizing the optimization objective of the distribution difference. Specifically, according to the N distribution loss functions, the layer parameters of the corresponding first convolutional layer and the second convolutional layer are adjusted by minimizing the optimization objective of the distribution difference, thereby training the initial synthesis model to obtain the image synthesis model.
[0090] In the process of training the initial synthesis model to obtain the image synthesis model, the layer parameters of the i-th first convolution layer and the i-th second convolution layer can also be adjusted through the mapping association relationship between the first output feature and the second output feature. Specifically, the first output feature can be converted into a feature corresponding to the second modality through the corresponding mapping matrix, as the conversion feature corresponding to the i-th first convolution layer, and according to the feature difference between the conversion feature and the second output feature, the i-th association loss function is constructed. According to the i-th association loss function, the layer parameters of the i-th first convolution layer and the i-th second convolution layer are adjusted by minimizing the feature difference and the optimization target of the mapping matrix. See Figure 6, which is a schematic diagram of another model training provided in an embodiment of the present application, wherein the i-th associated loss function can be constructed using the projection matrix, the first output feature, and the second output feature, and the layer parameters of the i-th first convolution layer and the i-th second convolution layer can be adjusted according to the i-th associated loss function. Figure 7 , is a schematic diagram of another model training provided in an embodiment of the present application, wherein the first output feature and the second output feature are features that have passed through IDS.
[0091] The feature difference between the conversion feature and the second output feature reflects the feature correlation between the first output feature and the second output feature. The larger the feature difference, the worse the feature correlation, and the less conducive it is to perform cross-modal image synthesis between the first modality and the second modality through the image synthesis model. Since the above training improves the correlation of effective features in cross-modal synthesis, improves the consistency of the spatial relationship between the first output feature and the second output feature, and enhances the transferability of effective features in cross-modal synthesis, the image synthesis model can accurately convert the image features of the missing image according to the output features of the target image, thereby synthesizing a missing image that is closer to the real one.
[0092] The mapping matrix can be expressed as P, and the mapping matrix corresponding to the first convolutional layer of the i-th layer can be expressed as P i , as discussed above, the first output feature can be expressed as: Z x,i =f(X,F x,i-1 ,λ), the second output feature can be expressed as: Z y,i =f(Y,F y,i-1 ,λ), two independently learned sub-models will produce irrelevant features. Connecting two sub-models in series will lead to the conversion of irrelevant features, which is easy to cause the loss of effective features and the increase of invalid features. Therefore, the joint optimization of the two sub-models can be achieved by constructing an associated loss function. The constructed i-th associated loss function can be expressed as:
[0093]
[0094] Among them, α is a constant, P i Z x,i To transform features, since the mapping matrix is an additional parameter introduced, it needs to be minimized to reduce its impact on the model.
[0095] In the above description, the i-th first convolution layer, the i-th second convolution layer and the i-th associated loss function are used as examples. In actual operation, the layer numbers i of the first convolution layer and the second convolution layer that need to be adjusted can be set in advance, and the layer parameters of the i-th first convolution layer and the i-th convolution layer can be adjusted according to the i-th associated loss function by minimizing the optimization objectives of feature differences and mapping matrices.
[0096] In actual operation, multiple associated loss functions can also be set, and multiple associated loss functions correspond to multiple first convolutional layers, and also correspond to multiple second convolutional layers. The number of associated loss functions can be equal to the number N of first convolutional layers, N is greater than 1, that is, each first convolutional layer can correspond to an associated loss function, then the layer parameters of each first convolutional layer and each second convolutional layer can be adjusted to make the model have higher accuracy; the number of associated loss functions can also be less than the number N of first convolutional layers, that is, some first convolutional layers can correspond to associated loss functions, then the layer parameters of multiple first convolutional layers and multiple second convolutional layers can be adjusted to make the model have higher accuracy.
[0097] When the number of associated loss functions is equal to the number N of first convolutional layers and N is greater than 1, the N first convolutional layers and the N second convolutional layers all have a one-to-one correspondence with the N associated loss functions, wherein the i-th associated loss function among the N associated loss functions corresponds to the i-th first convolutional layer and the i-th second convolutional layer. Then, according to the i-th associated loss function, the layer parameters of the i-th first convolutional layer and the i-th convolutional layer are adjusted by minimizing the optimization objective of feature difference and mapping matrix. Specifically, according to the N associated loss functions, the layer parameters of the corresponding first convolutional layer and the second convolutional layer are adjusted by minimizing the optimization objective of distribution difference, thereby training the initial synthesis model to obtain the image synthesis model.
[0098] In summary, we can minimize the i-th distribution loss function Minimize the i-th associated loss function To adjust the layer parameters of the i-th first convolutional layer and the layer parameters of the i-th second convolutional layer, the overall goal can be expressed as a new minimax function:
[0099]
[0100] Where β is a constant.
[0101] In a possible implementation, the present application embodiment provides a method for performing image synthesis through an image synthesis model. The method includes:
[0102] S201: Acquire a target image of a source modality of a target object.
[0103] S202: According to the target image, synthesize the missing image of the target modality through an image synthesis model.
[0104] In the embodiment of the present application, the trained image synthesis model has the ability to synthesize the image of the second modality using the image of the first modality, and also has the ability to synthesize the image of the first modality using the image of the second modality. Then, the target image of the source modality of the target object can be obtained, and the missing image of the target modality can be synthesized by the image synthesis model. See Figure 8 , is a schematic diagram of image synthesis provided in an embodiment of the present application, wherein the image synthesis model 30 can be configured in a server, the acquisition of the target image can be achieved through a terminal connected to the server, and the missing image can be sent to the terminal connected to the server. When the source modality is the first modality, the target modality is the second modality, that is, according to the target image of the first modality, the missing image of the second modality can be synthesized by the image synthesis model. Correspondingly, when the source modality is the second modality, the target modality is the first modality, that is, according to the target image of the second modality, the missing image of the first modality can be synthesized by the image synthesis model.
[0105] According to the target image, the missing image of the target modality is synthesized through the image synthesis model. Specifically, the target image is input into the feature extraction sub-model corresponding to the source modality in the image synthesis model, and the target output features of the Nth convolutional layer in the feature extraction sub-model corresponding to the source modality are converted into features to be synthesized of the target modality. Then, according to the features to be synthesized, the missing image is synthesized through the layer parameters of the Nth convolutional layer in the feature extraction sub-model corresponding to the target modality.
[0106] Specifically, when the source modality is the first modality and the target modality is the second modality, the target image is input into the first feature extraction submodel corresponding to the first modality in the image synthesis model, and the target output features of the first convolutional layer of the Nth layer in the first feature extraction submodel are converted into features to be synthesized of the second modality, and then the missing image is synthesized according to the features to be synthesized through the layer parameters of the second convolutional layer of the Nth layer in the second feature extraction submodel corresponding to the second modality. For example, if the first modality is modality 4 for obtaining the lattice relaxation time T1 of the brain, and the second modality is modality 1 for obtaining the spin relaxation time T2 of the brain, the image of modality 1 can be synthesized according to the image of modality 4.
[0107] Specifically, when the source modality is the second modality and the target modality is the first modality, the target image is input into the second feature extraction submodel corresponding to the second modality in the image synthesis model, and the target output features of the second convolutional layer of the Nth layer in the second feature extraction submodel are converted into features to be synthesized of the first modality, and then the missing image is obtained by synthesizing the layer parameters of the first convolutional layer of the Nth layer in the first feature extraction submodel corresponding to the first modality according to the synthesized features.
[0108] Before synthesizing the missing image of the target modality through the image synthesis model, the image synthesis model can also be verified. The verification of the image synthesis model can be achieved through a test sample pair, which includes a third sample image of the first modality and a fourth sample image of the second modality. The third sample image and the fourth sample image are generated based on the same target object, and the two are well aligned. The third sample image and the fourth sample image are basic facts, which can be used to verify the quality of the synthesis result of the image synthesis model. The test sample pair can be derived from a test sample set, which can include sample images generated by different image generation devices, sample images from different institutions, and images of the same category of target objects in different states.
[0109] The test sample set and the training sample set can have the same source, and the data of the two do not overlap. For example, among 578 healthy brain images, 80 can be extracted as test samples, and the rest can be used as training samples, and among 225 brain images with brain tumors, 45 can be extracted as test samples, and the rest can be used as test samples; or among 578 healthy brain images, 60 can be extracted as test samples, and the rest can be used as training samples, and among 225 brain images with brain tumors, 20 can be extracted as test samples, and the rest can be used as test samples.
[0110] Taking the conversion of the image of the healthy brain from the image of modality 2 for obtaining the PD of the brain to the image of modality 1 for obtaining the T2 of the brain, and the conversion from the image of modality 1 to the image of modality 2 as an example, the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) are used to compare the results, see Fig. 9 , is a schematic diagram of a model verification provided in an embodiment of the present application, wherein Fig. 9 A and Fig. 9 C shows a test sample pair. Fig. 9 A is the input image of modality 1, Fig. 9 C is the real image of modality 2 corresponding to the input image, Fig. 9 B is an image of mode 2 synthesized by the image synthesis model. It can be seen from the figure that the image synthesized by the image synthesis model has a high similarity with the real image. See Table 1, which is an example of a test result provided in an embodiment of the present application.
[0111] Table 1 An example of test results
[0112]
[0113] It can be seen from the table that when the convolutional sparse network is used for graphics conversion, the PSNR and SSIM are both low. After IDS or IDA, the PSNR and SSIM are improved, and after IDS and IDA, the PSNR and SSIM reach the highest.
[0114] Of course, it is also possible to verify the conversion of the FLAIR modality 3 image used to obtain the brain into the T1 modality 4 image used to obtain the brain in the brain image of the brain tumor state, see Fig.10 , is another model verification schematic diagram provided in an embodiment of the present application, wherein Fig.10 A and Fig.10 C shows a test sample pair. Fig.10 A is the input image of mode 3, Fig.10 C is the real image of modality 4 corresponding to the input image, Fig.10 B is an image of mode 4 synthesized by the image synthesis model. From the figure, it can be seen that the image synthesized by the image synthesis model has a high degree of similarity with the real image.
[0115] Based on the above method embodiment, Fig.11 The present invention provides a device structure diagram of a device for determining an image synthesis model according to an embodiment of the present invention. The device 1100 for determining an image synthesis model includes:
[0116] A sample pair acquisition unit 1101 is used to acquire a training sample pair, where the training sample pair includes a first sample image of a first modality and a second sample image of a second modality;
[0117] A sample pair input unit 1102 is used to input the training sample pair into an initial synthesis model, wherein the initial synthesis model includes a first feature extraction submodel for inputting the first sample image and a second feature extraction submodel for inputting the second sample image, wherein the first feature extraction submodel includes N first convolutional layers, and the second feature extraction submodel includes N second convolutional layers, where N≥1;
[0118] a distribution loss function determining unit 1103, configured to construct an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of an i-th first convolutional layer and a second output feature of an i-th second convolutional layer, wherein the first output feature is determined according to the first sample image, the second output feature is determined according to the second sample image, and i is a positive integer less than or equal to N;
[0119] The layer parameter adjustment unit 1104 is used to adjust the layer parameters of the i-th first convolution layer and the i-th second convolution layer according to the i-th distribution loss function by minimizing the optimization goal of the distribution difference, and train the initial synthesis model to obtain an image synthesis model, and the image synthesis model is used to perform cross-modal image synthesis between the first modality and the second modality.
[0120] Optionally, the device further comprises:
[0121] An initial feature acquisition unit, configured to acquire the first initial output feature of the i-th first convolutional layer and the second initial output feature of the i-th second convolutional layer before constructing the i-th distribution loss function according to the distribution difference between the first output feature of the i-th first convolutional layer and the second output feature of the i-th second convolutional layer in feature distribution;
[0122] A mapping unit is used to map the first initial output feature to a first image size through global normalization to obtain the first output feature corresponding to the first initial output feature; and map the second initial output feature to a second image size through global normalization to obtain the second output feature corresponding to the second initial output feature.
[0123] Optionally, the distribution loss function determining unit includes:
[0124] a space mapping unit, configured to determine the distribution difference in feature distribution by mapping the first output feature and the second output feature into a same feature space;
[0125] The distribution loss function determination subunit is used to construct the i-th distribution loss function according to the distribution difference.
[0126] Optionally, when N≥2, the first output feature is the input feature of the first convolutional layer of the i+1th layer, the second output feature is the input feature of the second convolutional layer of the i+1th layer, the feature size of the output feature of the first convolutional layer of the i+1th layer is smaller than the feature size of the first output feature, and the feature size of the output feature of the second convolutional layer of the i+1th layer is smaller than the feature size of the second output feature.
[0127] Optionally, the N first convolutional layers and the N second convolutional layers each have a one-to-one correspondence with N distribution loss functions, wherein the i-th distribution loss function among the N distribution loss functions corresponds to the i-th first convolutional layer and the i-th second convolutional layer;
[0128] The layer parameter adjustment unit is specifically used for:
[0129] According to the N distribution loss functions, the layer parameters of the corresponding first convolutional layer and the second convolutional layer are adjusted by minimizing the optimization goal of the distribution difference, and the initial synthesis model is trained to obtain an image synthesis model.
[0130] Optionally, the device further comprises:
[0131] an association loss function determining unit, configured to construct an i-th association loss function according to a feature difference between a conversion feature corresponding to the i-th first convolutional layer and the second output feature in a process of training the initial synthesis model to obtain the image synthesis model, wherein the conversion feature is a feature corresponding to the second modality converted from the first output feature through a corresponding mapping matrix;
[0132] The optimization unit is further used to adjust the layer parameters of the i-th first convolution layer and the i-th second convolution layer according to the i-th associated loss function by minimizing the optimization target of the feature difference and the mapping matrix.
[0133] Optionally, when N≥2, the N first convolutional layers and the N second convolutional layers each have a one-to-one correspondence with N associated loss functions, wherein the i-th associated loss function among the N associated loss functions corresponds to the i-th first convolutional layer and the i-th second convolutional layer;
[0134] The optimization unit is specifically used for:
[0135] In the process of training the initial synthesis model to obtain the image synthesis model, according to the N associated loss functions, the layer parameters of the corresponding first convolutional layer and the second convolutional layer are adjusted by minimizing the optimization target of the feature difference and the mapping matrix.
[0136] Optionally, the training sample pairs are derived from a training sample set, the training sample set includes sample images generated by different image generating devices, and the sample images include images of target objects of the same category in different states.
[0137] Optionally, the first sample image and the second sample image are generated based on the same target object.
[0138] Optionally, the device further comprises:
[0139] An image acquisition unit, used to acquire a target image of a source modality of a target object;
[0140] An image synthesis unit, configured to synthesize a missing image of a target modality according to the target image through the image synthesis model;
[0141] When the source modality is the first modality, the target modality is the second modality; or, when the source modality is the second modality, the target modality is the first modality.
[0142] Optionally, the image synthesis unit includes:
[0143] An image input unit, used for inputting the target image into a feature extraction sub-model corresponding to the source modality in the image synthesis model;
[0144] A feature conversion unit, used to convert the target output feature of the Nth convolutional layer in the feature extraction sub-model corresponding to the source modality into a feature to be synthesized of the target modality;
[0145] The image synthesis subunit is used to synthesize the missing image according to the features to be synthesized by using the layer parameters of the Nth convolutional layer in the feature extraction submodel corresponding to the target modality.
[0146] It can be seen that after obtaining the training sample pair, the training sample pair can be input into the initial synthesis model, wherein the training sample pair includes a first sample image of the first modality and a second sample image of the second modality, and the initial synthesis model includes a first feature extraction submodel for inputting the first sample image and a second feature extraction submodel for inputting the second sample image, the first feature extraction submodel includes N layers of first convolutional layers, and the second feature extraction submodel includes N layers of second convolutional layers, wherein the i-th first convolutional layer can output a first output feature determined according to the first sample image, and the i-th second convolutional layer can output a second output feature determined according to the second sample image, and determine the distribution difference in feature distribution between the first output feature and the second output feature. Since the distribution difference can identify the modal difference between the first modality and the second modality, the larger the difference is, the more unfavorable it is for the subsequent first synthesis through the image synthesis model. Cross-modal image synthesis between the first modality and the second modality, so the i-th distribution loss function is constructed through the distribution difference, and according to the i-th distribution loss function, the layer parameters of the i-th first convolution layer and the i-th second convolution layer are adjusted by minimizing the optimization goal of the distribution difference, so that the initial synthesis model is trained to obtain the image synthesis model. Since the above training reduces the influence of the modality difference between the first modality and the second modality on the image synthesis model, the transferability of effective features in cross-modal synthesis is enhanced, so that when the image synthesis model performs cross-modal image synthesis, for example, synthesizing the missing image of the second modality based on the target image of the first modality, since the feature distribution difference under the two modalities is very small, the image synthesis model can accurately convert the image features of the missing image according to the output features of the target image, thereby synthesizing a missing image that is closer to the real one, thereby improving the image acquisition efficiency in multimodal scenarios.
[0147] The embodiment of the present application further provides a computer device, which is the computer device described above and may include a terminal device or a server, and the aforementioned image synthesis model determination device may be configured in the computer device. The computer device is described below in conjunction with the accompanying drawings.
[0148] If the computer device is a terminal device, see Fig.12 As shown, the embodiment of the present application provides a terminal device, taking the terminal device as a mobile phone as an example:
[0149] Fig.12 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.12 The mobile phone includes: a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art can understand that Fig.12 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0150] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone:
[0151] The RF circuit 1410 may be used for receiving and sending signals during information transmission or communication or during a call. In particular, after receiving downlink information from a base station, the information is sent to the processor 1480 for processing. In addition, the uplink data is sent to the base station.
[0152] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1420. The memory 1420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0153] The input unit 1430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432 .
[0154] The display unit 1440 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1440 may include a display panel 1441.
[0155] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.
[0156] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 can provide an audio interface between the user and the mobile phone.
[0157] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1470. It provides users with wireless broadband Internet access.
[0158] Processor 1480 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in memory 1420 and calling data stored in memory 1420.
[0159] The mobile phone also includes a power source 1490 (such as a battery) for supplying power to various components.
[0160] In this embodiment, the processor 1480 included in the terminal device also has the following functions:
[0161] Acquire a training sample pair, wherein the training sample pair includes a first sample image of a first modality and a second sample image of a second modality;
[0162] Input the training sample pair into an initial synthesis model, wherein the initial synthesis model includes a first feature extraction submodel for inputting the first sample image and a second feature extraction submodel for inputting the second sample image, wherein the first feature extraction submodel includes N first convolutional layers, and the second feature extraction submodel includes N second convolutional layers, where N≥1;
[0163] Constructing an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of the i-th first convolutional layer and a second output feature of the i-th second convolutional layer, wherein the first output feature is determined according to the first sample image, the second output feature is determined according to the second sample image, and i is a positive integer less than or equal to N;
[0164] According to the i-th distribution loss function, the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer are adjusted by minimizing the optimization goal of the distribution difference, and the initial synthesis model is trained to obtain an image synthesis model, which is used to perform cross-modal image synthesis between the first modality and the second modality.
[0165] If the computer device is a server, the present application embodiment also provides a server, see Fig.13 As shown, Fig.13 The structural diagram of the server 1500 provided in the embodiment of the present application, the server 1500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1522 (for example, one or more processors) and a memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the server 1500.
[0166] The server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server 2003 or Windows Server 2008. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0167] The steps performed by the server in the above embodiment can be based on Fig.13 The server structure shown.
[0168] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided in the above embodiment.
[0169] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.
[0170] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.
[0171] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0172] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Moreover, based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for determining an image synthesis model, characterized in that: The method comprises: Acquire a training sample pair, wherein the training sample pair includes a first sample image of a first modality and a second sample image of a second modality; Input the training sample pair into an initial synthesis model, wherein the initial synthesis model includes a first feature extraction submodel for inputting the first sample image and a second feature extraction submodel for inputting the second sample image, wherein the first feature extraction submodel includes N first convolutional layers, and the second feature extraction submodel includes N second convolutional layers, where N≥1; Constructing an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of the i-th first convolutional layer and a second output feature of the i-th second convolutional layer, wherein the first output feature is determined according to the first sample image, the second output feature is determined according to the second sample image, and i is a positive integer less than or equal to N; According to the i-th distribution loss function, the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer are adjusted by minimizing the optimization goal of the distribution difference, and the initial synthesis model is trained to obtain an image synthesis model, which is used to perform cross-modal image synthesis between the first modality and the second modality.
2. The method according to claim 1, characterized in that Before constructing the i-th distribution loss function based on the distribution difference between the first output feature of the i-th first convolutional layer and the second output feature of the i-th second convolutional layer, the method further includes: Obtaining a first initial output feature of the i-th first convolutional layer and a second initial output feature of the i-th second convolutional layer; The first initial output feature is mapped to a first image size through global normalization to obtain the first output feature corresponding to the first initial output feature; the second initial output feature is mapped to a second image size through global normalization to obtain the second output feature corresponding to the second initial output feature.
3. The method according to claim 1, characterized in that The constructing the i-th distribution loss function according to the distribution difference in feature distribution between the first output feature of the i-th first convolutional layer and the second output feature of the i-th second convolutional layer includes: Determining the distribution difference in feature distribution by mapping the first output feature and the second output feature into the same feature space; An i-th distribution loss function is constructed according to the distribution difference.
4. The method according to claim 1, characterized in that: When N≥2, the first output feature is the input feature of the first convolutional layer of the i+1th layer, the second output feature is the input feature of the second convolutional layer of the i+1th layer, the feature size of the output feature of the first convolutional layer of the i+1th layer is smaller than the feature size of the first output feature, and the feature size of the output feature of the second convolutional layer of the i+1th layer is smaller than the feature size of the second output feature.
5. The method according to claim 4, characterized in that The N first convolutional layers and the N second convolutional layers each have a one-to-one correspondence with N distribution loss functions, wherein the i-th distribution loss function among the N distribution loss functions corresponds to the i-th first convolutional layer and the i-th second convolutional layer; The step of adjusting the layer parameters of the i-th first convolution layer and the i-th second convolution layer according to the i-th distribution loss function by minimizing the optimization goal of the distribution difference, and training the initial synthesis model to obtain an image synthesis model comprises: According to the N distribution loss functions, the layer parameters of the corresponding first convolutional layer and the second convolutional layer are adjusted by minimizing the optimization goal of the distribution difference, and the initial synthesis model is trained to obtain an image synthesis model.
6. The method according to any one of claims 1 to 5, characterized in that: In the process of training the initial synthesis model to obtain the image synthesis model, the method further includes: Constructing an i-th association loss function according to a feature difference between a conversion feature corresponding to the i-th first convolutional layer and the second output feature, wherein the conversion feature is a feature corresponding to the second modality converted from the first output feature through the corresponding mapping matrix; According to the i-th associated loss function, layer parameters of the i-th first convolutional layer and the i-th second convolutional layer are adjusted by minimizing the optimization objective of the feature difference and the mapping matrix.
7. The method according to claim 6, characterized in that When N≥2, the N first convolutional layers and the N second convolutional layers each have a one-to-one correspondence with N associated loss functions, wherein the i-th associated loss function among the N associated loss functions corresponds to the i-th first convolutional layer and the i-th second convolutional layer; In the process of training the initial synthesis model to obtain the image synthesis model, adjusting the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer by minimizing the feature difference and the optimization target of the mapping matrix according to the i-th association loss function, comprises: According to the N associated loss functions, the layer parameters of the corresponding first convolution layer and the second convolution layer are adjusted by minimizing the optimization target of the feature difference and the mapping matrix.
8. The method according to claim 1, characterized in that The training sample pairs are derived from a training sample set, wherein the training sample set includes sample images generated by different image generating devices, and the sample images include images of target objects of the same category in different states.
9. The method according to claim 1, characterized in that: The first sample image and the second sample image are generated based on the same target object.
10. The method according to claim 1, characterized in that The method further comprises: Obtain a target image of a source modality of a target object; According to the target image, synthesizing the missing image of the target modality through the image synthesis model; When the source modality is the first modality, the target modality is the second modality; or, when the source modality is the second modality, the target modality is the first modality.
11. The method according to claim 10, characterized in that The step of synthesizing the target modality missing image by using the image synthesis model according to the target image includes: Inputting the target image into the feature extraction sub-model corresponding to the source modality in the image synthesis model; Convert the target output features of the Nth convolutional layer in the feature extraction sub-model corresponding to the source modality into features to be synthesized of the target modality; According to the features to be synthesized, the missing image is obtained by synthesizing the layer parameters of the Nth convolutional layer in the feature extraction sub-model corresponding to the target modality.
12. A device for determining an image synthesis model, characterized in that: The device comprises: A sample pair acquisition unit, configured to acquire a training sample pair, wherein the training sample pair includes a first sample image of a first modality and a second sample image of a second modality; A sample pair input unit, used for inputting the training sample pair into an initial synthesis model, wherein the initial synthesis model includes a first feature extraction sub-model for inputting the first sample image and a second feature extraction sub-model for inputting the second sample image, wherein the first feature extraction sub-model includes N first convolutional layers, and the second feature extraction sub-model includes N second convolutional layers, where N≥1; a distribution loss function determining unit, configured to construct an i-th distribution loss function according to a distribution difference in feature distribution between a first output feature of an i-th first convolutional layer and a second output feature of an i-th second convolutional layer, wherein the first output feature is determined according to the first sample image, the second output feature is determined according to the second sample image, and i is a positive integer less than or equal to N; A layer parameter adjustment unit is used to adjust the layer parameters of the i-th first convolutional layer and the i-th second convolutional layer according to the i-th distribution loss function by minimizing the optimization goal of the distribution difference, and train the initial synthesis model to obtain an image synthesis model, wherein the image synthesis model is used to perform cross-modal image synthesis between the first modality and the second modality.
13. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method for determining the image synthesis model described in any one of claims 1-11 according to the instructions in the program code.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method for determining the image synthesis model described in any one of claims 1-11.
15. A computer program product comprising instructions, characterized in that When it runs on a computer, the computer executes the method for determining the image synthesis model described in any one of claims 1-11.
